Myself (David Stockton) and my colleagues (Darren Pywell & David Tattersall) are all at CFUnited this week. We have a FREE copy of FusionAnalytics to give away so don’t be shy – come and speak to us for a chance to win!!

Plus, ask nicely and we’ve got FREE goodies for everyone we speak to!

We’re in the vendor area between Adobe & Railo – look for the ShareDox, FusionReactor, FusionAnalytics & Intergral banners… plus this good looking guy:

See you there!

http://www.adobe.com/support/coldfusion/downloads_updates.html#cf9

Well it’s not always that simple is it? So if you want some professional help from the experts then call the experts – we’re ready & waiting to help.

In one such incident we were trying to identify the IP address of a slow uploading client – this we could then link to a client account and identify where the issue was coming from. At the first stage we weren’t able to access any of the remote clients network. Using good old email the client was able to send me a copy of all the FusionReactor Crash Protection alerts. These fire under certain conditions alerting the recipient of a potential issue. You can read more about crash protection on the FusionReactor website.

Now the emails are a great feature but they’re not very easy to analyse over 100’s of emails. So we created a quick tool to analyse the crash protection emails for just this sort of event. And now we’re making it available to you… FREE!

FusionReactor CrashProtection EMail Analyzer

1. Download & unzip the contents of the download into a folder under your ColdFusion webroot (eg c:\inetpub\wwwroot\fr-mail-analyzer)
2. Create a new folder called “mails” under this folder (eg c:\inetpub\wwwroot\fr-mail-analyzer\mails)
3. Put all your *.eml files inside the mails folder – I recommend naming them 01FusionReactor Crash Protection Alert [xxxxx-y], 01FusionReactor Crash Protection Alert [xxxxx-y], 02FusionReactor Crash Protection Alert [xxxxx-y], etc
4. Open your web-browser and point it at the “read.cfm” file

Now you have a list of all the slow pages (over 60seconds) and which IPs they’ve come from and which page(s) they’ve hit – all without direct access to FusionReactor. Also great if you’ve only got access to the emails (eg your logs have rotated).

Phew – Sound like too much work? Save the hassle and get FusionAnalytics or contact us now!

http://www.fusion-reactor.com/fr/howto/

… or read the official documentation here:

http://www.fusion-reactor.com/fr/support.cfm#doc

You may also be interested in our training webinars (which get some great feedback!) – for more details, please check the FusionReactor site:

http://www.fusion-reactor.com/support/training/

What is Native Code?

Underlying all your ColdFusion goodness is Java, underlying the Java is the runtime environment typically implemented in C/C++ code. When you hit a code-block that must “go native” this is inside the C/C++ code typically waiting for an event to occur. When a thread is executing this native method the thread cannot be killed by the JVM.

What to look for?

Some of the most common examples where native code is used are:

• CFHTTP calls
• WebService calls
• JDBC Queries

What you’re looking for is “Native Method” in the stack trace of the thread. Let’s look at some concrete examples…

CFHTTP Calls

Example CF Code:

<cfhttp url="http://localhost/blogs/dont_stop_me_now/slow.cfm" />

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
HTTPClient.BufferedInputStream.fillBuff(BufferedInputStream.java:172)
HTTPClient.BufferedInputStream.read(BufferedInputStream.java:110)
HTTPClient.StreamDemultiplexor.read(StreamDemultiplexor.java:273)
HTTPClient.RespInputStream.read(RespInputStream.java:155)
HTTPClient.RespInputStream.read(RespInputStream.java:115)
HTTPClient.Response.readResponseHeaders(Response.java:1000)
HTTPClient.Response.getHeaders(Response.java:720)
HTTPClient.Response.getStatusCode(Response.java:259)
HTTPClient.RetryModule.responsePhase1Handler(RetryModule.java:83)
HTTPClient.HTTPResponse.handleResponse(HTTPResponse.java:761)
HTTPClient.HTTPResponse.getStatusCode(HTTPResponse.java:191)
coldfusion.tagext.net.HttpTag.connHelper(HttpTag.java:850)
coldfusion.tagext.net.HttpTag.doEndTag(HttpTag.java:1140)
cfslow_cfhttp2ecfm1758959420.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_cfhttp.cfm:1)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

WebService Calls

Example CF Code:

<cfset ws = createObject("webservice", "http://localhost/blogs/dont_stop_me_now/slow.cfc?wsdl") />
<cfset ws.goSlow() />

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:581)
org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:142)
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
org.apache.axis.client.Call.invokeEngine(Call.java:2765)
org.apache.axis.client.Call.invoke(Call.java:2748)
org.apache.axis.client.Call.invoke(Call.java:2424)
org.apache.axis.client.Call.invoke(Call.java:2347)
org.apache.axis.client.Call.invoke(Call.java:1804)
blogs.dont_stop_me_now.SlowCfcSoapBindingStub.goSlow(SlowCfcSoapBindingStub.java:157)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)[Native Method]
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
coldfusion.xml.rpc.ServiceProxy.invokeImpl(ServiceProxy.java:224)
coldfusion.xml.rpc.ServiceProxy.invoke(ServiceProxy.java:154)
coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2360)
cfslow_ws2ecfm1005604111.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_ws.cfm:2)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

JDBC Queries

Example CF Code:

<cfquery name="wait" datasource="test">
   SELECT 1 waitfor delay '000:00:10:000'
</cfquery>

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
macromedia.jdbc.sqlserver.SQLServerByteOrderedDataReader.makeMoreDataAvailable(null:???)
macromedia.jdbc.sqlserver.SQLServerByteOrderedDataReader.receive(null:???)
macromedia.jdbc.sqlserver.tds.TDSExecuteRequest.submitRequest(null:???)
macromedia.jdbc.sqlserver.tds.TDSRequest.execute(null:???)
macromedia.jdbc.sqlserver.SQLServerImplStatement.execute(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.commonExecute(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.executeInternal(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.execute(null:???)
coldfusion.server.j2ee.sql.JRunStatement.execute(JRunStatement.java:348)
coldfusion.sql.Executive.executeQuery(Executive.java:1229)
coldfusion.sql.Executive.executeQuery(Executive.java:1008)
coldfusion.sql.Executive.executeQuery(Executive.java:939)
coldfusion.sql.SqlImpl.execute(SqlImpl.java:341)
coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:843)
coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:533)
cfslow_db2ecfm445915345.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_db.cfm:1)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

Why!?

All these examples are in native methods for socket reading. Socket functions (both reading and writing) are the most commonly found native methods in stack traces.

What can I do?

Unfortunately, the only current work-around is to restart your server. But this is a Java limitation that even without FusionReactor you would still have the problem – FusionReactor is just giving you visibility. The real solution is to investigate the root cause of the problem and solve that – that’s where we come in! We’re experts in this field and working on issues like this on a daily basis – give us a call!

One of the ways in which FusionReactor timeout protection is better is the ability to configure include (or exclude) lists of page URLs. This can be done with regular expressions. Let’s look at a couple of examples…

First of all we need to imagine our directory layout:

• wwwroot
  • public
    • my_first_page.cfm
    • my_second_page.cfm
    • my_third_page.cfm
    • my_fourth_page.cfm
  • scheduled_tasks
    • task1.cfm
    • task2.cfm
    • task3.cfm

In our examples, we’ll assume we’re looking at timeout protection and the crash protection settings are all configured for all URLs.

Example 1

Let’s look at exlcluding everything inside the “scheduled_tasks” folder. The first step is to ensure the restrictions are “enabled” and the behaviour mode is to “ignore matching requests”:

Next, we add a new regular expression RegEx for the exclusion:

You can see the RegEx matches on path only (unless you choose to include hostname). Additionally it optionally matches URL parameters and can exclude URL from statistics (eg average request time).

The RegEx’s are standard Java patterns. The online help describes some examples which are available from the FR interface from your server or on the FusionReactor website – http://www.fusion-reactor.com/fr/help/help.htm#creating_a_regular_expression_exclusion.htm

The Java (1.4.2) Pattern docs are available here: http://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/Pattern.html

Example 2

Exclude the public files “my_…._page.cfm” where “….” does not include the character “t”…

/public/my_[^t]*_page.cfm

[^t] = any character except “t”

* = any number of the previous matching group (ie [^t])

Hopefully this gets you on the way to configuring not only your crash protection but excluding your scheduled tasks from server level statistics so you get a better idea of the stats for public facing traffic. We can offer a lot of help and advice and have a wide range of consulting & development services available to assist you no matter the project size.

The standards layout that you should use strong cryptography and security protocols when transmitting card data over open, public networks (ie the Internet).

If you want to secure any data sent over HTTPS you need to make sure the protocols and ciphers used are secure. In practice, this means disabling SSLv2 and weak ciphers. This has to be done at the SSL endpoint – so if you’re using a load balancer, firewall or similar to terminate your SSL connections you’ll need to make the changes there.

We can offer advice and resell SSL terminating end-points. We also work with open-source SSl terminating solutions such as Pound ( http://www.apsis.ch/pound/ ).

How To Check

Use the SSLScan tool – http://sourceforge.net/projects/sslscan/

Use OpenSSL from the command line:

SSLv2

# openssl s_client -ssl2 -connect www.HOSTNAME.com:443

Weak ciphers

# openssl s_client -connect www.HOSTNAME.com:443 -cipher LOW:EXP

How to Fix

Apache 2.x:

SSLProtocol -ALL +SSLv3 +TLSv1

SSLCipherSuite ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM

Pound:

Ciphers "SSLv3:TLSv1:-LOW:-aNULL:-ADH:-EXP:-eNULL"

IIS:

(Unfortunately you have to edit the registry…)

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Server]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server]
"Enabled"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\DES 56/56]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\NULL]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 40/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 56/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 40/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 56/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 64/128]
"Enabled"=dword:0000000

You may also be interested to know that active CF community member Pete Freitag has created a commercial tool to help you make these changes in IIS – http://foundeo.com/products/iis-weak-ssl-ciphers/

Of course not everyone’s environment is straight-forward and often you’ll hit issues or knock-on effects – so get the experts in… we’re waiting!

Every application is different, there’s no golden rules but there are some simple steps you can take to improve performance, security & stability. For a typical web & application server machine let’s look at an example Linux setup. Here’s just the tip of the iceberg…

Tune TCP/IP Kernel Parameters

• Disable response to ICMP Echo broadcasts
  • net.ipv4.icmp_echo_ignore_broadcasts = 1

• Filter packets not meant for this network.
  • net.ipv4.conf.eth0.rp_filter=1
  • net.ipv4.conf.lo.rp_filter=1
  • net.ipv4.conf.default.rp_filter=1
  • net.ipv4.conf.all.rp_filter=1

• Set buffer TCP buffer MAX limits
  • net.core.rmem_max = 16777216
  • net.core.wmem_max = 16777216

• Increase Linux autotuning TCP buffer limits (min, default, and max number of bytes to use)
  • net.ipv4.tcp_rmem = 4096 10000000 16777216
  • net.ipv4.tcp_wmem = 4096 65536 16777216

• Disable IP spoofing
  • net.ipv4.conf.eth0.accept_source_route=0
  • net.ipv4.conf.lo.accept_source_route=0
  • net.ipv4.conf.default.accept_source_route=0
  • net.ipv4.conf.all.accept_source_route=0

• TIME_WAIT sockets for new connections can be reused. This helps on any server that receives many connections at the same time.
  • net.ipv4.tcp_tw_reuse=1
  • net.ipv4.tcp_fin_timeout=30

• Move keepalive from 2hrs to 30 min. (May want to tune this up or down)
  • net.ipv4.tcp_keepalive_time=1800

• Help protect from denial-of-service (syn-flood) attack:
  • net.ipv4.tcp_max_syn_backlog=4096

• Allow redirects from trusted sources (pick only trusted sources)
  • net.ipv4.conf.eth0.secure_redirects=1
  • net.ipv4.conf.lo.secure_redirects=1
  • net.ipv4.conf.default.secure_redirects=1
  • net.ipv4.conf.all.secure_redirects=1

• Don’t allow ICMP redirects (pick only un-trusted sources)
  • net.ipv4.conf.eth0.accept_redirects=0
  • net.ipv4.conf.lo.accept_redirects=0
  • net.ipv4.conf.default.accept_redirects=0
  • net.ipv4.conf.all.accept_redirects=0

• Do not send redirects (we’re not acting as a router)
  • net.ipv4.conf.eth0.send_redirects=0
  • net.ipv4.conf.lo.send_redirects=0
  • net.ipv4.conf.default.send_redirects=0
  • net.ipv4.conf.all.send_redirects=0

Tune User Security Limit Parameters

• Define “soft” & “hard” limits for max open file handles for all users
  • * soft nofile 20000
  • * hard nofile 20000

• Define “soft” & “hard” limits for max concurrent processes for all users
  • * soft nproc 8192
  • * hard nproc 8192

What’s Next?!

Making good progress? What about the network, firewall, loadbalancer, web server, JVM, application server, application code…. the list goes on – save yourself the headache, call the experts!

There are many possible causes of this – a common one being using “Registry” as the CLIENT variable backing store.

Have you seen this combined with “java.lang.OutOfMemoryError: PermGen space” errors in your logs?

Again, there are several causes for filling the PermGen space but one common one is too many templates for the allotted space. The PermGen space stores information about classes. Behind the scenes of ColdFusion each CFM translates to a Java class. This means that if you have many templates used by your server, you’ll have lots of classes and use a lot of PermGen space. Remember this class information gets stored in the PermGen for the life of the server and is never unloaded!

Careful not to get confused with the CF administrator setting “Maximum number of cached templates” which are cached in the Heap space.

So, how many is too many?

Well, I looked at an example with a very simple set of CFMs. I took 10,000 CFM templates containing the single line:

<cfset x = now() />

The mean average PermGen increase per template (after execution of course) was 2,677 bytes. This probably doesn’t sound like a lot but put this into practice on a live server with a real application and it only takes ~1,000-2,000 templates before you’re out of PermGen space and an unstable server.

Note: It’s not just CFMs that are Java classes behind the scenes, your CFC functions count too!