FusionReactor is a great monitoring tool and one of my favorite features is the ability to kill rogue requests. FusionReactor is sometimes limited by Java itself. Java has a known limitation that threads running “Native Code” can’t be killed (until the thread returns from the native code block).

What is Native Code?

Underlying all your ColdFusion goodness is Java, underlying the Java is the runtime environment typically implemented in C/C++ code. When you hit a code-block that must “go native” this is inside the C/C++ code typically waiting for an event to occur. When a thread is executing this native method the thread cannot be killed by the JVM.

What to look for?

Some of the most common examples where native code is used are:

• CFHTTP calls
• WebService calls
• JDBC Queries

What you’re looking for is “Native Method” in the stack trace of the thread. Let’s look at some concrete examples…

CFHTTP Calls

Example CF Code:

<cfhttp url="http://localhost/blogs/dont_stop_me_now/slow.cfm" />

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
HTTPClient.BufferedInputStream.fillBuff(BufferedInputStream.java:172)
HTTPClient.BufferedInputStream.read(BufferedInputStream.java:110)
HTTPClient.StreamDemultiplexor.read(StreamDemultiplexor.java:273)
HTTPClient.RespInputStream.read(RespInputStream.java:155)
HTTPClient.RespInputStream.read(RespInputStream.java:115)
HTTPClient.Response.readResponseHeaders(Response.java:1000)
HTTPClient.Response.getHeaders(Response.java:720)
HTTPClient.Response.getStatusCode(Response.java:259)
HTTPClient.RetryModule.responsePhase1Handler(RetryModule.java:83)
HTTPClient.HTTPResponse.handleResponse(HTTPResponse.java:761)
HTTPClient.HTTPResponse.getStatusCode(HTTPResponse.java:191)
coldfusion.tagext.net.HttpTag.connHelper(HttpTag.java:850)
coldfusion.tagext.net.HttpTag.doEndTag(HttpTag.java:1140)
cfslow_cfhttp2ecfm1758959420.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_cfhttp.cfm:1)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

WebService Calls

Example CF Code:

<cfset ws = createObject("webservice", "http://localhost/blogs/dont_stop_me_now/slow.cfc?wsdl") />
<cfset ws.goSlow() />

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:581)
org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:142)
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
org.apache.axis.client.Call.invokeEngine(Call.java:2765)
org.apache.axis.client.Call.invoke(Call.java:2748)
org.apache.axis.client.Call.invoke(Call.java:2424)
org.apache.axis.client.Call.invoke(Call.java:2347)
org.apache.axis.client.Call.invoke(Call.java:1804)
blogs.dont_stop_me_now.SlowCfcSoapBindingStub.goSlow(SlowCfcSoapBindingStub.java:157)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)[Native Method]
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
coldfusion.xml.rpc.ServiceProxy.invokeImpl(ServiceProxy.java:224)
coldfusion.xml.rpc.ServiceProxy.invoke(ServiceProxy.java:154)
coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2360)
cfslow_ws2ecfm1005604111.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_ws.cfm:2)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

JDBC Queries

Example CF Code:

<cfquery name="wait" datasource="test">
   SELECT 1 waitfor delay '000:00:10:000'
</cfquery>

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
macromedia.jdbc.sqlserver.SQLServerByteOrderedDataReader.makeMoreDataAvailable(null:???)
macromedia.jdbc.sqlserver.SQLServerByteOrderedDataReader.receive(null:???)
macromedia.jdbc.sqlserver.tds.TDSExecuteRequest.submitRequest(null:???)
macromedia.jdbc.sqlserver.tds.TDSRequest.execute(null:???)
macromedia.jdbc.sqlserver.SQLServerImplStatement.execute(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.commonExecute(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.executeInternal(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.execute(null:???)
coldfusion.server.j2ee.sql.JRunStatement.execute(JRunStatement.java:348)
coldfusion.sql.Executive.executeQuery(Executive.java:1229)
coldfusion.sql.Executive.executeQuery(Executive.java:1008)
coldfusion.sql.Executive.executeQuery(Executive.java:939)
coldfusion.sql.SqlImpl.execute(SqlImpl.java:341)
coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:843)
coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:533)
cfslow_db2ecfm445915345.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_db.cfm:1)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

Why!?

All these examples are in native methods for socket reading. Socket functions (both reading and writing) are the most commonly found native methods in stack traces.

What can I do?

Unfortunately, the only current work-around is to restart your server. But this is a Java limitation that even without FusionReactor you would still have the problem – FusionReactor is just giving you visibility. The real solution is to investigate the root cause of the problem and solve that – that’s where we come in! We’re experts in this field and working on issues like this on a daily basis – give us a call!

FusionReactor – the leading ColdFusion server monitoring software – has a nifty Crash Protection feature allowing it to abort requests that take too long. This works in a similar way to the ColdFusion server page timeouts but at a lower level allowing FusionReactor to stop requests under many more circumstances. FusionReactor also gives you the options not to abort the request, but just to email you a stack trace of the slow running page. There are several forms of crash protection FusionReactor provides but I won’t get in to those just now – take a look at the FusionReactor site for more information ( http://www.fusion-reactor.com/fr/featurefocus/crashprotection.cfm ).

One of the ways in which FusionReactor timeout protection is better is the ability to configure include (or exclude) lists of page URLs. This can be done with regular expressions. Let’s look at a couple of examples…

First of all we need to imagine our directory layout:

• wwwroot
  • public
    • my_first_page.cfm
    • my_second_page.cfm
    • my_third_page.cfm
    • my_fourth_page.cfm
  • scheduled_tasks
    • task1.cfm
    • task2.cfm
    • task3.cfm

In our examples, we’ll assume we’re looking at timeout protection and the crash protection settings are all configured for all URLs.

Example 1

Let’s look at exlcluding everything inside the “scheduled_tasks” folder. The first step is to ensure the restrictions are “enabled” and the behaviour mode is to “ignore matching requests”:

Next, we add a new regular expression RegEx for the exclusion:

You can see the RegEx matches on path only (unless you choose to include hostname). Additionally it optionally matches URL parameters and can exclude URL from statistics (eg average request time).

The RegEx’s are standard Java patterns. The online help describes some examples which are available from the FR interface from your server or on the FusionReactor website – http://www.fusion-reactor.com/fr/help/help.htm#creating_a_regular_expression_exclusion.htm

The Java (1.4.2) Pattern docs are available here: http://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/Pattern.html

Example 2

Exclude the public files “my_…._page.cfm” where “….” does not include the character “t”…

/public/my_[^t]*_page.cfm

[^t] = any character except “t”

* = any number of the previous matching group (ie [^t])

Hopefully this gets you on the way to configuring not only your crash protection but excluding your scheduled tasks from server level statistics so you get a better idea of the stats for public facing traffic. We can offer a lot of help and advice and have a wide range of consulting & development services available to assist you no matter the project size.

The PCI-DSS standards are designed to help protect card-holder data. The specific section we’re interested in is section 4.1 of the v1.2 revision of the standards which are available for download here: https://www.pcisecuritystandards.org/security_standards/pci_dss_download_agreement.html

The standards layout that you should use strong cryptography and security protocols when transmitting card data over open, public networks (ie the Internet).

If you want to secure any data sent over HTTPS you need to make sure the protocols and ciphers used are secure. In practice, this means disabling SSLv2 and weak ciphers. This has to be done at the SSL endpoint – so if you’re using a load balancer, firewall or similar to terminate your SSL connections you’ll need to make the changes there.

We can offer advice and resell SSL terminating end-points. We also work with open-source SSl terminating solutions such as Pound ( http://www.apsis.ch/pound/ ).

How To Check

Use the SSLScan tool – http://sourceforge.net/projects/sslscan/

Use OpenSSL from the command line:

SSLv2

# openssl s_client -ssl2 -connect www.HOSTNAME.com:443

Weak ciphers

# openssl s_client -connect www.HOSTNAME.com:443 -cipher LOW:EXP

How to Fix

Apache 2.x:

SSLProtocol -ALL +SSLv3 +TLSv1

SSLCipherSuite ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM

Pound:

Ciphers "SSLv3:TLSv1:-LOW:-aNULL:-ADH:-EXP:-eNULL"

IIS:

(Unfortunately you have to edit the registry…)

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Server]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server]
"Enabled"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\DES 56/56]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\NULL]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 40/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC2 56/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 40/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 56/128]
"Enabled"=dword:00000000
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\RC4 64/128]
"Enabled"=dword:0000000

You may also be interested to know that active CF community member Pete Freitag has created a commercial tool to help you make these changes in IIS – http://foundeo.com/products/iis-weak-ssl-ciphers/

Of course not everyone’s environment is straight-forward and often you’ll hit issues or knock-on effects – so get the experts in… we’re waiting!

A common consulting engagement for us is configuring a new system for optimal use. It’s not just the application server that’ll need configuring, don’t forget the application server relies on a lot of underlying infrastructure including the network and OS. Often this stage gets over-looked or you’ll (often incorrectly) assume your hosting provider will have performed these steps already.

Every application is different, there’s no golden rules but there are some simple steps you can take to improve performance, security & stability. For a typical web & application server machine let’s look at an example Linux setup. Here’s just the tip of the iceberg…

Tune TCP/IP Kernel Parameters

• Disable response to ICMP Echo broadcasts
  • net.ipv4.icmp_echo_ignore_broadcasts = 1

• Filter packets not meant for this network.
  • net.ipv4.conf.eth0.rp_filter=1
  • net.ipv4.conf.lo.rp_filter=1
  • net.ipv4.conf.default.rp_filter=1
  • net.ipv4.conf.all.rp_filter=1

• Set buffer TCP buffer MAX limits
  • net.core.rmem_max = 16777216
  • net.core.wmem_max = 16777216

• Increase Linux autotuning TCP buffer limits (min, default, and max number of bytes to use)
  • net.ipv4.tcp_rmem = 4096 10000000 16777216
  • net.ipv4.tcp_wmem = 4096 65536 16777216

• Disable IP spoofing
  • net.ipv4.conf.eth0.accept_source_route=0
  • net.ipv4.conf.lo.accept_source_route=0
  • net.ipv4.conf.default.accept_source_route=0
  • net.ipv4.conf.all.accept_source_route=0

• TIME_WAIT sockets for new connections can be reused. This helps on any server that receives many connections at the same time.
  • net.ipv4.tcp_tw_reuse=1
  • net.ipv4.tcp_fin_timeout=30

• Move keepalive from 2hrs to 30 min. (May want to tune this up or down)
  • net.ipv4.tcp_keepalive_time=1800

• Help protect from denial-of-service (syn-flood) attack:
  • net.ipv4.tcp_max_syn_backlog=4096

• Allow redirects from trusted sources (pick only trusted sources)
  • net.ipv4.conf.eth0.secure_redirects=1
  • net.ipv4.conf.lo.secure_redirects=1
  • net.ipv4.conf.default.secure_redirects=1
  • net.ipv4.conf.all.secure_redirects=1

• Don’t allow ICMP redirects (pick only un-trusted sources)
  • net.ipv4.conf.eth0.accept_redirects=0
  • net.ipv4.conf.lo.accept_redirects=0
  • net.ipv4.conf.default.accept_redirects=0
  • net.ipv4.conf.all.accept_redirects=0

• Do not send redirects (we’re not acting as a router)
  • net.ipv4.conf.eth0.send_redirects=0
  • net.ipv4.conf.lo.send_redirects=0
  • net.ipv4.conf.default.send_redirects=0
  • net.ipv4.conf.all.send_redirects=0

Tune User Security Limit Parameters

• Define “soft” & “hard” limits for max open file handles for all users
  • * soft nofile 20000
  • * hard nofile 20000

• Define “soft” & “hard” limits for max concurrent processes for all users
  • * soft nproc 8192
  • * hard nproc 8192

What’s Next?!

Making good progress? What about the network, firewall, loadbalancer, web server, JVM, application server, application code…. the list goes on – save yourself the headache, call the experts!

Have you noticed requests stop processing and your CPU usage is high?

There are many possible causes of this – a common one being using “Registry” as the CLIENT variable backing store.

Have you seen this combined with “java.lang.OutOfMemoryError: PermGen space” errors in your logs?

Again, there are several causes for filling the PermGen space but one common one is too many templates for the allotted space. The PermGen space stores information about classes. Behind the scenes of ColdFusion each CFM translates to a Java class. This means that if you have many templates used by your server, you’ll have lots of classes and use a lot of PermGen space. Remember this class information gets stored in the PermGen for the life of the server and is never unloaded!

Careful not to get confused with the CF administrator setting “Maximum number of cached templates” which are cached in the Heap space.

So, how many is too many?

Well, I looked at an example with a very simple set of CFMs. I took 10,000 CFM templates containing the single line:

<cfset x = now() />

The mean average PermGen increase per template was 2,677 bytes. This probably doesn’t sound like a lot but put this into practice on a live server with a real application and it only takes ~1,000-2,000 templates before you’re out of PermGen space and an unstable server.

Note: It’s not just CFMs that are Java classes behind the scenes, your CFC functions count too!