USA: (978) 496-9990
Germany: +49 7031 221461
UK: +44 207 193 1212

Killing Rogue Requests – Going native, don’t stop me now!

FusionReactor is a great monitoring tool and one of my favorite features is the ability to kill rogue requests. FusionReactor is sometimes limited by Java itself. Java has a known limitation that threads running “Native Code” can’t be killed (until the thread returns from the native code block).

What is Native Code?

Underlying all your ColdFusion goodness is Java, underlying the Java is the runtime environment typically implemented in C/C++ code. When you hit a code-block that must “go native” this is inside the C/C++ code typically waiting for an event to occur. When a thread is executing this native method the thread cannot be killed by the JVM.

What to look for?

Some of the most common examples where native code is used are:

  • CFHTTP calls
  • WebService calls
  • JDBC Queries

What you’re looking for is “Native Method” in the stack trace of the thread. Let’s look at some concrete examples…

CFHTTP Calls

Example CF Code:

<cfhttp url="http://localhost/blogs/dont_stop_me_now/slow.cfm" />

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
HTTPClient.BufferedInputStream.fillBuff(BufferedInputStream.java:172)
HTTPClient.BufferedInputStream.read(BufferedInputStream.java:110)
HTTPClient.StreamDemultiplexor.read(StreamDemultiplexor.java:273)
HTTPClient.RespInputStream.read(RespInputStream.java:155)
HTTPClient.RespInputStream.read(RespInputStream.java:115)
HTTPClient.Response.readResponseHeaders(Response.java:1000)
HTTPClient.Response.getHeaders(Response.java:720)
HTTPClient.Response.getStatusCode(Response.java:259)
HTTPClient.RetryModule.responsePhase1Handler(RetryModule.java:83)
HTTPClient.HTTPResponse.handleResponse(HTTPResponse.java:761)
HTTPClient.HTTPResponse.getStatusCode(HTTPResponse.java:191)
coldfusion.tagext.net.HttpTag.connHelper(HttpTag.java:850)
coldfusion.tagext.net.HttpTag.doEndTag(HttpTag.java:1140)
cfslow_cfhttp2ecfm1758959420.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_cfhttp.cfm:1)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

WebService Calls

Example CF Code:

<cfset ws = createObject("webservice", "http://localhost/blogs/dont_stop_me_now/slow.cfc?wsdl") />
<cfset ws.goSlow() />

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:581)
org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:142)
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
org.apache.axis.client.Call.invokeEngine(Call.java:2765)
org.apache.axis.client.Call.invoke(Call.java:2748)
org.apache.axis.client.Call.invoke(Call.java:2424)
org.apache.axis.client.Call.invoke(Call.java:2347)
org.apache.axis.client.Call.invoke(Call.java:1804)
blogs.dont_stop_me_now.SlowCfcSoapBindingStub.goSlow(SlowCfcSoapBindingStub.java:157)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)[Native Method]
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
coldfusion.xml.rpc.ServiceProxy.invokeImpl(ServiceProxy.java:224)
coldfusion.xml.rpc.ServiceProxy.invoke(ServiceProxy.java:154)
coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2360)
cfslow_ws2ecfm1005604111.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_ws.cfm:2)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

JDBC Queries

Example CF Code:

<cfquery name="wait" datasource="test">
   SELECT 1 waitfor delay '000:00:10:000'
</cfquery>

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
macromedia.jdbc.sqlserver.SQLServerByteOrderedDataReader.makeMoreDataAvailable(null:???)
macromedia.jdbc.sqlserver.SQLServerByteOrderedDataReader.receive(null:???)
macromedia.jdbc.sqlserver.tds.TDSExecuteRequest.submitRequest(null:???)
macromedia.jdbc.sqlserver.tds.TDSRequest.execute(null:???)
macromedia.jdbc.sqlserver.SQLServerImplStatement.execute(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.commonExecute(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.executeInternal(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.execute(null:???)
coldfusion.server.j2ee.sql.JRunStatement.execute(JRunStatement.java:348)
coldfusion.sql.Executive.executeQuery(Executive.java:1229)
coldfusion.sql.Executive.executeQuery(Executive.java:1008)
coldfusion.sql.Executive.executeQuery(Executive.java:939)
coldfusion.sql.SqlImpl.execute(SqlImpl.java:341)
coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:843)
coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:533)
cfslow_db2ecfm445915345.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_db.cfm:1)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

Why!?

All these examples are in native methods for socket reading. Socket functions (both reading and writing) are the most commonly found native methods in stack traces.

What can I do?

Unfortunately, the only current work-around is to restart your server. But this is a Java limitation that even without FusionReactor you would still have the problem – FusionReactor is just giving you visibility. The real solution is to investigate the root cause of the problem and solve that – that’s where we come in! We’re experts in this field and working on issues like this on a daily basis – give us a call!

FusionReactor Crash Protection – Regular Expressions “HowTo”

FusionReactor – the leading ColdFusion server monitoring software – has a nifty Crash Protection feature allowing it to abort requests that take too long. This works in a similar way to the ColdFusion server page timeouts but at a lower level allowing FusionReactor to stop requests under many more circumstances. FusionReactor also gives you the options not to abort the request, but just to email you a stack trace of the slow running page. There are several forms of crash protection FusionReactor provides but I won’t get in to those just now – take a look at the FusionReactor site for more information ( http://www.fusion-reactor.com/fr/featurefocus/crashprotection.cfm ).

One of the ways in which FusionReactor timeout protection is better is the ability to configure include (or exclude) lists of page URLs. This can be done with regular expressions. Let’s look at a couple of examples…

First of all we need to imagine our directory layout:

  • wwwroot
    • public
      • my_first_page.cfm
      • my_second_page.cfm
      • my_third_page.cfm
      • my_fourth_page.cfm
    • scheduled_tasks
      • task1.cfm
      • task2.cfm
      • task3.cfm

In our examples, we’ll assume we’re looking at timeout protection and the crash protection settings are all configured for all URLs.

Example 1

Let’s look at exlcluding everything inside the “scheduled_tasks” folder. The first step is to ensure the restrictions are “enabled” and the behaviour mode is to “ignore matching requests”:

Next, we add a new regular expression RegEx for the exclusion:

You can see the RegEx matches on path only (unless you choose to include hostname). Additionally it optionally matches URL parameters and can exclude URL from statistics (eg average request time).

The RegEx’s are standard Java patterns. The online help describes some examples which are available from the FR interface from your server or on the FusionReactor website – http://www.fusion-reactor.com/fr/help/help.htm#creating_a_regular_expression_exclusion.htm

The Java (1.4.2) Pattern docs are available here: http://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/Pattern.html

Example 2

Exclude the public files “my_…._page.cfm” where “….” does not include the character “t”…

/public/my_[^t]*_page.cfm

[^t] = any character except “t”

* = any number of the previous matching group (ie [^t])

Hopefully this gets you on the way to configuring not only your crash protection but excluding your scheduled tasks from server level statistics so you get a better idea of the stats for public facing traffic. We can offer a lot of help and advice and have a wide range of consulting & development services available to assist you no matter the project size.