Killing Rogue Requests – Going native, don’t stop me now!

FusionReactor is a great monitoring tool and one of my favorite features is the ability to kill rogue requests. FusionReactor is sometimes limited by Java itself. Java has a known limitation that threads running “Native Code” can’t be killed (until the thread returns from the native code block).

What is Native Code?

Underlying all your ColdFusion goodness is Java, underlying the Java is the runtime environment typically implemented in C/C++ code. When you hit a code-block that must “go native” this is inside the C/C++ code typically waiting for an event to occur. When a thread is executing this native method the thread cannot be killed by the JVM.

What to look for?

Some of the most common examples where native code is used are:

  • CFHTTP calls
  • WebService calls
  • JDBC Queries

What you’re looking for is “Native Method” in the stack trace of the thread. Let’s look at some concrete examples…

CFHTTP Calls

Example CF Code:

<cfhttp url="http://localhost/blogs/dont_stop_me_now/slow.cfm" />

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
HTTPClient.BufferedInputStream.fillBuff(BufferedInputStream.java:172)
HTTPClient.BufferedInputStream.read(BufferedInputStream.java:110)
HTTPClient.StreamDemultiplexor.read(StreamDemultiplexor.java:273)
HTTPClient.RespInputStream.read(RespInputStream.java:155)
HTTPClient.RespInputStream.read(RespInputStream.java:115)
HTTPClient.Response.readResponseHeaders(Response.java:1000)
HTTPClient.Response.getHeaders(Response.java:720)
HTTPClient.Response.getStatusCode(Response.java:259)
HTTPClient.RetryModule.responsePhase1Handler(RetryModule.java:83)
HTTPClient.HTTPResponse.handleResponse(HTTPResponse.java:761)
HTTPClient.HTTPResponse.getStatusCode(HTTPResponse.java:191)
coldfusion.tagext.net.HttpTag.connHelper(HttpTag.java:850)
coldfusion.tagext.net.HttpTag.doEndTag(HttpTag.java:1140)
cfslow_cfhttp2ecfm1758959420.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_cfhttp.cfm:1)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

WebService Calls

Example CF Code:

<cfset ws = createObject("webservice", "http://localhost/blogs/dont_stop_me_now/slow.cfc?wsdl") />
<cfset ws.goSlow() />

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:581)
org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:142)
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
org.apache.axis.client.Call.invokeEngine(Call.java:2765)
org.apache.axis.client.Call.invoke(Call.java:2748)
org.apache.axis.client.Call.invoke(Call.java:2424)
org.apache.axis.client.Call.invoke(Call.java:2347)
org.apache.axis.client.Call.invoke(Call.java:1804)
blogs.dont_stop_me_now.SlowCfcSoapBindingStub.goSlow(SlowCfcSoapBindingStub.java:157)
sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)[Native Method]
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
coldfusion.xml.rpc.ServiceProxy.invokeImpl(ServiceProxy.java:224)
coldfusion.xml.rpc.ServiceProxy.invoke(ServiceProxy.java:154)
coldfusion.runtime.CfJspPage._invoke(CfJspPage.java:2360)
cfslow_ws2ecfm1005604111.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_ws.cfm:2)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

JDBC Queries

Example CF Code:

<cfquery name="wait" datasource="test">
   SELECT 1 waitfor delay '000:00:10:000'
</cfquery>

Example Java Stack Trace (available from FusionReactor):

java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)[Native Method]
java.net.SocketInputStream.read(SocketInputStream.java:129)
macromedia.jdbc.sqlserver.SQLServerByteOrderedDataReader.makeMoreDataAvailable(null:???)
macromedia.jdbc.sqlserver.SQLServerByteOrderedDataReader.receive(null:???)
macromedia.jdbc.sqlserver.tds.TDSExecuteRequest.submitRequest(null:???)
macromedia.jdbc.sqlserver.tds.TDSRequest.execute(null:???)
macromedia.jdbc.sqlserver.SQLServerImplStatement.execute(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.commonExecute(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.executeInternal(null:???)
macromedia.jdbc.sqlserverbase.BaseStatement.execute(null:???)
coldfusion.server.j2ee.sql.JRunStatement.execute(JRunStatement.java:348)
coldfusion.sql.Executive.executeQuery(Executive.java:1229)
coldfusion.sql.Executive.executeQuery(Executive.java:1008)
coldfusion.sql.Executive.executeQuery(Executive.java:939)
coldfusion.sql.SqlImpl.execute(SqlImpl.java:341)
coldfusion.tagext.sql.QueryTag.executeQuery(QueryTag.java:843)
coldfusion.tagext.sql.QueryTag.doEndTag(QueryTag.java:533)
cfslow_db2ecfm445915345.runPage(C:\inetpub\wwwroot\blogs\dont_stop_me_now\slow_db.cfm:1)
coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231)

Why!?

All these examples are in native methods for socket reading. Socket functions (both reading and writing) are the most commonly found native methods in stack traces.

What can I do?

Unfortunately, the only current work-around is to restart your server. But this is a Java limitation that even without FusionReactor you would still have the problem – FusionReactor is just giving you visibility. The real solution is to investigate the root cause of the problem and solve that – that’s where we come in! We’re experts in this field and working on issues like this on a daily basis – give us a call!

Menu