USA: (978) 496-9990
Germany: +49 7031 221461
UK: +44 207 193 1212

JRun

CVE-2010-4476 – ColdFusion / Java hangs when converting 2.2250738585072012e-308 (or 2.2250738585072011e-308)

This JVM bug seems to be getting some high-level attention in the IT press so I thought I’d lay out the issue where CF is concerned:

History

The bug is in the JVM (it has been since ~2001) and so ColdFusion running on Sun JVMs are affected.
Someone out there has obviously made the link between the same issue happening in PHP and brought this issue to light again ( http://bugs.php.net/bug.php?id=53632 ). There’s a Java related discussion happening here: http://www.exploringbinary.com/java-hangs-when-converting-2-2250738585072012e-308/

How to reproduce

To have the bug show, you must call the parseDouble() method of the java.lang.Double class. There are several ways this can happen. Many people are discussing this as a vulnerability that can be executed at the HTTP header level like so:

Accept-Language: en-us;q=2.2250738585072012e-308

However, this requires a call to HttpServletRequest’s getLocale() method, something that isn’t done trivially on a JRun4, CF 9.0.1 instance (even when calling the ColdFusion function “getLocale()”). Thus, to show this problem, you must do something like…

#GetPageContext().getRequest().getLocale()#

… within your ColdFusion page.

From our experience, a more likely attack could be performed with code like this:

<cfparam name="URL.pageNum" default="1" />
<cfparam name="URL.itemsPerPage" default="10" />
<cfquery name="qProducts" datasource="mysql_dsn">
    SELECT * FROM products
    LIMIT #((URL.pageNum-1) * URL.itemsPerPage) + 1# , #URL.pageNum * URL.itemsPerPage#
</cfquery>

The problem here is “URL.pageNum-1“. This calculation causes a call to parseDouble() behind the scenes which means that if the page were called with “page_name.cfm?pageNum=2.2250738585072012e-308” then the thread would hang in an infinite loop.

What doesn’t show the issue?

Note that in this example, “URL.itemsPerPage” could also cause the issue because it is used in the multiplication calculation. If the variable were not used in any calculations but only output, it would not show the issue. This example does NOT show the problem:

<cfset x = 2.2250738585072012e-308 />
<cfoutput>#x#</cfoutput>

What can you do?

Short term

If you have FusionReactor installed and configured with CrashProtection enabled and configured, the threads can be automatically killed by FusionReactor, saving your server from almost certain failure. To do this, enable Crash Protection and configure a “Request Timeout” value and set it to use the “Abort and Notify” strategy. This will cause requests taking longer than this time to quit – even if they are stuck in the infinite loop bug as in this scenario.

For those of you who are wondering, this is NOT the same as the ColdFusion timeout mechanism and so the ColdFusion page timeout alone will not help you in this scenario.

It’s good practice to have FusionReactor installed and Crash Protection enabled because it can save you from a lot of these issues without you needing to do anything.

Long term

I’m sure Oracle/Sun will offer a new update in due course. However, you can also download the “Java SE Floating Point Updater Tool”:
Download: http://www.oracle.com/technetwork/java/javase/downloads/index.html#fpupdater
Read Me: http://www.oracle.com/technetwork/java/javase/fpupdater-tool-readme-305936.html

Further Help

If you’re in need of help updating your JVM and/or patching it then we can offer assistance in this area from as little as $800. The FusionReactor product is available from as little as $249 and contains a wealth of other features – the majority of which are not covered by the ColdFusion Server Monitor – http://www.fusion-reactor.com/fr/ for more information.

Notes

This article refers to JRun4, CF9 installations. The issue is apparent on a wide variety of Java platforms (we offer consulting for most Java environments) and is more prevalent on Tomcat installations (which includes JBoss).

References

Official security alert (CVE-2010-4476): http://www.oracle.com/technetwork/topics/security/alert-cve-2010-4476-305811.html

Why you should track issues… forever!

An interesting issue cropped up today which involved taking a trip back in time to look at our old issue tracking software. Back in 2004 we were using a Windows client/server based system which had a MS SQL back-end. From day 1 employees are trained to track track and track again. Emails, calls, suggestions – it all gets tracked. Added to that, most new employees – even senior engineers – get started on support duties. This gets them familiar with their environments, customers and our software – plus familiarity with tracking everything. This means we have a LOT of knowledge built up in our ticket-base.

One of our products at this time (used by some of the worlds largest corporations – HP, Philips, etc) was called “Tornado” – this evolved into a product called ShareDox ( http://www.sharedox.com ). The product is a knowledge management solution built on ColdFusion technology.

At the time there were no monitoring tools and thus this led to the now leading monitoring solution – FusionReactor ( http://www.fusion-reactor.com ). So, during the upgrade process to CFMX 6.1 with one of our customers, we started seeing huge CPU usage, hanging threads and various other nasties. At the time this was all quite serious and a major thorn in our sides. Eventually (I believe with some help from our CTO but don’t quote me on that – this was several years ago!) this got resolved and all was calm again. Until today!

Fast-forward 6 years, a move to JIRA and some changes in our business focus – ie our old tracking system is just a distant memory. I started working with a client and getting very distinct feelings of de-ja-vu… CFMX6.1, MS-SQL DB, high CPU, multiple failures per day, hanging threads – essentially a whole heap of stability issues.

Now, I knew I’d seen this before. I knew it was something to do with problematic DB drivers. What I couldn’t remember is how to solve the issue. As you can imagine from a company that’s been doing ColdFusion consulting for over ten years that brought back a lot of issues. A little bit of date filtering and some extra keywords and… result #1 of #5 “ColdFusion MX 6.1: Updated DataDirect drivers for 100% CPU utilization and other issues”.

Of course there are many questions here… why is the customer still on CFMX6.1 amongst others. However, my real point is that tracking is your saviour. In a consulting company like ours we’re truly able to assist more rapidly to a huge variety of issues because generally – we’ve seen it all before. It’s very common for us to have identified, resolved and documented the problem you’re having. This allows us to give you the best value for money on your consulting investment.

If knowledge is power and the key to results – you want us on your team! For all your consulting needs – whether issues from 2004 or today, we can help, contact us now.

Hello from CFUnited 2010

A big hello from CFUnited!

Myself (David Stockton) and my colleagues (Darren Pywell & David Tattersall) are all at CFUnited this week. We have a FREE copy of FusionAnalytics to give away so don’t be shy – come and speak to us for a chance to win!!

Plus, ask nicely and we’ve got FREE goodies for everyone we speak to!

We’re in the vendor area between Adobe & Railo – look for the ShareDox, FusionReactor, FusionAnalytics & Intergral banners… plus this good looking guy:

See you there!

ColdFusion Update 1 – 9.0.1

ColdFusion 9.0.1 has been released for about a week now. I’m sure everyone has done due diligence in their test environments, run a full test-suite and deployed to production right?

http://www.adobe.com/support/coldfusion/downloads_updates.html#cf9

Well it’s not always that simple is it? So if you want some professional help from the experts then call the experts – we’re ready & waiting to help.

JVM PermGen memory usage with many CFM templates

Have you noticed requests stop processing and your CPU usage is high?

There are many possible causes of this – a common one being using “Registry” as the CLIENT variable backing store.

Have you seen this combined with “java.lang.OutOfMemoryError: PermGen space” errors in your logs?

Again, there are several causes for filling the PermGen space but one common one is too many templates for the allotted space. The PermGen space stores information about classes. Behind the scenes of ColdFusion each CFM translates to a Java class. This means that if you have many templates used by your server, you’ll have lots of classes and use a lot of PermGen space. Remember this class information gets stored in the PermGen for the life of the server and is never unloaded!

Careful not to get confused with the CF administrator setting “Maximum number of cached templates” which are cached in the Heap space.

So, how many is too many?

Well, I looked at an example with a very simple set of CFMs. I took 10,000 CFM templates containing the single line:

<cfset x = now() />

The mean average PermGen increase per template (after execution of course) was 2,677 bytes. This probably doesn’t sound like a lot but put this into practice on a live server with a real application and it only takes ~1,000-2,000 templates before you’re out of PermGen space and an unstable server.

Note: It’s not just CFMs that are Java classes behind the scenes, your CFC functions count too!