What this is telling you is that you were able to connect but your client and the remote server were unable to find a mutually acceptable encryption algorithm to communicate over so further communication was aborted.

An easy way to tell if this is your problem as a Java/JVM client is to use the Qualys SSL analyzer and see whether it reports a Java client can connect. In my case, I saw the following failures when I entered the SSL endpoint I wanted to access:

If you happen to be a ColdFusion user, you can add -Djavax.net.debug=ssl,handshake,verbose to your jvm.config, restart and make your HTTPS request and see the full list of ciphers being attempted and the ultimate failure in your log file. My logs with the handshake_failure can be seen in this gist for comparison.

Fix with Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files

The fix is easy to implement but not easy to find. I experienced this with Oracle’s JVM and not the OpenJDK so other JVM implementations may behave differently. In the case of Oracle’s JVM, strong encryption is disabled out of the box for export reasons so the JVM can not use an encryption algorithm stronger than 128-bit. Many web servers are now disabling 128-bit SSL in favor of 256-bit and 384-bit encryption algorithms so the fix is to turn these on in your JVM.

1. Download the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files from Oracle (this link is for Java 1.8, Google for alternative JVM versions): http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
2. Extract the two JAR files into your jre/lib/security folder overwriting the US_export_policy.jar and local_policy.jar.
3. Restart the JVM

You should now be able to connect to the remote server. If you’re still unable to connect, try enabling debugging and see which ciphers are attempted/ignored. The remote server may have an exotic configuration that requires you to contact them but the unlimited strength jurisdiction policy files should fix handshake_failure issues.

<cffunction name="extendRequestTimeoutDuringInit" output="false" access="public" returntype="any">

  <cfthread action="run" name="delayRequestTimeoutDuringInit">
    <cfset thread.adminapi = createObject("component", "cfide.adminapi.administrator") />
    <cfset thread.adminapi.login('mysecretpassword') />
    <cfset thread.runtime = createObject("component", "cfide.adminapi.runtime") />
    <cfset thread.timeout = thread.runtime.getRuntimeProperty("TimeoutRequestTimeLimit") />
    <cflog file="application" text="Extending RequestTimeout to #2*thread.timeout# seconds" />
    <cfset thread.runtime.setRuntimeProperty("TimeoutRequestTimeLimit", 2*thread.timeout) />
    <cfset sleep(2 * thread.timeout * 1000) />
    <cfset thread.runtime.setRuntimeProperty("TimeoutRequestTimeLimit", thread.timeout) />
    <cflog file="application" text="Restored RequestTimeout to #thread.timeout# seconds" />
  </cfthread>

</cffunction>

I call this in Server.cfc when my instance starts up but you could also call it from any reinit routine in OnRequestStart or OnApplicationStart. Previously when we pushed code, the first user request would kick off the initialization process while other requests queued. Many of those first requests would exceed our page timeout setting of 60 seconds. Running the above function uses a background thread to double the timeout and later reset it once the application has initialized so users no longer see timeout/error screens.

It turns out there is a bug from JDK 1.7 u75 onwards that is present at least through JDK 1.8 u60 which prevents Jstat, when run as root, from introspecting JREs running as other users.

To solve, simply run jstat as the user who owns the JRE process. You can pass the username to sudo when running jstat like:

sudo -u nobody jstat -gcutil -t <pid> 1s 30

And you’ll be treated to your output again. In my case, I have two server instances running so I like to see their output side by side at the console. Here’s my script for doing that using /usr/bin/paste and some console redirection trickery:

The username can be figured out from the process but this was a quick fix for the above bug.

docker-compose up -d

I’m simultaneously moving from a Windows machine to a Mac and I’ve been really missing the console logging for watching application activity and errors. It’s easy when installed locally to tail the coldfusion-out.log file but once containerized you don’t have that luxury (without using docker exec to log in to the running container which is a bad practice). The issue is on Linux, ColdFusion has no option to log to stdout like on Windows. As a result, the docker logs command can’t see the output from ColdFusion once the container is running.

It turns out there is a very simple and very elegant way to force ColdFusion, or any app which wants to daemonize and write to log files, to write to stdout. From this ServerFault thread referencing the Nginx Dockerfile, you can symlink the coldfusion-out.log file to /dev/stdout:

ln -sf /dev/stdout /opt/coldfusion10/cfusion/logs/coldfusion-out.log

My Dockerfile for ColdFusion has these two lines:

# redirect logs so `docker logs -f <container>` works
RUN ln -sf /dev/stdout /opt/coldfusion10/cfusion/logs/coldfusion-out.log
RUN ln -sf /dev/stderr /opt/coldfusion10/cfusion/logs/coldfusion-err.log

Now the logs can be collected and observed in a centralized location using Docker best practices.

I’ll be speaking at dev.Objective() this year in Minneapolis on a hybrid business/programming topic titled, “Building Multi-Tenant Software-As-A-Service (Saas) Applications” This will be a comprehensive review covering the hurdles I’ve crossed growing MotorsportReg.com from a one-man nights-and-weekends operation to a team that has processed more than $100,000,000 in event entry fees. The talk will cover:

• Database design and tenancy model trade-offs
• Third party services and designing for resiliency
• E-commerce strategies for cards on file, split payments, multiple currencies and marketplace-style apps
• Implications of going global including i18n, l10n, currency and domiciling
• Cross-customer security and the impact of banking and legal requirements such as PCI DSS, IRS and Know-Your-Customer (KYC)
• Code strategies for implementing customer or domain-specific functionality without a “kitchen sink” user experience problem

Developers who are interested in SaaS apps or have the entrepreneurial itch themselves will walk away with a roadmap of the issues a SaaS app must solve and strategies to achieve success. I’ll be speaking immediately after the keynote Wednesday morning, May 13, at 10:15am.

This is my second time attending dev.Objective() and I’m returning because I got a ton of value from it. I hope you’ll consider attending if you’re not already signed up as it will be a positive boost for your creativity and skills. Registration is still available for $999.

Connect with me on twitter ahead of the conference to keep in touch at @ghidinelli. You can connect with other speakers and attendees on Lanyrd

This year I’ve been working to do more A/B testing. A/B tests are mathematically-backed competitions between two or more options which are scored by the actions of the users. They may also be the single highest ROI tactic for software companies. These tests could be anything from a simple color change of a button (shocking, to what extent that can matter) all the way to a completely different web page. I was first exposed to the technique back in 2001 at Yahoo! when I was helping redesign their search results but it wasn’t until the last 18 months I ran the first A/B test on MotorsportReg.com.

Which brings me to how I grew one of my mailing lists by 53% in 45 days. In the history of this 9-year old list, the last two months would make a hockey stick look like a rolling foothill. This is like the face of a glacier where it meets into the ocean: a vertical line that reads like an error.

The technique is perhaps too simple to be valuable by itself: I put the signup form where many more people would see it in their routine use of our platform. Previously, we had a few CTAs in our transactional emails and on our event calendar but the actual signup took place on a mailing lists screen under “My Account”. Few people went there looking to add more email to their inbox so list growth was steady but slow. Now, when users create a new account or reconfirm their details, they see checkboxes at the bottom of their personal information letting them opt-in to the lists. Thousands of people see these screens every month and because the lists are valuable, thousands of them are signing on.

“It was just a test”

The obvious question is why didn’t we do this sooner? The answer is because I wasn’t sure if these transactional flows were appropriate places to insert a list signup. I had concerns that it might negatively impact registration conversion. So I hemmed. And I hawed. And I periodically looked at the issue in our bug tracker but never quite got over the “ewwwww” feeling to put it in place.

That’s the beauty of A/B testing. A feature request in our bug tracker is a stone being added to a wall with mortar: it becomes a seemingly permanent piece of your architecture, of your user interface and of your responsibility. But an A/B test, why, that’s just a test! We have made no commitments to keeping it around. We’re not even sure we like it! Implementation only took a few hours so we could rip it out at any time for any reason and respond, “It was just a test.” Simply doing it, rather than talking about it, is rule #2 of A/B testing fight club.

Since it was just a test, we hoped for a nice bump to justify keeping it around. We were blown away. At the current rate, we will reach 25,000 subscribers (333% growth) by the end of 2015. And because we have a large percentage of first-time participants come through our service, we should see a permanently scaled growth rate. As an additional bonus, this list drives participation for our events so we’re creating a positive feedback loop for our event organizers and our bottom line too.

All because of a feature I was skeptical of “implementing” but happy to “test”.

Remember: making the call is making progress. Doing is better than planning. Execution is more valuable than ideas. Look at your to-do list and find one or two things you’ve been putting off. What can you do to “test” it (whatever that might mean in your case) to move it ahead and gain confidence in your choice?

The challenge is when I want to update or synchronize the listings later. The search makes it difficult to identify an event or venue that I’ve created in the system vs. one that someone else has created (which may have different attributes) or determine if I’m selecting the matching item at all (do we trust a string comparison on the name?) The venues in this API do support an arbitrary properties collection where I could stash an ID from our system but you can’t search based on those properties which results in searching, then looping to see if a property matches our original request and, if not, creating a new venue.

It doesn’t have to be that tedious. As a rule of thumb, any API which allows the creation of a logical object (people, events, places, etc) where the record of authority may originate from another system should accommodate a foreign identifier as a standard property and allow searching against the identifier.

I’m sure there are APIs which follow this practice but it should be a standard. Simplify my integration efforts so that I can easily send and synchronize data with you. Keep us loosely coupled. It’s one more field on your end or it’s a ton of code on my end that results in a more fragile relationship. Don’t make me maintain my own database table of ID mappings.

Update May 2015 – Since writing this, I’ve found another dozen instances where this would be valuable and plan to bake this into the second version of our API. However, I’ve come to believe that webhooks are an alternative 80% solution. Rather than polling and synchronizing, receiving a webhook when events of interest take place saves me the typical flow of grabbing a set of records, looping over them and comparing them locally for changes. The best of both worlds would be for the webhook payload to include my foreign key.

As a Software-as-a-Service for online event registration, however, we face the dilemma of separating out potential customers of our SaaS app and their customers who come to our site to also participate in the marketplace. For every event organizer who visits our site and might be interested in using our system, hundreds of event attendees come to get registered. Including the Hubspot tracking code on our home page makes our traffic numbers look great but has the negative side effect of dropping our lead and conversion ratios well under 1%. Ignore for a moment that this is demoralizing, it also makes reports difficult to use. Picture comparing the success of a landing page with 0.05% conversions and another with 0.07%. They both look effectively equivalent because of the scale but if those numbers instead read 50% and 70%, you would not only be impressed but recognize the latter is performing 40% better!

One option here would be to simply remove the tracking code from the home page and that would fix the statistic and volume issues. But we would lose important referring data. While Google is no longer giving us much in the way of search keywords, we still can see referring URLs when people surf to MotorsportReg.com. If we removed the tracking code from the home page and cookie’d them when they hit the About Us page, we would lose that original traffic source.

So, the challenge began to find a way to cookie the people who were potential leads and do it on the home page so we could capture referrer data. I asked our sales guy, our onboarding rep and customer support. Nobody had any idea how to segment the traffic to weed out the event attendees from skewing our data.

But I had an idea, and it turns out it works pretty well.

What I want to know, specifically, is when someone views our “How it works” or another marketing/sales page as that indicates interest in our SaaS product. I started by tagging all links in the top and bottom navigation that link to our marketing and sales pages with a CSS class like:

< a href="/about" class="hs">How it Works< /a>

On those marketing pages, we include the Hubspot tracking code on every view. But for the home page, I wrapped it inside of a jQuery click handler:

$(document).ready(function()
{
// dynamically load hubspot code only if someone clicks a marketing-related link on the home page
$('.hs').click(function(e)
{
var anchor = $(this), h;
h = anchor.attr('href');
if (typeof _hsq === "undefined")
{
e.preventDefault();
$.getScript('//js.hubspot.com/analytics/'+(Math.ceil(new Date()/300000)*300000)+'/{your-hubspot-id}.js', function(){
setTimeout(function(){window.location = h},1000);
});
}
});
});

The code basically says: if someone clicks a link with the class “hs”, grab the link they clicked, load the Javascript file from Hubspot’s servers, wait 1 second, and then continue to the original link. In the 1 second while we sleep, the code should load from Hubspot’s servers which results in the user getting cookie’d appropriately and Hubspot can still access the referring URLs, search keywords, etc.

This approach is not only limited to Hubspot. It would work equally well with any Javascript-based tracking code like Google Analytics or others if you had some reason to exclude certain users from participating under certain circumstances.

It’s not perfect – the one second pause is too long in some cases, not long enough in others, and adds a delay in all cases which can increase page abandonment. However, we’ve been using this in production now for about 3 months and it’s been working well. Lead volume is slightly up despite the brief delay and we see the full browsing history for contacts in Hubspot. Our lead and traffic numbers are now properly scaled so we can make sense of the data we’re seeing.

If you have multiple audiences viewing your site and only want to apply tracking to some of them, this technique is worth taking a look at.

To encourage you to leverage rather than rebuild, I’ve moved the source to GitHub where you can fork and pull:

https://github.com/ghidinelli/cfpayment

There’s also a Google group where I provide support if you want to use a gateway we don’t yet include. I’ll help you understand how to use the library and extend it so you don’t go through the pain of learning the hard way how transactions can fail in production.

ava.lang.StackOverflowError at org.apache.xerces.dom.ParentNode.item(Unknown Source) at
net.sf.saxon.dom.NodeWrapper$ChildEnumeration.skipFollowingTextNodes(NodeWrapper.java:1166) at
net.sf.saxon.dom.NodeWrapper$ChildEnumeration.next(NodeWrapper.java:1194) at
net.sf.saxon.tree.util.Navigator$EmptyTextFilter.next(Navigator.java:918) at
net.sf.saxon.tree.util.Navigator$DescendantEnumeration.advance(Navigator.java:1052) at
net.sf.saxon.tree.util.Navigator$DescendantEnumeration.advance(Navigator.java:1111) at
net.sf.saxon.tree.util.Navigator$BaseEnumeration.next(Navigator.java:949) at
net.sf.saxon.tree.util.Navigator$DescendantEnumeration.advance(Navigator.java:1043) at
net.sf.saxon.tree.util.Navigator$BaseEnumeration.next(Navigator.java:949) at
... (endless repeat)

Something is stuck in a loop but because the stack trace fills up, you can’t see any of the lines of code being executed just before the ka-blammo. So, what to do?

Brute Force Debugging

My solution was to swipe a perl script to run commands at very short intervals and dump jstack output into a series of text files. Low-tech, but effective. This is the timer.pl script:

#!/usr/bin/perl

use strict;
use warnings;
use Time::HiRes qw/time sleep/;

sub launch {
    return if fork;
    exec @_;
    die "Couldn't exec";
}

$SIG{CHLD} = 'IGNORE';

my $interval = shift;
my $start = time();
while (1) {
    launch(@ARGV);
    $start += $interval;
    sleep $start - time();
}

And a separate shell script, dump.sh, so the redirect would create a new file every time it was run. 9892 is just the process id of the ColdFusion instance:

#!/bin/sh
sudo -u nobody /usr/java/latest/bin/jstack 9892 > t_$(date '+%Y%m%d%H%M%S_%N')

I tried running it every second, but that wasn’t enough to catch the stack overflow in the act. I upped it to every 500ms, 100ms and finally 50ms until it caught the offending code. Here’s the command in action:

perl timer.pl 0.05 ./dump.sh 

That says run the dump.sh every 50ms which will output to files formatted like t_20130402171521_009840232 using the current date and time.

Results

Running every 50ms, it doesn’t take long to fill up a directory so I started the script on the command line and in my web browser, triggered the offending code. Here’s what my directory looks like:

 52 t_20130402171519_959011475
 52 t_20130402171520_009648175
 52 t_20130402171520_060316767
 52 t_20130402171520_110450371
 52 t_20130402171520_160301392
 52 t_20130402171520_210357359
 52 t_20130402171520_260432922
 52 t_20130402171520_309171913
 52 t_20130402171520_360033048
 52 t_20130402171520_409978373
 52 t_20130402171520_459903958
 52 t_20130402171520_510488116
 52 t_20130402171520_558777256
 60 t_20130402171520_610244244
 72 t_20130402171520_659494494
136 t_20130402171520_709058569
136 t_20130402171520_760286651
136 t_20130402171520_810351161
136 t_20130402171520_860382457
136 t_20130402171520_910136459
136 t_20130402171520_960370954
136 t_20130402171521_009840232
 72 t_20130402171521_059986004
136 t_20130402171521_110340195
136 t_20130402171521_160277982
136 t_20130402171521_209090149
 84 t_20130402171521_260090368
136 t_20130402171521_310428721
136 t_20130402171521_367027484
136 t_20130402171521_410303418
136 t_20130402171521_460314216
136 t_20130402171521_509053355
136 t_20130402171521_560120477
136 t_20130402171521_610417764
128 t_20130402171521_677647036
136 t_20130402171521_712667159
136 t_20130402171521_758884870
136 t_20130402171521_810562747
136 t_20130402171521_860425517
...

That’s kilobytes in the first column, filename in the second. You can see the steady state while I’m going to push the button in the web browser and then very quickly the stack traces get larger as code is executing. What I wasn’t getting until I stepped down to 50ms iterations were the intermediate 72kb and 84kb stack trace which actually showed the offending lines between the stack overflows churning.

While this hasn’t solved the problem yet, at least I know where to go looking (and it’s at you, XmlSearch()) to continue the troubleshooting process.

Conclusion

This is a pretty hacky solution but if you’re this far into the deep end of the “what do I do?” pool, it’s a useful technique. A line debugger would be far better but I can’t replicate this in my test environment. So, forget the “right” way to do it and let’s just get some useful information so we can make progress.