clicky.com/blog

Local search now supports 'pretty' searches and multiple parameters

Wed, 15 May 2013 16:04:56 -0800

By popular demand, local searches now support multiple parameters, as well as path based pretty searches. You can mix the two types as well. To use multiple of either one, just separate with commas.

For path based searches, what you want to enter is everything that goes between the domain name and the search term. For example, if someone searches for apples on your web site and they end up here:

http://yourwebsite.com/search/apples

Then you would enter in /search/ into the box. Without quotes of course.

For good measure, here is an example of a comma separated list of both types, pretending your site has both q and s as search parameters, and also does path based search with the search term appearing after /search/, as shown above:

/search/,q,s

Tracking tel: URLs, custom heatmap objects, and other tracking code updates

Wed, 17 Apr 2013 22:34:47 -0800

Note 1: DO NOT PANIC. You don't need to change anything with the code installed on your site. We've simply made some changes to the way the code works, adding a couple of features and fixing a few bugs.

Note 2: Other than the tel: tracking, most of this only applies to advanced users.

New features

Tracking tel: URLs - This has been requested here and there over the years, but as Skype et al become more ubiquitous, these URLs are on more and more pages. So we just added support for automatically tracking them. Like mailto: links, these will show up in your outbound link report. You don't need to do jack diddly, it should just work.
Custom heatmap object tracking - Our heatmap code by default listens for clicks that bubble up to the document.body element. The default for all events is to bubble up, but there are plenty of events that don't do this, which means those clicks weren't being captured. We ourselves have been affected by this too, mainly by our Javascript menus.

So now there is a new clicky_custom property, heatmap_objects. With this you can specify custom elements, by tag name, ID, or class. It can be a string if you just need to specify one thing (most likely), or it can be an array of strings if you need to specify more than one. Using this we can track clicks on these elements. Which reminds me, I forgot to update our own Javascript code to track our menus! Mental note.

You should ONLY use this for events that don't bubble up, or you will experience oddness.
clicky.goal() changes - This likely does not affect you but if you use the clicky.goal() javascript method at all, you may want to read on.

When we released heatmaps, we added a new event queue system for logging some items in batches: heatmap data, javascript events, and javascript goals. The reasoning behind this change was to reduce bandwidth for heatmaps, and increase accuracy for events and goals. The accuracy part being that if you sent a hit to clicky.log() or clicky.goal() when someone clicked on a link that would result in a new page being loaded, chances were good that it would not be logged because the page would be unloaded from the browser before the logging request went through.

So the queue system was made to store events and goals in a cookie, which is then processed every 5 seconds. So if the person is just sitting on the same page still, the queue will be processed shortly and send that event/goal to us. But if instead a new page is loaded, the cookie is still there holding the event/goal that wasn't logged on the last page, and can be processed immediately on the new page view (which we do before processing the new page view itself, to ensure things are in the correct chronological order).

ANYWAYS... there were some customers who were using clicky.goal to log goals when visitors were leaving their site. The queue would intercept these goals though, resulting in a snowball's chance in hell of the goal ever being logged.

SO... we added a new parameter to clicky.goal() called no_queue, which will tell our code to skip the queue and just log the goal immediately. Check the docs for more.

This doesn't affect many of you, but if it does, the back story I've written above is probably worth a read.
New method to check if a site ID has been init()'d - for customers using multiple tracking codes on a single site/page. This was a specific request from one customer, but we realized our code itself wasn't even doing this sanity check, so if you had the SAME code on your site multiple times, there were some minor bugs that resulted from this.

If for some reason you think this applies to you, the new method is clicky.site_id_exists(123), which returns true or false indicating whether this site ID has been passed through the clicky.init() function yet. Note: 123 is an example site ID. Use a real one.

Bug fixes for sites using multiple tracking codes

In addition to the last item above about loading the same site ID multiple times resulting in oddities (and which is now fixed), we've made another change to the way the init process works.

There are a number of things that happen when a site ID is init()'d, but it turns out most of those things only needed to happen once, even if you had multiple site IDs on a aingle page. However, our code was executing this entire init process for every site ID on a page, which resulted in bugs such as:

clicky_custom.goals and clicky_custom.split only working with the first site ID that was init()'d.
The automatic pause that we inject for tracking downloads and outbound links was being called once for every site ID, rather than once per click (which is all that's needed)
When loading heatmaps by clicking the heatmap link from clicky.com, the heatmap would sometimes load twice (making it extra dark).

There were a few other much more minor bugs, but those were the ones that were really irritating. So now what happens is we split the setup procedure into a different method, and wait 100 milliseconds before calling it (just once), giving a chance for all site IDs to be passed into the init process first. And the actual init() method now just puts each site ID into an array which we loop through when any request to log data is called.

Coming soon

Been requested a number of times and something we will definitely add in the coming months. That being when you set custom visitor data with clicky_custom.session (or utm_custom), we will store this data in a cookie so the data will be applied to all future visits by this person, so even if they're not logged in, they'll still be tagged as they were last logged in / tagged visit.

We'll probably only do this with a few specific keys though, since people use clicky_custom.session for all kinds of crazy purposes, many of which can be session specific. But we'll probably do something like, only do it for keys like username, name, email, and a few others.

Just something to watch out for. We think this will be a nice addition when we add it.

Local / internal search support!

Tue, 16 Apr 2013 16:00:06 -0800

Local searches (searches performed with your site's own search engine) has been one of the biggest feature requests we've had over the years, so we're happy to finally support it!

First, you need to tell us what the search parameter is that your site uses. Common ones would be q or search. Examples:

http://yoursite.com/search?q=lollipops
http://yoursite.com/search?search=care+bear+stare

You can set this in your site preferences:

You will then start seeing data in the new local searches report:

You can click on any of the searches to filter down to the visitors who performed said searches:

They will also show up in Spy:

As well as the actions log (both globally, and when viewing a session):

The action log can also be filtered down to just show local searches:

And that about covers everything!

Tracking Youtube videos no longer requires PhD

Tue, 09 Apr 2013 16:06:04 -0800

Recently we were inspired by this post detailing how to automatically track (with Google Analytics) all Youtube videos embedded on a page, with zero work required other than including a single Javascript file (or two if you don't have jQuery).

Our old method for tracking Youtube was really ugly, requiring a good bit of custom code for every single video you wanted to track. We wanted it to work more like what we read above.

So, now it does! The old method still works for those of you who already have it deployed, but the new method is great because it works with the default iframe embed code that Youtube gives you, and it requires pretty much no work on your end.

Head on over to the video analytics docs to see what you need to do to get it working (scroll down, click 'youtube').

Coming soon

We've got a couple new features we hope to release this week. One is local search support, probably our biggest feature request of all time. Another is tracking clicks on tel: URLs. Another is the ability to click on any graph to view/segment visitors based on what you clicked. Last, Monitage (uptime monitoring) is being finalized, which also means we'll have up to 1 minute monitoring available and the ability to setup more than 3 checks. Monitage won't launch this week, but soon thereafter.

API throttling

Fri, 08 Feb 2013 16:11:42 -0800

The analytics API has been a complete free for all in almost 6 years of existence. This has rarely been an issue, save for once or twice a year maybe, we'd have to ask someone to please relax themselves.

But recently it's become a serious ongoing problem. We've had at least 3 different people in the last few weeks all doing utterly massive exports of data, causing some of the database servers to lag quite badly (up to almost 2 hours in the most severe case).

When a server is lagging it affects thousands of customers. We can't have this anymore, so today we have implemented some API throttling functionality and it is live now.

Throttling will only apply for visitors-list, actions-list, and segmentation requests, as those are by far the biggest drain on resources. All other requests are unaffected.

Here is how it works:

Maximum of 1 simultaneous request per IP address per site ID at any point in time. Part of the issue recently has been people doing automated simultaneous requests for exporting data, in one case over 20 requests at the same time for the same site ID, from the same IP. This will no longer work. You will receive an API error.

Maximum of 500 results per request (down from 5,000), maximum date range of 3 days. This one is pretty strict and we will likely raise these limits, but we have to get API usage under control immediately. We will be monitoring things and plan to raise the limits as things calm down. UPDATE: things have been stable so we've raised the limits to 1000 results and 7 days.

To repeat, these changes only apply for visitors-list, actions-list, and segmentation requests. No other types of requests are affected by anything mentioned here.

We know this is pretty lame, but it's in the interest of keeping the service as close to real time as possible for all customers and that's important. Hope you understand.

-------
Update, Monday Feb 11:

Since we made this change on a Friday, and Friday-Sunday is a complete trickle compared to the rest of the week, it wasn't until today (Monday) that we could really see the effect of this change.

Good news: All servers are keeping up with real time no problem now. A few are 1 minute behind right now, which sometimes happens when caches expire on the servers and have to regenerate, and usually they're back up to normal shortly thereafter.

We'll likely raise the single day restriction soon, first to 3 days, and if things keep up, then probably 7 days. I don't know if we'll ever let it go beyond 7 days again though. As well, the limit of 500 per request will probably be raised to 1000, but again I'm not sure if we'll ever let it go beyond that.

----
Update, Tuesday Feb 12:

Things have been stable so we've raised the date range limit to 3 days now. We'll see how things go from here.

----
Update, Friday Feb 15:

Things continue to be stable so we've raise the date range limit to 7 days, and the result set limit to 1,000 items. We've also changed the one request per IP address limit, so that it's one request per IP address, per site ID.

Monitage: Uptime monitoring beta

Tue, 22 Jan 2013 18:23:16 -0800

Many Clicky users have asked us about site monitoring. We are happy that this is top-of-mind for some of you because its importance should not be understated. Often we will receive emails from Clicky users asking why there was a dramatic lag in visitors tracked or a complete drop-off altogether. While there can be many explanations, it is not uncommon that the site itself went down unbeknownst to the site owner. Standard web analytics will not be able to tell you this, but site uptime monitoring and alerts will.

With this in mind, we are excited to announce a closed beta in partnership with Monitage, a newly-developed site uptime monitoring service by Roxr Software (that's us). We have integrated Monitage into Clicky to give you a bigger picture of the health and activity of your web sites.

Monitage monitors web sites from five locations around the world (three in the US, one in Paris, one in Japan) and only declares a downtime event if a majority of its servers agree on it. This prevents network hiccups on the monitoring end from sending false alarms.

Pro Plus users and above receive access to the Monitage closed beta. When we officially launch, you will have the ability to create up to 30 checks per site with intervals as fast as 1 minute, but during testing we want to keep resource usage within a reasonable range. So for the time being, we are limiting to 3 checks per site with a max interval of 5 minutes. We expect to officially launch within 4 weeks, at which point Monitage will also be available as a standalone service.

To access Monitage, go to your site dashboard and click the Uptime tab. You can create checks for HTTP, HTTPS, SSH, FTP, IMAP, IMAPS, and ICMP (ping). We've also created a dashboard module.

You can also access uptime stats from the API. Check the API docs and search for uptime. type=uptime will give you the current status of all of your checks for a site. type=uptime-list will give you a chronological list of all downtime events for your site for the date range requested.

Last, we added uptime stats as an option in email reports as well.

When Monitage officially launches, we will determine what intervals and types of tests will be included with Clicky Pro Plus plans and above.

We are asking that you test Monitage, and let us know your thoughts, what you like, don't like, and want to see. As Monitage is in its infancy, we want your feedback to help mature it into a stalwart companion to Clicky.

Note to white label customers: Monitage will be added as an option to white label service when it officially launches, but for now it is only available to Clicky users.

----
Update, Feb 4: Just pushed an update that integrates uptime monitoring into the Big Screen report. Also added web hooks to the setup page. Enter a URL and we will POST a JSON object (documented in the setup page) to that URL for events.

Also we've been getting reports of false positives since launch, we are pushing some updates later today that should fix it entirely, or at least close to.

'GetClicky' no more!

Wed, 19 Dec 2012 00:34:27 -0800

You may have noticed that we're now live on clicky.com! Yay!

For 6 years, our brand has always been Clicky. Excluding our domain name, nowhere on our site has the brand GetClicky ever been mentioned, other than some testimonials that we didn't edit. However, because of the old domain name, the majority people have always referred to us as GetClicky. It didn't help that our Twitter handle was also @getclicky, but we wanted it to match our domain name (bad idea). It's always kind of driven us insane, but it's been an interesting lesson. If you can afford the right domain name, do yourself a favor and buy it as soon as possible. We paid a pretty freaking penny for clicky.com, but the second it was in our possession was just pure unadulterated joy.

We've had the @clicky Twitter handle from the beginning, but it was just a placeholder. We've now swapped @getclicky with @clicky so all of our followers and tweet history remains in place. Fun fact, Twitter does not offer an official method to do this - you have to actually relinquish your brand for a brief few seconds while you swap things around, which is extremely nerve racking to say the least. So @clicky became @clickyx, @getclicky became @clicky, and last @clickyx became @getclicky. Scary 20 seconds or so. Anyways, please note: If you tweet at us in the future, use @clicky, not @getclicky! But if you were already following us you will keep getting our tweets without having to do anything.

Any page viewed on getclicky.com is now redirected to clicky.com, with a little note at the top about updating your bookmarks. That note will go away in a few weeks, but for now we want as many people to notice the change as possible.

You do NOT need to update your tracking code, because getclicky.com isn't going anywhere. Your existing tracking code, along with any integrations you have with the API, widgets, or whatever via getclicky.com, all will continue to work. Going forward, getclicky.com will only be seen in the tracking code (we are using it indefinitely for tracking). And we'll continue to use it for email, for now at least, since that's a nightmare to change. So... clicky.com = web site domain, getclicky.com = tracking domain, email, and indefinite backwards compatibility.

This was not as easy as just acquiring clicky.com and pointing it to a new IP address. A service of our size and complexity, there's a LOT of things to deal with when changing your domain name. My checklist is ridiculous in length and I hope to go into it more in a later blog post.

There may be a few bugs lying around... we'll squash them tomorrow as they come to our attention.

On-site analytics update

Fri, 16 Nov 2012 14:00:46 -0800

Just pushed a much needed tracking update for sub-user accounts, so now they can use on-site analytics, and their traffic will now also be automatically ignored.

Previously this was not possible because of the way our tracking servers were caching information about each site in memcached. After banging heads on keyboards for a while, we finally figured out a way to do it (it's more complicated than it sounds). So, now sub-users will have their traffic ignored automatically, and they will see the on-site analytics widget.

One caveat - the master account's preference for disabling on-site analytics on an account wide basis will override a sub-user's preference.

Making this change also addresses a bug that popped up after the four-week heatmap trial ended a few days ago. All sub-user accounts are created as Pro, since Pro has always had access to every feature, and a Pro account was required to have sub-user accounts. But now that heatmaps require Pro Plus or higher, heatmap tracking broke for some of you if you had created another sub-user account and assigned them admin access to a site in your account. Their account features were overriding yours on the tracking servers for the sites they had admin access to. So this update fixes that. But we went ahead anyways and made sub-users account types mirror the master accounts - any time the master account type changes, the sub-users are updated accordingly, to address any future updates that might cause this scenario to come up again.

Update on heatmaps

Mon, 12 Nov 2012 14:28:37 -0800

Heatmaps launched four weeks ago. As we said in that post, everyone would have it for four weeks, but then we would be requiring a Pro Platinum account to continue usage after that point. The reason for this is because the extra bandwidth and storage requirements for this data are significant, which has proven to be the case as we've monitored resource consumption since launch.

After listening to your feedback, we've decided to add a new plan (Pro Plus) with pricing in between Pro and Pro Platinum. It is the exact same as standard Pro, except that it includes heatmaps.

As of now, these changes are live. So if you are the standard Pro plan, you will need to upgrade to at least Pro Plus in order to keep heatmap tracking going forward.

Note to white label account holders: We've had a few of you ask about heatmaps for white label, we are including this feature with all account levels.

Please do not place third party javascript in your HTML head.

Wed, 17 Oct 2012 16:25:24 -0800

There were a few bugs with our new heatmaps for people who were putting our tracking code in their HTML head tag, and there were an alarmingly large number of you doing so. Those bugs are fixed now, but that's beside the point of this little post.

Web browsers download page elements in the order that they are listed in your HTML - CSS, javascript, and images (mostly). If the domain that any of those are hosted on is offline or going slow, a web browser will hang for up to 60 seconds while it waits for the item to load, before moving on to the next item. (Asynchronous javascript is an exception but that's still not very common - although we do offer it an option - but again, that's not the point. And besides, if you're loading it asynchronously, you're not depending on it being available for immediate execution, so what would be the point of putting it in your HTML head?).

If you have third party javascript in your HTML head and that third party server goes offline, your web site is effectively dead because nothing else is going to load for up to 60 seconds. When I visit sites that hang like that, I immediately close them as I'm sure 99% of people do.

If the javascript was instead in the footer of your web site, the web browser would still hang trying to download the item, but most of your site would already be loaded in the visitor's web browser. In other words, your site would still be (mostly) usable while the engineers for the third party service go into panic mode.

Our CDN never really goes offline but there are obviously thousands of services out there that ask you to put their javascript on your web site. Any third party javascript code that's not critical to your web site working properly should never go in the HTML head. Tracking code, ad code, widgets, social plugins, etc - these are all non-critical and have no business up there. Help make the internet a better place and always put that code at the bottom of your web site, k?

The only valid exception here is if it's hosted by a major provider such as Google, and you rely on the functionality of that code as the page loads (which we do for Google's map API, but that's it). That is the only exception.

So... if you've got our code in your HTML head, please move it instead to the bottom of your HTML, right before the closing /body tag, as has always been recommended on the page where you grab your tracking code from.

Bonus pro-tip: For first party elements, your CSS file should always be the very first item in your HTML head. That way your site will always look proper as soon as possible, beacuse this will be the very first thing the browser downloads after the HTML itself.

Heatmaps!

Sun, 14 Oct 2012 20:51:46 -0800

We've got two great new features launching today, after nearly 4 months of development: heatmaps, and on-site analytics.

Here's an example of a heatmap report, after the new tracking code has been live on our site for a couple of hours:

(Note: Everyone's user homepage is unique so some of the hot spots look weird here when conglomerated on a global scale. That's the nature of dynamic pages).

IMPORTANT! Heatmaps won't start logging anything until you specify your site's width and layout in your site preferences. You can see where to do that in this screenshot. And as with on-site analytics, you must have the newest tracking code in your browser, so clear your cache!

Over the years we've had a lot of requests for this feature and we're psyched to finally offer it to you. But once we started on it, the idea for on-site analytics was born soon thereafter, making this otherwise fairly-simple feature spiral out of control (in a good way). Hence nearly 4 months of development and testing!

Most services, to view heatmaps, you go to their site (e.g. getclicky.com) and go to your heatmap report, click a button for the page you want to view heatmaps for, and then your site is opened in an iframe, and then a heatmap overlay is loaded on top. That's unideal for a lot of reasons, but particularly the iframe (ugh). Loading a site in an iframe introduces a lot of potential problems, and a lot of sites have frame busters that would totally break this feature. So we took a different approach.

Ok, we won't deny you the boring way, going to your reports and clicking the appropriate link in the Content report:

What we do differently is that your page loads directly in a new window with a special parameter in the URL fragment, that our tracking code recognizes and automatically loads a heatmap dynamically on your site - without a frame, and without requiring being logged in to Clicky (i.e. you can share the URL). So that's good.

But we thought it would be so terrific if you could just be on your web site and say I want a heatmap of this page, now and just click a button on that page and see it immediately. So that's what we did, and hence made it part of the on-site analytics experience, as you can see to the right.

This was still not enough for us. Even though very few of our real-time competitors offer heatmaps, we didn't want to just make a me too feature. We believe most of the features we are releasing with our heatmaps are unlike anything that any other service offers.

First of all, beyond per page heatmaps, we also do per session. For every single page view of every session we log for your site, we log a unique heatmap for that visitor. You can view session heatmaps by viewing a session, and next to each action, on the right-hand side, will be a heatmap icon to click if we have any heatmap data for that page/session combo. Click the icon to view all of their clicks on any given page:

Second, what would be analytics without segmentation? You can view global heatmaps (all clicks) for any page, or you segment them by a number of criteria. For example, only view clicks on a page for people who completed a specific goal, or only people who arrived via a specific search or campaign or referring domain or type of referrer (e.g. advertising), or any/every version of every split test you're running.

Here's an example of our homepage for users completing the new user goal. Unsurprising, the majority of the clicks are on the register now button:

The order of items in these reports is a bit different than what you see on our web site. We take the top 24 items ordered by popularity, e.g. the top 24 goals completed today, then re-order them alphabetically. This is because we think you'll generally want to see the most popular segments but you'll also want to find them quickly - so instead of showing them in order of popularity, we grab the top ones and re-order them alphabetically so you can quickly narrow down what you're looking for in the interface.

To generate segmented heatmaps, we needed to add a new filter type to quickly find visitors with heatmap data attached. So we made this a general filter as well. In the main visitors report on Clicky, when you click add a filter, you will see a new option callled heatmaps. Click that to filter down your visitors to just people with heatmap data attached. This also works with the analytics API, by specifying heatmap=1 in combination with type=visitors or type=segmentation requests. (We also added a filter for online now, since we needed to be able to internally filter visitors who were online now in order to display in the widget. Via the API, use online=1).

When viewing your visitors report, anyone with heatmap data attached will have a new icon next to them:

Not every visitor will have heatmap data though... We anticipate this type of data to take up a lot of space, so for now we only do it for a random 50% of your visitors. We will likely adjust this in the future as we see what kind of impact it has on our resources. For example, it is quite likely we'll increase the sampling rate for low traffic sites, maybe even do all visitors, and for higher traffic sites, do less than 50%. But for now, just doing 50% was an easy safety measure and we'll analyze resource impact as time goes on.

Most heatmaps use server-side image generation to overlay on your site, which takes up a lot of resources on the service providing the heatmap - and is also quite slow. Earlier this year we discovered the excellent open-source heatmaps.js by Patrick Wied, and we knew that was just what we needed. This is a javascript library that generates the heatmap on the fly, using your web browser's resources, so it's fast and efficient. There are lots of features this library has that aren't yet documented, some of which we wanted to use (animated heatmaps for example) - but we didn't want to take the time to figure them out ourselves. We hope this library matures in the future and we'll take advantage of it when it does. But for now, for what we need, it does the job extremely well.

A difficult decision

This will be the first feature we offer that is not included in the standard Pro plan. The Pro plan has always had every feature we offer, so this was hard to do. But, the resources needed for this feature are quite significant, both in terms of bandwidth and storage, so we can't justify the standard Pro pricing for this feature.

We do want all of you to experience it first hand though, so for the first 4 weeks, all Pro or higher accounts will have this feature. After 4 weeks, a Pro Platinum or Custom plan will be required to keep the heatmap feature. On-site analytics will be standard with any Pro or higher account, though.

Data retention

As we said above, we anticipate this taking up a lot of storage space, so we built the database structure around monthly chunks that are auto-purged every 30 days. Initially, daily/session data will only be guaranteed for 30 days, but be available up to 60 days, and monthly data will only be guaranteed for 2 months, but available up to 3 months. It's hard to anticipate the exact impact this will have on our resources so we may (and hope to) expand this limit in the future, but for now, this is all we can guarantee.

Internet Explorer, may I count the ways in which I love thee?

Prior to IE8, IE does not have native JSON support, and even then, only when your site runs in standards mode. IE causing developers headaches is as sure as the sun rising in the east, regardless, the point is that IE users will not have heatmap data logged unless they are on at least version 8 and your site renders in standards mode. Good luck!

On-site analytics!

Sun, 14 Oct 2012 20:47:08 -0800

We've got two great new features launching today, after nearly 4 months of development: heatmaps, and on-site analytics.

On-site analytics is a new feature that embeds a widget on the bottom right corner of your web site automatically to view information about your visitors who are on your web site right now. Don't worry, only you (the site owner) can see it - but if you want to disable it, you can do so on your user preferences page.

This feature is available to all Pro or higher customers. Need to upgrade?

Before we get into the details, a few important notes for this to work properly:
- This requires the latest version of our tracking code, so clear your cache!
- Third party cookies must be enabled (or at least, have Clicky white listed)
- You need to check the remember me box when you login to Clicky, so that a cookie is set to remember who you are.

There are three components to on-site analytics:

Visitors on this page

The on page number represents how many visitors are currently viewing the page that you are currently viewing. Clicking on it shows the exact same type of reports as the global on site report, except it's limited to visitors currently viewing the same page that you are.

The on site and on page numbers will automatically update once per minute for 5 minutes, then once every 5 minutes for an hour, while you are idle on a single page on your web site. But after an hour, the updating will stop.

Heatmaps

If you have enabled heatmap tracking in your site preferences, and there is heatmap data in the last 7 days for the page you are currently viewing, you will see the heatmap icon on the right hand side (large colorful pixels). Click that to view a heatmap report for the page you're viewing for the current date, along with segmentation options for the heatmaps.

There's a lot to talk about with heatmaps, so we wrote a separate post for that. Read about heatmaps here.

Other notes/features

Clicking on the Clicky Web Analytics link will open a new tab/window on getclicky.com with your site's dashboard.

On-site analytics requires jQuery. We automatically check if your site already has it, and if not, we side-load it from our CDN - after which we call jQuery.noConflict() to ensure we don't interfere with any other libraries you may have on your site that use the $() shortcut. In order for all features to work, we do require at least version 1.6, which is almost 30 months old - if you're on a version older than that, please upgrade ;)

To authenticate your access to on-site analytics, we had to cache your user cookie on the tracking servers. While we were at it, we went ahead and enabled ignoring your traffic to your own web site automatically, without having to set a IP filter / filter cookie. If you need to test something on your own web site and see it in Clicky, simply logout of Clicky or use a different web browser. We know this will be a PITA for some of you, but the majority of users want to ignore their own traffic so this makes sense for us to implement as a convenience feature.

Calls to clicky.log() are now stored in a cookie queue (unless you have disabled cookies), to help ensure these calls aren't missed. Previously we recommended calling clicky.pause() manually after any clicky.log() or clicky.goal() event that resulted in a new page view, as otherwise the call would almost never be logged. This new queue system is designed to fix this issue. As long as the visitor's next page view is still on your own web site, the cookie will be seen by our tracking code and processed as an event and sent to our tracking servers upon the next page loading on your web site. The queue is processed every 5 seconds when idle on a page, and immediately upon a fresh page view.

Hope you enjoy!

History of Spy (also, ZeroMQ rocks)

Thu, 20 Sep 2012 14:13:18 -0800

This post is written by Alexander, hacker extraordinaire, who rewrote the Spy backend from scratch. Next time you're in Portland, kindly buy him a beer.

Clicky is almost 6 years old and hence was one of the first real-time web analytics platforms. Spy is an important part of that offering. Spy allows you to glimpse important information about active visitors on your website, as that information flows out of your visitors' browsers and into our system.

Spy has been a part of Clicky since the very beginning and has always been one of the most popular features. The name and functionality were both heavily influenced by Digg Spy. Sean, our lead developer, says that when he saw Digg Spy circa 2005, he thought to himself how sweet would it be if you had that kind of real time data stream, but for your own web site? That was in fact one of his motivations to create Clicky in the first place.

Since 2006, we've grown by leaps and bounds, and many parts of our service have had to absorb the shocks of scale. Recent tweets about our major data center move and infrastructure changes confirm this, but much is afoot at a deeper level than that meeting the eye.

Spy has gone through multiple complete rewrites to get to where it is today. We thought some of you might find it interesting.

Initial implementation

The first version of Spy was drafted in a few days. Back then, all infrastructure components for Clicky were located on a single machine (yikes).

Incoming tracking data was simply copied to local storage in a special format, and retrieved anew when users viewed their Spy page. It was extremely simple, as most things are at small scale, but it worked.

Spy, evolved

By mid-2008, we had moved to a multi-server model: one web server, one tracking server, and multiple database servers. But the one tracking server was no longer cutting it. We needed to load balance the tracking, which meant Spy needed to change to support that.

When load balancing was completed, any incoming tracking hit was sent to a random tracking server. This meant that each dedicated tracking server came to store a subset of Spy data. Spy, in turn, had to open multiple sockets per request to extract and aggregate viewing data. Given X tracking servers, Spy had to open X sockets, and perform X^2 joins and sorts before emitting the usage data to the user's web browser.

Initially the Spy data files were served via Apache from all of the tracking servers, but that tied up resources that were needed for tracking. Soon after this setup, our first employee (no longer with us) (Hi Andy!) wrote a tiny python script that ran as a daemon on each tracking server. It took over serving the Spy files for the PHP process on the web server that was requesting them. This helped free up precious HTTP processes for tracking, but there was still a lot of resources being wasted because of the storage method we were using.

Tracking servers had no mechanism for granularly controlling the memory consumed by its Spy data. Therefore, each tracking server had a cron job that would indiscriminately trim its respective Spy data. Sometimes, however, this mechanism would grow out of control, so trims happened randomly when users viewed their Spy page. These competing mechanisms and their implementations presented an opportunity for datum linearity issues to arise occasionally.

The number of websites we tracked continued rising, and we made several changes to this implementation to keep resource usage in check. Among them was the move to writing Spy data to shared memory (/dev/shm) instead of the hard drive. This helped a great deal initially, but as time wore on, it became clear that this was just not going to scale much further without a complete rethinking of Spy.

Trimming the fat

In the end, we decided to reimplement Spy's backend from the ground up. We devised a list of gripes that we had with Spy, and it looked like this:

*N tracking servers meant N sockets opened/closed on every request for data
*Data had to be further aggregated and then sorted on every request
*Full list of Spy data for a site was transmitted on every request, which consumed massive bandwidth (over 5MB/sec per tracking server on internal network) and required lots of post-processing by PHP

Our goals looked like this:

*Improve throughput
*Improve efficiency
*Reduce network traffic
*Reduce per-request and overall resource usage

Enter Spyy

The modern version of Spy, called Spyy to fit in with our internal naming conventions, is implemented in two custom daemons written in C. The first runs on each tracking server and dumbly forwards incoming Spy data to the second daemon, the Spy master. Data travels over a persistent socket established by ZeroMQ.

The Spy master daemon stores the information efficiently for each site, using a circular buffer whose size is determined and periodically reevaluated by a site's traffic rate. This removes the need to manually trim any data from a site as we've been doing for years. Once the buffer is filled for a site, old data is automatically purged as new data arrives. When we read data from the new daemon, it gives us only the data we need (basically, since timestamp X) instead of all data for site X, drastically reducing network bandwidth and post-processing needed to be done by PHP.

Like the previous version of Spy, all data is stored in RAM to ensure peak performance. Because of this, however, the data is still volatile. We have other mechanisms for permanently storing tracking data, and Spy data excludes much information from what is permanently stored. This means that when we implement new features or apply bugfixes to Spyy, it must be killed and started anew.

To mitigate this blip in the availability of data, we have implemented a rudimentary transfer mechanism between the existing Spy program and a new one coming online to take its place. This mechanism also uses ZeroMQ and, basically, drains datums from the existing process to the new process. At completion, the old process shuts itself down and the new process claims its occupied network interfaces.

Conspycuous benefits

We met our goals, and then some. The overall project took about 8 weeks.

Benchmarks at the start of the project demonstrated the old Spy implementation barely able to keep up at peak load, tapping out at about 5,000 reqs/sec. In comparison, the new implementation can handle upwards of 50,000 reqs/sec on the same hardware.

In the old architecture, read performance decreased as the number of tracking servers increased. In the new architecture, read performance is unaffected by the number of tracking servers, and is roughly 2N times better than the old architecture, assuming at least 1 tracking server N. Write performance in both cases is constant.

RAM usage for storing Spy data was approximately 4GB per tracking server under the old architecture. At the time we had 5 tracking servers which meant 20GB total (we have 7 tracking servers now). New Spy, on the other hand, only uses up 6GB total RAM on a single virtual machine, and it takes up so little CPU power that the same server hardware also hosts one tracking server and four database servers without issue.

Bandwidth wise, the old Spy used over 20MB/sec of combined bandwidth for read requests from the tracking servers. New Spy? About 500KB/sec on average, reducing network footprint to barely 1% of what it was before.

In the event of a Spy master server outage, our tracking servers simply drop Spyy-bound packets automatically and continue persisting other data to disk, with absolutely no performance impact.

Because of the way that ZeroMQ is implemented, we can scale this architecture very readily and rapidly. It removed a lot of application complexity and let us focus on the implementation. With ZeroMQ, business logic itself drives the network topology.

Additionally, because of ZeroMQ, we can easily segment sites out onto different Spy masters with little change to the rest of Clicky, should the need or desire arise. In fact, we already do this because of some legacy white label customers (which we, of course, thank for the challenge provided by their existence in this implementation).

As stated above, redundancy is not a goal of ours for Spy, because of the volatile and transient nature of its data. But if we ever change our mind, we can simply set up a ZeroMQ device between the tracking servers and Spy master, and have each datum be split off to N master servers.

Overall, we are extremely happy with the improvements made. We are also very impressed with ZeroMQ. It has been getting quite a bit of hype recently, and in our opinion it lives up to it.

Jigsaw company API integration is dead and we're looking for alternatives

Mon, 17 Sep 2012 14:27:27 -0800

About two weeks ago, our integration with Jigsaw broke. Jigsaw is (was) a crowdsourced database of information on businesses, mainly US ones but some in Europe and other places too. It let us provide you with information like this.

We looked into it and discovered that our API key got revoked. We did some digging and realized that Jigsaw had been bought out by SalesForce last year, and Salesforce has decided to kill free access to this API, which apparently became effective around September 1.

The API is still available, if you're willing to pay. We are definitely willing to pay for this kind of data, but the price is $25,000/year. That works out to just over $2,000/month, which would make it our biggest monthly expense, other than payroll. Sorry, can't justify that.

For now, we have modified the links that pulled in this data to instead just open up a Google search page with the organization name pre-filled. In some ways this is actually better because Google will almost always find the company in question, regardless of physical location, whereas with Jigsaw, when I wanted to look up info on a company it was only maybe 50% success rate. With Google you'll have to do a bit of work on your end to find the details that we were previously providing, but at least it's something.

Alternatives?

If you know of an alternative service that is reasonably priced and includes an API, please let us know. We haven't found anything worthwhile as of yet. A lot of these services seem more geared towards providing you with leads at these companies, rather than just information about the companies themselves, which is not what we're really interested in at the moment.

We're moving!

Mon, 30 Jul 2012 12:44:48 -0800

This Saturday, August 4, from approximately 2pm to 5pm PST (GMT -0700), our web site will be offline while we move our servers to a new data center in downtown Portland.

We have carefully planned this over the last 3 weeks to ensure that tracking will still be online during this time (no data will be lost and there will be no impact on your site's performance), and that the move itself will be as fast as humanly possible. The new data center is already pre-railed and pre-wired with power and ethernet, so de-racking and re-racking will be extremely fast. So why will it take ~3 hours? Well, the old data center is about 90 miles away... :(

New machines are already setup at the new data center to support tracking during the move. When the database servers get plugged in here, they will automatically start parsing the ~3 hour back log of traffic they will each have waiting for them on the tracking servers. It will take a good 3-6 hours from that point for all servers to catch back up with real time again.

This is something we've wanted to do for a while but as we grew to over 50 physical servers, it became unfeasible. However, thanks to our full virtualization that was completed in June after many months of work, we are down to just 11 physicals! Suddenly this dream became a real possibility so we jumped at the opportunity to make it happen before we needed to add any more hardware to the rack.

To say we're excited would be the understatement of the year. We've been with the same host as we've grown to enormous bandwidth over 6 years, so they've had to grow with us, which has been the cause of most of our major outages. The new data center is enterprise class with internet connectivity across 8 unique providers, so problems with internet connectivity should be near zero. Portland is much more major hub than where we were before, so connectivity should also be significantly faster, especially for those of you outside the US. Last, being 15 minutes away from our data center instead of 90 will be a very welcome change when we need to take a trip there.

A lot of time has been spent over the last 4 months on backend/sysadmin work like this, which has interrupted our regular flow of feature releases. We'll be back to that very soon, don't worry.

New custom data report

Thu, 21 Jun 2012 13:35:33 -0800

If you log custom data with Clicky, you're probably going to like this new set of features. If you're not logging custom data, you should - it's one of our best features.

Up until yesterday, for our own reports on getclicky.com we have only been logging usernames of those of you logged in to our site. This adds a lot of personality to the visitor reports. But a lot of you log a lot more types of data such as shopping cart information, account status, things like that. I've been wanting to add summary reports for this custom data for quite a while, not only for you, but also because there were other types of data we were interested in seeing about who is using our service on a day to day basis. There wasn't much point though since there was no way to see a summary of it. But now there is!

If you log custom data, you will see a new item in the main tabs when viewing your site. And if you don't log custom data, this item doesn't show up. Here's what the main report looks like:

As you can see, when someone is logged in to Clicky, we're now also tracking what type of account they have and how long they've been a member. And we added support for attaching goals to this data, so we can see what goals the different types of accounts are completing, as well as their revenue (hidden here).

What I really love is that the sub-tabs for this Custom report are dynamically generated based on the different types of custom data you've been logging to your site. So you can click on any of those sub-tabs to see a report for just that family of data, or, you can just click the 'more...' link at the bottom of any family, just like other family style reports (browsers etc - speaking of which, some of those were broken because of horrible code, and have now been fixed as well as optimized to generate faster). (Note: Internally, data types that have parents and chidlren are called families, in case you are confused).

You can click any of the items in this report to immediately see all visitors with that custom data attached to them, and you can also click any parent, for example account type, to see all visitors who have any account type data attached to them, no matter what its value is.

Not done yet!

The new goal report we released about 4 months ago has been a big hit. We thought it would be pretty great to see custom data in this report too, so we added it: (Screenshot has been slightly modified to make it smaller)

We also created a dashboard module for custom data, and the sub-tabs are dynamically generated just like they are for the main custom report:

Last but not least, all of these items are graphable too. Just click the trend percentage next to any custom data in any report (main report, dashboard, goal report) to see its history over time. Of course, we've only been logging this data for our stats for about 24 hours so it's not terribly exciting yet:

If you use custom data with Clicky, we think this will be a nice addition. And if you're not using it yet, you really ought to look into it. Full documentation is here.

Cookies and you: An update

Tue, 22 May 2012 15:47:51 -0800

The EU cookie law goes into effect May 26. This is a great law because finally, all privacy issues will be permanently eradicated from the internet.

If there was ever an example of politicians really getting the internet, it would be this. The law requires that you list on your web site all of the cookies that get set by your web site, and the purpose of each cookie. Not even session cookies, or Javascript cookies that are never sent anywhere but simply used as boolean flags, are exempt; and this makes sense, because cookies of this nature are infamous in hacker circles for their unparalleled ability to steal your credit card, read your email, and sleep with your wife. The law also requires you to get opt in permission from a visitor before any cookies can be set. Redirecting every visitor to a page with this information and the ability to opt in to cookies is a great solution, because every additional step between a visitor and a conversion increases revenue by 10x, according to a study from AreYouFreakingKiddingMe, LLC.

We just updated our privacy policy to list all of the cookies that may get set by your site if you have Clicky installed on it, in case you need it. If this law applies to your web site, we're sorry, but you can either explain all of this to your visitors and let them opt in, or follow the instructions here to disable cookies.

We've had the option to disable cookies in tracking for a while, however that was created back when we only had one cookie (the tracking cookie). Since then we have added several more, whose purpose is to make the code more efficient and save resources on our end from deactivated/non-paying site (these being the session/boolean cookies mentioned above). Up until now, disabling cookies had no effect on these extra cookies. Because they were never sent to us, we didn't think it would matter. And it shouldn't. But now it does. Our tracking code has been updated so all cookies will be disabled now when this option is set (except one, but that will only be set for sites that are no longer using Clicky but still have the code installed, which is against our terms of service).

UPDATE: A user pointed out that Cloudflare still sets a cookie. We use them for our CDN (static.getclicky.com). Their wiki says that cookies cannot be disabled. We have reached out to them for their plans regarding this law and will update you when we know more.

UPDATE: Interestingly, the BBC, one of the biggest UK web sites, specifically states on their new cookie settings page that they embed third party items such as Youtube and Flickr that set cookies and that these are beyond their control so those cookies will still be set no matter how they decide about cookies served directly by BBC. Does this mean that sites embedding third party content or services, such as Clicky, are exempt from having to worry about these kinds of cookies? Who knows, but it's interesting.

Database server migrations over the next couple of weeks

Mon, 26 Mar 2012 17:09:14 -0800

We're migrating to a new server infrastructure over the next couple of weeks, which means all of our database servers will be finding new homes. While any given database server is being migrated, traffic processing will be halted, but the server will still be online so you can view existing data. Once the files have been copied, it will need to be taken offline for 10-20 minutes while some rsync'ing magic finishes everything up, then it will be brought back online with its new hardware. At this point it will be anywhere from 30 - 120 minutes behind real time, and traffic processing will then resume.

Anything that interrupts traffic processing is something we'd always prefer to do on the weekend. However, we can only do 2-4 servers per day and we have almost 50 database servers, so that would severely slow down the process. So, we are going to be doing it on weekdays too, and expect it to take 2-3 weeks total. But it will be worth it!

Android Widget (beta)

Tue, 20 Mar 2012 11:50:58 -0800

We're releasing a beta of our Android widget today. It should work on any phone running Android 2.0 or higher.

There are three widgets to choose from, depending on your needs. You can setup one or more widgets for every site in your account. The widgets are very battery friendly, even when set to pull every 5 minutes. I've been testing it for several months now with very aggressive settings and it's had zero impact on my battery life.

Why a widget and not a full blown app? We don't believe our resources are best spent maintaining an app that tries to do (almost) everything our web site does. Apps are much costlier to write and keep up to date. We are constantly adding new features and reports and tracking new types of data, etc. A simple widget living on your desktop (still one of my favorite Android features) that lets you keep up with your basic stats with a quick glance, with a few more details available when you click the widget (as shown here), fits the bill perfectly for us.

Handy dandy installation guide

Since this is a beta, we're not releasing it through the official marketplace just yet. So you'll need to enable the option to install apps from unknown sources in order to install this on your phone.

- Android 4.0+, go to Settings -> Security -> Unknown sources (checkbox).
- Android 2.0+, go to Settings -> Applications -> Unknown sources (see a small guide here)
Download this file onto your phone!
Once completed, click the item in your notification bar (screenshot) and install.
Setup a widget! The first time you do this, it will prompt for your username and password but you'll only have to do that once.

Important note: Widgets don't have icons you click to open them up like standard apps do. Instead, you must install the widget onto your Android desktop before it can be used.

In Android 4.0+, there is a new dedicated widgets area in your app drawer, so go into there and long-press on the widget you want, then place it on the desktop. Older versions of Android, you'll need to long-press on your desktop until a menu pops up, then select widgets then find Clicky. There are three widgets to choose from.
Android 4.0 potential installation bug - sometimes, on Android 4.0, you have to install the downloaded file twice before it will show up in your widget drawer. This only happens on 4.0 and only sometimes, we're not sure what causes it yet, but if you don't see the widget after installing it, install it again and it will show up.

That's all there is to it. As mentioned above, you can add one or more widgets to your desktop for every site in your account, and data is pulled in a very battery friendly manner.

This is a beta so we want your feedback and bug reports. We hope to have an official release on the marketplace within 1 month!

blogspot tracking issue

Mon, 19 Mar 2012 12:40:30 -0800

[Update: This has been fixed!]

On Friday, Google appears to have made a change that's affecting tracking for blogspot.com hosted sites. A significant portion of our userbase is tracking blogspot.com sites so we want to get this figured out as fast as possible.

The issue is strange. The majority of people saying tracking broke, it's only affecting visitors from Germany. This raises a red flag to me, as I know Germany has strict privacy laws. Google may have agreed to block third party scripts from loading for people who visit blogspot.com sites and live in Germany. That's a stretch but... who knows.

Another possibility is that Google is forcing their dynamic view templates on visitors in certain countries now, either as a test or that's just the way it is. I know that it is not possible to track blogspot sites that use this new format, so that would explain it. However it seems strange Google would do such a thing, especially without publicly announcing it first. The last post on their blog was 3 weeks ago.

The frustrating thing is that everyone who has contacted us, I've gone to their site and verified the code is still installed. I click around on multiple pages, then view their visitor report on Clicky and verify every page I viewed was logged correctly. So everything seems fine. The code is there. I get tracked. But the stats for all of these sites are definitely way off compared to last week, so yes, something is wrong - but it's not on our end.

And of course, there's no way to contact Google about this, other than praying they see this post.

Does anyone know what is going on or have any ideas?

Filter visitors by time spent online

Fri, 16 Mar 2012 16:00:04 -0800

We've had the ability to filter by actions, e.g. exactly 10, or greater than 10, or less than 10, for quite a while now, but we didn't have the same things for time spent online. Also, we didn't have double boundary support, e.g. more than 10 AND less than 20, which means we couldn't link to these filters directly from our engagement reports, something that has always driven me crazy (and you too, based on the emails we've received).

All of this is now fixed. It also works with the API, and the docs have been updated.

Time online is specified in seconds. Formats for both parameters are specified as shown below, if you are manually typing one in to filter by, or using the API:

10 = exactly 10
>10 = more than 10
<10 = less than 10
10,20 = range of 10 through 20 (including 10 and 20)

Unique ID tags and filters (yes, finally)

Thu, 15 Mar 2012 14:36:39 -0800

The ability to tag visitors based on IP address is a very popular feature and something we've had for 5 years now, well before we ever started using cookies to better track unique visitors. When we added tracking cookies to the mix, we never updated the tagging system to support them. That was almost 2 years ago but as of today, we now finally support these cookies too.

When viewing the details of a visitor session, there used to be a name this visitor link next to their IP address. We've reworded this to just say tag, but there's now two of these links: one for the IP and one for the UID so you can click either one.

In site preferences, the sub-menu has been renamed from IP tags filters to Visitor tags filters, and in here you can manage all of them. When creating one from scratch, we determine if it's an IP or a UID based on the pattern of the string you enter. UID's are just numbers like 123456 so if that's what you enter, we'll use it for UID tagging or filtering. If however it's standard IP format (1.2.3.4) then it will be a normal IP tag/filter as has always been.

That may sound confusing but the good news is that the vast vast majority of you create tags and filters directly from the session details page, so you don't have to worry about the number you're actually entering, it's done for you automatically. It still works that way, so just keep doing what you're doing and all shall be good.

Track your Facebook storefront with Clicky and Zibaba

Wed, 07 Mar 2012 17:08:37 -0800

We have partnered with Zibaba to offer real time analytics to anyone who sells, or wants to sell, products or services through Facebook.

Zibaba lets you create an online store directly on your Facebook fan page. Visitors can buy, share, recommend, and like products, ask advice, and share discounts, generating buzz among their growing networks of Facebook friends. And now, thanks to Clicky, you can track it all in real time.

There numerous advantages to selling through Facebook. Facebook users spend a ridiculous amount of time online, and are 9x more likely to engage with online retailers, to name a few.

If you're interested, but not sure about selling on Facebook, Zibaba offers new users a 14 day trial period to test out the software for free. Sign up here!

Path analysis (beta release)

Tue, 06 Mar 2012 16:49:57 -0800

Almost 3 months in the oven, we are finally ready to release a BETA of our path analysis feature. We're releasing as beta because there are still a few quirks and we want your bugs reports and feedback on the interface and what you think it could do better. This is a Pro or higher feature, of course.

Part of what took so long was making it fast, because we know our visitor filtering isn't exactly the fastest thing in the world, especially for high traffic sites over large date ranges. But we are limiting this to a maximum range of 31 days at once for the time being, to make sure resource usage doesn't get too ridiculous, until we know how you are using it.

There are two different features in this release.

Simple path analysis

I like this one because it's quick and dirty and requires zero forethought. When viewing your Content or Events reports (planning to add downloads and outbounds later), you'll see an icon next to each item:

Click this icon (not the star, the other one!) to see an inline popup of the top 30 next and previous pages, for whatever date range you are viewing. If you're interested in seeing top next/previous pages for a page listed in there, click it and it will keep on going. Here's a good example for our upgrade page:

The top previous page is /user/. Makes sense as that's the starting point for most registered visitors, and there are links to upgrade right from that page.

Second previous page is /user/sites - this is where people register new sites on Clicky. If they're at or over their limits for how many sites they can track or how much traffic they're allowed, we prompt them to upgrade first.

Third previous page is /compare/, which is our competitor comparison matrix. From our upgrade page, we have a link to this comparison matrix, so people can compare our pricing and feature set, which we think will help win over more customers since most of our competitors are 400+% more expensive than us (ouch!). So how would this be a previous page? Well, after people look at it, they go back to the upgrade page! Well, some of them anyways :|

The next two aren't horribly relevant but the sixth one, /user/users is. That's where you add additional user accounts. If you're at your user limit or your account doesn't have access to that feature, then we say, hey, you gotta upgrade!

I could go on all day but the point is simply that seeing what pages are actually pushing people towards paying us for our service is extremely interesting. I of course had a hunch about these most popular paths since I designed the entire flow of our web site, but I didn't actually know. Now I do and it's great!

Real path analysis

This other aspect is much more powerful but takes longer to setup. That's to be expected of course, and if you take the time to do so it is pretty fantastic. In the future, we plan to group this together with goals for multi-step funnels. That's not making the initial cut though.

To reach this report, go to Content, then the Path Analysis sub-tab. It gives a brief overview of how to use it. Click the link at the top to start analyzing. Once you enter in some paths and click run analysis on the bottom right, it will analyze your input and return a report like the one below. This example is fairly simplistic but I don't want to showcase too many private details of our business here.

Until we integrate this into goals, you can use this as a kind of faux multi-funnel goal system, because you can save paths to quickly access them later.

Important note: In the screenshot above, see how the green arrow on the right has a little break in it? This represents infinite steps in between each path. In other words, it means the visitor didn't necessary go directly from the first page to the next, but after viewing the first they eventually viewed the second one. This is always the default when adding new paths in this interface. But if you only want to analyze direct path 1 -> path 2 flow, just click that arrow and it will become closed, representing a direct step:

After you make that change, you will have to click the Run Analysis button again. In fact, any change you make in this interface you will have to do that. The reason for this is if we were auto refreshing it with every change you made, it would take a lot longer to create exactly the report you want. So instead, we let you do all the steps you want first, and only process it when you tell us too. This is an important note because it works differently than all other reports on Clicky, which all react instantly to anything you click.

So, play around with it and let us know of any bugs you find or additional features you'd like. Barring any serious bugs, we're going to take at least a few weeks break from this because we could really use it. So we'd expect that the official release with some of your suggestions and hopefully all bug fixes would be out in about 1 month.

Database maintenance this weekend

Thu, 01 Mar 2012 13:33:23 -0800

Database servers 1-3, 6-10, and 14-33 will halt traffic processing for up to 10 hours this weekend while we perform some necessary maintenance. The servers will remain online so you will still have access to all of your data, there just won't be any new data processed for up to 10 hours. Most servers will be done much sooner than that, 10 hours is just the absolute max that any of them should take.

This process will start at about 10pm on Friday (USA PST), which is the same time that our normal database backups occur.

When each server finishes, it will begin processing its backlog of traffic that hasn't been processed yet. This means there will be an additional couple of hours for each server before it is caught up with real time again.

The purpose of this is in preparation for transitioning to our new infrastructure. We have invested heavily in new hardware over the last 6 months and we are finally (almost) ready to make it live. This is one of the last steps before we start.

To determine which database server a site is hosted on, take a look at the preferences page for that site.

Custom outbound link pattern matching, and iframe tracking

Wed, 15 Feb 2012 12:03:01 -0800

A lot of people seem to run plugins or custom code on their site that automatically convert links to internal redirects, so that you can count them internally. For example, instead of linking to http://getclicky.com/123 (affiliate link), you might link to http://yoursite.com/go/clicky instead. This page then redirects the person on to the real link.

The problem with this has been that our tracking code has no idea what's actually going to happen once someone clicks that link. Without actually following the link, it simply appears to be an internal link, so we ignore it by default. There have always been several ways to track these links (CSS tagging and manual data logging), however these methods required way too much pain on your end. We don't like pain.

So, we added a new customizable tracking variable today, called outbound_pattern. This allows you tell our tracking code, with just a few lines of code in one place, what your redirect URLs look like so that we can track clicks on them automatically. You can define it as a string, if there's only one pattern we should be looking for (should apply to most of you), or if you have multiple patterns, you can define more than one in an array.

The most common redirect URLs that I've seen are one of these three:

http://yoursite.com/go/clicky
http://yoursite.com/outbound/clicky
http://yoursite.com/aff/clicky

What you'll want to use as your custom outbound pattern is the the base part of these URLs, e.g. '/go/' or '/outbound/' or '/aff/'. You'll want to make sure whatever it is you use though is unique to your redirects, otherwise other links may start getting logged as well.

Single pattern example:
var clicky_custom = {};
clicky_custom.outbound_pattern = '/go/';

Multiple pattern example:
var clicky_custom = {};
clicky_custom.outbound_pattern = ['/go/', '/aff/'];

This code would go directly above your existing tracking code. And wrapped in HTML script tags of course.

If you're running WordPress, our plugin has been updated to support this new functionality as well.

iframe tracking

A while back, we tried to make tracking iframes better, in an automated way. This ended up causing some problems though so we killed it fast. Now we're bringing it back, but it's disabled by default so it will only process if that's what you want it to do.

Here's the problem. If the tracking code is installed in an iframe, we track the iframe, but in most cases your goal is to track the parent document instead. Otherwise the page URLs and page titles you see in our reports will be for the iframe itself, and we won't be able to log referrers.

So, if your goal is to track the parent document but can only put the code in an iframe or widget (e.g. with Apple's iWeb), set clicky_custom.iframe=1, so that our code will grab the parent's URL, title, and referrer.

99% of you that have the code in an iframe, your goal is to actually track the parent document. If this applies to you, set this variable, and you should be good to go.

Long term metrics, and other goodies

Wed, 01 Feb 2012 21:46:44 -0800

Our newest update is all about the long term, and it's one of the best things we've ever done. I'd recommend grabbing a fresh pair of pants before reading on because you're going to need them before you're done reading this.

It all started out innocently enough: A fairly simple update to goals that before I knew it had morphed into something else entirely. 80% of the stuff in this update was never planned, all of these ideas just starting coming to me as I kept working on it. Hmmm this would be sweet, oh this too, oh and this... Within days my bullet point todo list for this update was ridiculously long and ended up taking almost 3 weeks to complete. But that's the way it usually goes when I feel I'm on to something good.

New goal report

The main goal report still looks the same - until you click on a goal, that is. This new single goal report is awesome. Here's an example of one goal we track for new users registering with Clicky:

Talking points:

You can now graph all goal metrics (conversions, conversion rate, revenue [not shown], visits to goal, and time to goal), simply by clicking on any of those large-sized metrics at the top of the report. Previously you could only graph conversions. We've had a million requests to graph this other data - now you finally can. We considered graphing all of them at once but they all have vastly different scales and it simply did not work well.
This visit and first visit - Like most non-enterprise level services, Clicky used to only give you this visit data, e.g. data from the current session where the visitor converted. While interesting, most visitors do not convert on their first visit to your web site, so this data doesn't show the long term effects of your marketing efforts - example, which campaign they originally came in from when they bought something two weeks later, or what site they originally were referred by, if any.

And that's exactly what first visit is all about. When a visitor converts, we now look up both their current session and their very first session on your site (that we know about it) immediately and summarize it in a table that lets us generate this report really fast, even over large date ranges. I can pull up this new report for a 30 day date range for any of the goals we track in less than 1 second. Not bad!

Going down these two new columns are five categories (only two are shown here to keep the size reasonable). These categories are campaigns, referring domains, searches, split tests, and goals, which we feel are the most relevant metrics to track in relation to tracking conversions. Previously you could see this data if you filtered your visitors by a goal then chose the appropriate segment, but we wanted to bring these important metrics front and center and make it load super fast.

Trends are also shown for all data types in this/first visit columns, and you can of course click the red/green percentage next to any of them to graph historical data just like you can with any other report. Of note however is that we've only been tracking this data for 3 days, so there isn't much history yet.

New visitor details page

The old visitor details page did the job, but I used to refer to it as Old Betty if that tells you anything about my feelings towards it. I wanted to make it look nice, and add way more details to it. So... I did!

Jam packed with details - Click here to see what this report used to look like, for the same visitor. There used to be a lot of wasted white space, with things like one line for browser, one for OS, and one for screen resolution. These are all related, so why not group together? Same with hostname/org, IP + ARIN/RIPE lookups, etc.
All time goals - Everytime a unique visitor completes a goal for the very first time, we've logged it to a table for over a year now I think. This was originally so we could calculate the time to goal and visits to goal metrics, making sure we were only counting each person once per goal. While making this new update, I realized how interesting this data could be. I can't believe we've had it for as long as we have and never shown it to you like this. So, that's what this is. Every unique goal that this unique and beautiful snowflake has ever completed is shown here, in chronological order, with the date shown next to each one. Clicking on that date will take you to the session where they completed this goal!
This visit / first visit - Just like the new goal report, we also show this kind of information in the visitor details pain now. So now whenever you're viewing the details for any of your visitors, you can immediately see interesting details from their first visit as well - including how long ago it was (in this example, 288 days and 15 hours ago). Want to see that first visit? Click on the date. Bam.

Oh, you thought we were done?

New visitors report

Reorganization - If you use goals or campaigns, icons for those were shown near the left for visitors who had either one attached to them. The problem with this is that most visitors don't have goals or campaigns attached, so this added a lot of empty white space in the middle of the report which always bothered me. So, we moved these icons to the very right hand side of the report. (This example is a bit cluttered because I filtered by visitors who had completed any goal, just to make sure there was lots of things to see).
Direct Spy access - In the screenshot above, you'll see a third icon as well next to some visitors. What is this? Oh, it's only the best thing ever! If you're old school, you'll recognize this used to be the icon we had for the Spy report, before we removed the icons because we ran out of space. If you see this eye, that means that the visitor is online now, and you can click the icon to go straight to Spy with this visitor already pre-filtered! (If you don't have a premium account, you don't have Spy, so instead we just show a green dot icon indicating their online status).

(My only problem here is that the eye icon is available as a choice to you when setting up goals and campaigns. So you might want to change that if you're using it for anything, to prevent confusion. But there is no better icon to represent what this new function does, so I had to use it. Had to.)
Quick access to visitor details - So those visitor details I was just telling you about before? Well, there's no need to actually go to the visitor details page anymore, because they're now available directly on the main visitors report. Next to the actions link (e.g. 3 actions), is a grey arrow pointing top right. Click this icon and you'll immediately see the full visitor/session details page in a lightbox without having to go to a new page.

Bonus

The code that generates graphs when you click on a trend (those red/green percentages) was a mess and had been rewritten multiple times in several places for specific functionality. This is called horrible design. I took the time to rewrite it from scratch and it's way smarter and centralized in one place now, as it should be. Previously this code was unable to graph anything that had a parent/child relationship - for example, the this/first visit columns in the new goal report, that data is stored as children of the parent goal. I wouldn't have been able to let you graph this without rewriting this awful code. I really had no choice here.

So what's the bonus? Well, if you use our Twitter keyword monitoring, of course we keep history for all of that data - but we never showed you the trend percentage because we didn't want you to try to click it and have nothing happen (this data is stored with same parent/child relationship in the database). But since that mess is fixed, I have enabled the trends on the Twitter report, including the ability to graph any of them by clicking it. I use our Twitter monitoring a bit obsessively so, considering we released this feature almost 3 years ago, it's about damn time is all I have to say.

One more thing

From a UX perspective, the fact that we graph both hourly and daily data using the same colors is confusing, because we only log some data as hourly, and we show hourly by default if we have it. The only solution I can think of is to have the two types be different colors, but I can't find anything that works with our overall color scheme. We have a heavy emphasis on blue and orange. So I tried changing one of them to orange, and it looked horrible (screenshot). I'm just wondering if anyone has any ideas in this department, because I'm at a loss.

Ok, one more

In development for 2 months now is our path analysis feature. We're still working out the wiggles but expect it to be out within a few weeks and it will make an excellent partner to goals (although of course you will be able to use it without goals).

HAPPY GROUNDHOG DAY!!!

Count your chickens while they're still hatching

Wed, 04 Jan 2012 22:40:02 -0800

About 3 months ago, Google announced that all users logged in to their Google account, their search terms would be hidden from the referrer string when they clicked through. Net result is no trackers can see these search terms (surprisingly, this includes Google Analytics). Google says this is for privacy reasons. That's great, but I've said a thousand times, search analytics is one of the best reasons to run analytics in the first place so it sucks for site owners.

Anyways, the way our code processed blank search terms, they would show up like this:

This has led to a lot of confusion and many people thinking Clicky is broken, because they weren't aware of this change on Google's end (understandable).

Google updated their Analytics product to show the search as (Not provided) if it was blank. We've decided to do the same thing, although we are labeling it as [unknown] instead:

(Update: Per request, we changed this to [secure search] instead)

The main difference though is that this will now show up in your main searches report too. Previously blank searches just weren't logged in there. Now you will see an item for [unknown], and unfortunately it's probably pretty high on the list. For our own stats on getclicky.com, it's the #3 search term, representing about 1 out of every 6 searches.

I wouldn't be surprised if Google made this the default for all searches on Google sites within the next year or two. Hence I say to you, count your chickens etc, because that will be a horrible day for site owners the world round.

At the very least however, I am happy to know that Google is not providing a back door to their own analytics product that allows them to log the searches but not anyone else. That would just be evil, not to mention anti-competitive.

Load balancing the load balancers

Tue, 15 Nov 2011 01:28:14 -0800

Load balancing your load balancers sounds a bit ridiculous but yes, that's what we have to do to scale.

For about three and a half years, we've had a single pair of load balancers. One active, one in hot standby mode to take over if the active one goes offline. It's worked really well, but they were reaching capacity. I've pulled all sorts of tricks out of my hat to reduce excess load, but the time finally arrived where new hardware was the only option left.

So, we just added two more load balancers to the equation. But these new ones are twice as powerful as the old ones, and instead of active/passive pairing, they're both active. This means we have a total of three active load balancers, and these are load balanced with DNS round robin, with DNS monitoring/failover on top of that to quickly and automatically remove one from the pool in case it has a problem. The old ones act as a front for our entire service with a bunch of virtual services running on them, and hence remain paired (keeping them in sync would otherwise be a nightmare). The new ones are just for additional tracking capacity and nothing else though, so this setup is great for us.

Speaking of tracking capacity, this just quintupled it to ~25,000 pageviews/second. Just in time too, because last year, online activity on Black Friday and (cringe) Cyber Monday sent a ton of extra traffic our way and knocked us offline for a bit. This year, we're ready, so bring it!

Something else we just changed today is that the static resources for our web site are no longer hosted on static.getclicky.com. A problem common with many trackers is that their domains are on blacklists of ad-blockers and other privacy software. These days, almost weekly we get an email from someone saying our site looks funny. This is because their browser, or their OS, or their ISP, or who knows, is blocking static.getclicky.com from loading anything, so the stylesheet doesn't get loaded (nor any of the images, javascript, etc).

But we've had a secondary domain, staticstuff.net, that we've used as a generic CDN domain for our white label service for a while now. I had to change a few things around and get SSL running on it, but now that that's done, all assets for our web site (except the tracking code itself) will load from this domain instead. It points to the same servers, but it should bypass 99% of these blacklists because they are almost always based on domain name.

Google search rankings :D

Thu, 10 Nov 2011 22:22:56 -0800

Considering how much Google has been playing the privacy card recently, I'm surprised they do this, but it turns out the referrer string for Google searches typically includes a variable in the URL, cd, which signifies the approximate ranking of the link someone clicked on to get to your site for a search term. e.g. 1 would mean your page was the top result.

We've had requests to parse this data. As of about 20 minutes ago, we are now doing just that! (Pro+ account required).

Note that this is passive, in other words, we only have the data that Google gives us as people click through on search results. We're not scraping SERPs, and we only have this data from Google because they're the only ones who do this. (If any other major engines do it too, let us know and we'll try to add it). If you want scraped results, you should use our SheerSEO integration.

The screenshots below show how we report this data (5 different places - Spy is my favorite). Keep in mind the data shown in these screens is from our own stats and it's only been live for a short time so there's only a few results logged so far. But I took a peak at some super high traffic sites to see what the reports looked like there, and it's awesome.

Also, the ranking numbers you see for any given search are the average of all searches for that term. Not all visitors see the exact same search results! So if you had two people with a search and one saw it at position 4 and the other at position 5, the number reported would be 4.5.

First, rankings integrated right into the main search report:

But of course there's also a report dedicated to just rankings, and in this case, they are sorted from best to worst rank (menu included in the screen so you can see where to go to get this):

We integrated it into the keywords report too, so you can see your average ranking for any given single word. Note that the ranking numbers in this shot aren't accurate, since we're dividing the sum of all rankings for any word by the total number of searches for that word on that day, and since we only have ranking data for 20 minutes so far today, the divisor is out of proportion. Come tomorrow, it will be correct.

We also added it to the dashboard search module:

And last but certainly not least, we put it in Spy, which is my absolute favorite. As live searches from Google stream in, we'll show you the ranking right there (if it's included in the referrer string). You might also notice we're now showing the search in the same way we used to with the old version of spy, where it's a separate string instead of just the full referrer string which is harder to read searches from:

I freaking love this and hope you do too!!

Here's what's been happening

Wed, 09 Nov 2011 01:48:57 -0800

It's been a rough week. I wanted to explain what has been happening recently with our CDN, and talk about all of the problems we've had with CDNs in general. If you can stomach a novel, you'll discover the good news that it's been resolved to the point where we don't foresee any further issues.

The quest

In June, we decided to move away from our home brew CDN and get a real one, because we were outgrowing it and it was becoming a real pain to manage amongst other things.

The main requirement was that we needed support for HTTPS with our own domain name (static.getclicky.com). There are surprisingly few CDN's out there that offer this service without selling your soul and first born child. Most CDN's only let you use a generic sub-domain of their CDN's domain to get HTTPS, such as omg-secure.somecdn.net. This is fine the assets on the CDN are only for your web site, but that obviously is not the case with us.

Literally the only two we could find that offered this feature at a reasonable price were CloudFlare and MaxCDN, so we decided to test these out. We also wanted to try one of the enterprise level ones, just to see the difference in performance. For this we chose the 800lb gorilla that is Akamai.

MaxCDN offers HTTPS for $99 setup + $99/month, on top of the normal bandwidth costs. Very reasonable. The service was perfectly fine, but they only have locations in the US and Europe. This is definitely a majority of our market but we wanted Asia too. Well, they do offer Asia, but you have to upgrade to their enterprise service, NetDNA, for considerably more money. It was still less than what we were paying for our home brew CDN though, so I decided to try it.

This was one of the worst days I've ever had. I didn't know when the transition was occurring, because I had to submit a ticket for it and then just wait. When they finished it, they let me know, but they messed up the configuration so the HTTPS didn't work. (They forgot the chain file. If you know how certificates work, that's kind of important). It was several hours before I realized this however, because DNS hadn't propagated yet - I was still hitting their old servers for a while, which were still working fine. Once I realized there was a problem, the damage had already been done to anyone who was tracking a secure site. Not to mention it completely broke our web site for our Pro+ members, since they get HTTPS interface by default and none of the assets were loading for them. I immediately emailed them to get it fixed, meanwhile I pointed the domain back to our old CDN so HTTPS would work in the meantime. But they never actually got it fixed. I don't know what the problem was, we had a lot of back and forth, but it was clear this was not going to work.

Next was Cloudflare. I'd met the founders at TechCrunch Disrupt the previous September, they're great. Thing is, they're not technically a pure CDN. You point your DNS to them, and then all of your site's traffic passes through their network. They automatically cache all of your static resources on their servers, and then accelerate your HTML / dynamic content. Accelerating means requests to your server pass through their network directly to speed them up, but they don't cache the actual HTML - it just gets to you faster because the route is optimized.

All in all it's a fantastic service, and I'd be all for it, but they didn't (and still don't) support wildcard DNS - which is another do-or-die feature for us because of our white label analytics service. But their rock star support guy, John, told me they could setup a special integration with us where we could just point a sub-domain to them to act as a traditional CDN. Well, it was worth trying because there weren't any other options at this price level, especially since HTTPS only costs $1/month on top of their normal pricing, and they have servers in Asia too. It seemed too good to be true really. How could they be doing this for such a great price and have such good support? I'm pretty sure John doesn't sleep, no matter what time I email him I have a reply in minutes it seems.

Anyways, the service worked great. We had it live for a week or two. At some point there was a problem that caused us to move back to our home brew CDN, although I don't recall what it was exactly. But overall I was happy and planned to test it again in the future, but I still had Akamai to test.

Akamai is what the big boys use. Facebook, etc. I knew it was good, but also expensive. However, I figured it was worth it if the service was as good as I expected it to be. They literally have thousands of data centers, including South America and Africa which very very few CDN's have, and my speed tests on their edge servers were off the charts. Using just-ping.com, which tests response time from over 50 locations worldwide, I could barely find a single location that had higher than 10ms response time. Ridiculous to say the least.

They gave us a 90 day no commitment trial to test their service, which was appreciated. Their sales and engineer team were great. Very professional, timely, and helpful. But man did I hate their control panel. It was nothing short of the most confusing interface I have ever laid eyes on. I had no idea how to do anything, and I'm usually the guy who figures that kind of thing out.

They walked me through a basic setup, but then the next thing I didn't like was discovered - any changes you want to make take 4 hours to deploy. What if you screw something up? That's gonna be a nail biting 4 hour ball of stress waiting for it to get fixed.

I never actually got to really test their service because I was just too scared of screwing it up. A few weeks had passed and I had forgotten how to configure anything. My patience was wearing thin, as our custom CDN continued to deteriorate and I was dealing with other junk too. There's always a thousand things going on around here.

John from Cloudflare continued to email me to ask how our testing was going with these other services. He was confident Cloudflare would meet our needs. I was pretty sure too, just hadn't made up my mind yet. But I decided to go back to them because I didn't have much other choice.

That was early August and, well, we've been with them ever since. No problems at all. Great service. Overall I have nothing but good things to say.

But then...

Well, it turns out there was a problem. A few weeks ago, our pull server (that they pull static files from) crashed, and at the same time our tracking code stopped being served. It was fixed quickly but... How could this be? They should be caching everything from this server, right?

I emailed them about it and they weren't sure how the server crashing would affect cached files being served. But unless the cache expired at the exact same time as the crash, something was definitely up.

I did some digging and finally ended up watching the ifconfig output on the pull server, which shows bandwidth usage amongst other things. We were pushing almost 3MB per second of data out of that thing. Hmm, that doesn't seem right.

I renamed the tracking code file as a quick test, and sure enough, suddenly Cloudflare wouldn't serve it. Put it back, bam, it worked.

Clearly this file was not being cached. But why? Well, it wasn't their fault. The problem was the rather strange URL for our tracking code. Instead of e.g. static.getclicky.com/track.js, the URL is just static.getclicky.com/js. This is one of those Why the hell did I ever do that type things, but is too late to change now with almost 400,000 sites already pointing to it.

I emailed them about this and only then discovered that they cache based on file extension, not mime type or cache headers, which we of course properly serve. I wish I knew this beforehand, but wish in one hand shit in the other, see which one fills up first.

At this point I knew I needed to do something, since this single file was not being cached properly, it relied 100% on the single pull server being online at all times. I should have made it my #1 priority but with only a single 5 minute outage in 2 months, I somehow convinced myself I could think about it for a while. This was a big mistake on my part and I apologize profusely for it - it won't happen again. I could have spent a few grand with Dyn to get failover immediately to give us a safeguard until I found the right (affordable) solution, but I didn't (more on this in a second). I'm really sorry and I won't compromise our reliability like that again. Clearly it was not worth it.

So anyways, the same day I discover this caching issue, the server crashed... again. I got it fixed quickly, and as a quick precaution I setup another server and setup round robin DNS to serve both IPs so in case one crashed, there'd be backup. However there was not monitoring/failover on this config, but if DNS serves multiple IPs for a domain, theoretically the requester is supposed to fall back on the second one if the first one fails. I had never actually tested this scenario, but it was just an intermittent fingers-crossed fix until I got a real solution in place.

And then the server crashed again... and I discovered this did not work as I hoped (surprise).

Ok, so we need failover on this, like yesterday. This is now my #1 priority. Our DNS provider, Dyn, offers this feature, but what I hate about their implementation is the restrictions they place on the TTL (time to live), which is how long DNS will cache a query for. Obviously the TTL should be fairly short for maximum uptime, but the max they allow you to set with failover is 7.5 minutes. And with our level of traffic, this increases our bill several thousand dollars a month which is a bit steep for my liking. Not to mention the expensive monthly base fee just to have this feature enabled in the first place.

The plan

I finally came up with a plan though. I found another DNS provider, DNSMadeEasy.com, that offers monitoring/failover for very reasonable pricing and no restrictions on TTL. I specifically emailed them about this like 4 times to confirm it would work exactly as I expected. However I can't just transfer getclicky.com to be hosted there, because we're in a contract with Dyn (sigh). So I was going to setup a different domain on their servers, and then using CNAME's, point Cloudflare to pull files from that domain, instead of the sub-domain we were using for getclicky.com.

That was yesterday. Great!, I said to myself. I'll set it up first thing tomorrow because it's almost midnight!

And then this morning.......... that's right, the freaking server crashed again. My phone was on silent by accident and I slept in, so for almost 2 hours our tracking code was only being served for about 75% of requests (because DNS IP fallback does work some of the time, it seems). Hence, more problems this morning.

ARGH. I screamed at my computer and just about burned down my house I was so mad. I had come up with a plan that I knew would work and was going to implement it first thing the next day, but the server crashes in the meantime and here I am in bed, blissfully dreaming of puppies and unicorns, unaware of any problems because my STUPID PHONE IS ON SILENT. WHY. ME.

The fix

But the good news is, today, I got this all setup. Monitoring/failover is now live on our pull servers, and they are checked every 2 minutes - so if there is a problem with any of them, DNS will stop serving that IP to Cloudflare within 2 minutes at the most, and I verified it works properly by intentionally killing a server. And the TTL is only 5 minutes, so the absolute maximum amount of time there could potentially be a problem for any individual person is 7 minutes. And we added a third pull server, so at the most this would only affect 1/3 of anyone, and even then, for a maximum of 7 minutes.

(Note: Above I was complaining about Dyn's 7.5 minute max TTL, and here I am with a 5 minute one. Well, this one's a bit different because only Cloudflare's servers talk to it, so the total queries generated are quite small. The real issue is we're also going to be doing this same thing in order to load balance the load balancers (really?), because we're adding two more of them this week. Using failover on this is what would be really expensive, so we're avoiding that by using another DNS provider for it, and we figure we might as well do all of that monitoring and failover in one place. Load balancers are stable and reliable, so the TTL will be a bit higher - and even if not, their pricing is considerably cheaper than Dyn's, so it's all good).

On top of all that, Cloudflare desperately wants to fix this caching problem on their end too. (I say problem in quotes because their service is working exactly as they designed it to work, I just didn't know ahead of time that caching was based on file extension only). They are working on a solution that will allow us to rewrite URLs on their end so that their servers will see the tracking code file as something that ends with a .js file extension and hence cache it properly, without us having to make any changes on our end. Once that's live, even if all 3 of our pull servers were offline (knock on wood), it should have zero impact because that stupid legacy URL file will be actually be cached!

In conclusion

So that, my friends, is as short a summary as I can write about everything we've been through with CDNs.

And on top of all this, we also made an update to the tracking code on Nov 1 that caused issues for some of you. This update has been reverted but that was the last thing we needed with the CDN also causing issues at the same time. [Update: And there was a small network hiccup at our data center on Nov 9 that caused a short outage. Worst week ever.]

So I don't really feel like we have earned your money this month (and to think, it's only the 8th...) If anyone wants a refund, send us an email we'll happily refund you a full month of service.

No matter what, know that I value the quality of our service above anything else and will always do everything in my power to make sure it works flawlessly. This has been a horrible week, but as of now the CDN should not impact anyone.

Thanks for reading and (hopefully) understanding.

iframe tracking, copying dashboards, Google search encoding, etc

Tue, 01 Nov 2011 13:05:31 -0800

It's new feature Tuesday!

Better iframe support

A common problem we have is that people can only install the tracking code inside an iframe, but they want to track the parent document, not the iframe. But unfortunately this only tracks the iframe itself. Now I know there are plenty of people who want to track the iframe specifically, but there are way more people in the other camp. So now, by default, our tracking code will detect if it's in an iframe and use the parent documents URL and title instead of the iframe's. This is already what most other services do by default.

There is a way to override it though if you are actually wanting to track the iframe on purpose, via the new clicky_custom.iframe property.

[Nov 3 update: This change may have caused general tracking problems for some sites. We tested it, as we always do, against all major browsers before deployment and it worked fine, but something with it is causing problems for some of you, so the change has been reverted. We will look to add it back in the future.]

Copying dashboards between sites

If you have a bunch of sites, you may have created the most awesome amazing customized dashboard ever. And then you have to recreate it for every site in your account. So fun!

Well, now when you go to your customize dashboard page, there will also be a list of all the dashboards you've created for your other sites. One click and bam, that dashboard is now copied into the new site. After it's copied you can edit it if you want, or just leave as is.

Google search encoding

Some change Google made to their URL structure is resulting in double URL encoding, and you might be seeing searches+like+this instead of searches like this. It's not just affecting us either, I checked my Google Analytics account (gasp!) and was seeing the same thing. As of today, we now just double URL decode all searches before storing them, so this problem is history. I imagine Google will fix it on their end eventually but patience is not one of my virtues.

Black nav bar

We know some of you don't like this but we feel it's important to have these links highly visible and easy to find. If you stick something in a footer, no one clicks the links because no one sees them. Two designs ago, when we actually had a footer, as soon as we moved a bunch of those links into the sidebar we added, the number of clicks each one was getting skyrocketed. I'm talking 10-20x as much activity. That's a good thing.

But anyways, today, I reduced the padding so it takes up a bit less space, and also removed the position:fixed style rule so it's not always on the screen, instead it's only visible when the page is scrolled up all the way. I hope that appeases some of you to a small degree at the very least.

More ways to view hourly data... and more!

Thu, 27 Oct 2011 16:46:52 -0800

We just deployed a bunch of changes to hourly data, amongst other things:

Goals and Revenue now support hourly data, so you can more easily see your best converting and most profitable times of day. However, we just started doing this today, so earlier dates will not have hourly data.

Anytime we add hourly support for something, it essentially require 24x as much storage space, which is why we only do it for a few types of data (currently: visitors, actions, tweets, clicky.me short URLs, goals, and revenue). If you saw how big our databases were already, you'd cry and then realize why this is necessary.
Hourly averages - there are three new options in the drop down menu for hourly graphs:
- Same day of week average - example, this Monday vs the average of the last 4 Mondays
- Weekday average - Today vs the average of all weekdays (Monday-Friday) from the last 4 weeks
- Weekend average - Same as Weekday average but for Saturday/Sunday only
These are all insanely useful, particularly the first one!
You can set same day of week average as your default trend comparison in your dashboard preferences, in which case your hourly graphs will also default to displaying this mode. We initially coded in support for weekday/weekend stuff too, but they generated WAY too many queries; there were up to 20 extra pieces of data that needed to be pulled from the database for each item in any given report, and it couldn't be optimized since there are holes in the date ranges.
Daily graphs default to 28 days instead of 30 days, to more cleanly fit week boundaries. We think you will find this especially useful when comparing vs previous period. An example is shown on the right.
Compare menu fixes/additions - When viewing daily graphs, the Compare... menu has been broken for a while now. Not sure when it happened but we finally got it fixed. We also added some more options to it that were much needed (revenue, goals, campaigns, pages, and tweets, to name a few).

24-hour time formatting and smarter defaults for new sites

Fri, 21 Oct 2011 14:42:47 -0800

We have finally added a 24 hour time formatting option, so you'll see e.g. 15:30 instead of 3:30pm. This change should affect everywhere you see time within a site's reports, but if we missed something, let us know. You can change this setting in your site preferences.

The defaults when registering a new site also just got a lot smarter. Those of you with lots of sites probably get annoyed with how many preferences you have to change every time you register a new site - particularly if you are not on the west coast of the US. Now, any time you register a new site, we'll grab the following preferences from the last site you registered and make them the default for the new one (which you can of course change if desired):

Time format (12/24 hour)

Time zone

Daylight savings

Anonymous IP logging

Hide hostnames in visitors list

Hide ISPs

:D

utm_custom: a new URL parameter to attach custom data to visitors

Thu, 20 Oct 2011 19:41:41 -0800

One feature that gets requested a lot is to be able to set a variable in the URL that could then be attached to the visitor as custom data. This would be particularly useful for things like email newsletters, so when someone clicks through, they can be identified automatically.

The variable name needed to be generic because of our white label program, and since we pictured this being used with campaign activity more than anything else, we decided to call the variable utm_custom (related to Google/Urchin's utm_campaign etc variables).

You can see full documentation here. Because custom data requires a Pro or higher account (upgrade), this variable will also only be processed if you have a Pro or higher account.

utm_custom is an associative array so you can set multiple key/value pairs on a single page. (It must be an array with at least one key/value pair, or it will be ignored). For example, if you sent a visitor to this page:

http://yoursite.com/landing/page?utm_custom[username]=Bob+Jonesutm_custom[email]=bob@jones.com
utm_campaign=Email+blastutm_content=Oct+20+2011

You would see this in your visitor's list:

And this when viewing visitor/session details:

We've had requests for this countless times over the years so we know many of you will find it quite useful :D

SHUT. DOWN. EVERYTHING.

Thu, 29 Sep 2011 13:59:18 -0800

Hmm... did anyone actually read the announcement that Google made today? This isn't real time Google Analytics, this is a single report in GA that is real time. The rest of GA remains the same. This is more akin to Chartbeat, to be used as a real time compliment to a standard analytics package, rather than a full standalone real time service like Clicky is. But I guarantee you Chartbeat will be just fine, as will everyone else. We've all had, and continue to have, plenty of advantages over GA other than real time data.

If anything, I'm glad Google has done this, as it will bring more awareness to the concept of real time web analytics in general. This will inevitably lead to more people searching about it, and we just so happen to have the #1 organic result for this search on both Google and Bing (and hence Yahoo). Everything is going to be just fine!

Multiple dashboards and custom dashboard modules

Wed, 21 Sep 2011 15:10:39 -0800

Paying members can now create up to 5 unique dashboards for each site, allowing you to have multiple birds-eye views of your analytics - all a click a way in the sidebar (as seen to the left).

This is especially handy because you now have the ability to create your own dashboard modules. For example, if you want to see both top countries and top cities at the same time, you could create two new boxes, each one having one of these pieces of data. Previously, dashboard modules were categorized, so you could only view countries OR cities, but not both at once. You can add more than one data type to each module too:

And a much requested feature is the ability to assign a default dashboard to each of your sub-users. This allows you to create things like boss mode, where he will see only the things he cares about, but you, the analytics junkie, can see everything you want.

New option to anonymize IP addresses

Fri, 26 Aug 2011 12:28:23 -0800

We just added a new site preference to log IP addresses anonymously, which takes the last octet and changes it to 0. For example, 123.123.123.123 would become 123.123.123.0. And we're not just hiding the last octet from you - we remove it before storing it, so it should comply with laws in places like Germany where it's technically illegal to log the full IP address of a visitor.

This preference is available to all users. You can find it in your site preferences, under advanced.

BREAKING: TWITTER'S INFLUENCE IS NOW ZERO. FACEBOOK WINS. ALL HAIL THE ZUCK.

Sat, 20 Aug 2011 10:49:05 -0800

The media loves to hate Twitter when it comes to measuring their influence, or how many people click links shared on Twitter. Even when they state the obvious reasons (HTTPS sends no referrer, 99% of people use Twitter apps which also send no referrer, etc), they brush them off and pretend no one uses Twitter.

I'm not here to defend Twitter, but they just made a big change that almost 100% guarantees headlines like the one we're using here.

What am I blathering on about? I can't exactly tell what Twitter changed, as I never paid attention to t.co really, but either they finally started actually routing every link through their shortener, t.co, or they changed the way t.co redirects. The bottom line is this: you will pretty much never see a single referrer from twitter.com again. Instead you will be seeing t.co. (At least from non-Clicky analytics services - more on that in minute).

The reason is because of the way t.co redirects. Instead of using a 301 redirect, which every shortener should do, they are using a combination of a META redirect and Javascript to send you on your merry way. If you click a t.co link and hit escape really fast, your browser will pause on that page, allowing you to see the source:

noscriptMETA http-equiv=refresh content=0;URL=http://www.addedbytes.com/blog/if-php-were-british//noscriptscriptlocation.replace(http://www.addedbytes.com/blog/if-php-were-british/)/script

The problem is that both META redirects and Javascript redirects overwrite the original referrer. The referrer becomes the page with that code on it. Hence, no more twitter.com referrals, because every single link passes through t.co now.

Here are screenshots of our 30 day history of referral traffic for twitter.com vs t.co. Notice the last 3 days, twitter.com is zero, and t.co has come out of essentially nowhere:

What to do?

We just updated Clicky to convert t.co referrers into twitter.com automatically. Knowing the specific t.co link someone came from has no value, but knowing someone came from Twitter does. No other analytics service has done this yet that we know of, so as usual our tiny size lets us move lightning fast to address issues like this.

[Update: ok, we changed it back. See comments.]

But wait there's more! There's a damn nice silver lining here. The main reason twitter.com doesn't show up as a referral much, even before this change, is because so many people use apps instead of the actual twitter web site. When you click a link in an app, that app passes the URL to your browser, which then opens it - but that means no referrer.

However, now that all links are being passed through t.co, and this happens via the API as well (I confirmed via our own Twitter keyword monitoring feature), this means that even people who click links from apps will be passing through t.co first. If we automatically convert t.co links into twitter.com in the backend, this means we will be able to give you a MUCH more accurate picture of Twitter's traffic to your web site. (We just pushed this change so we can't guarantee this will be the case, but based on what we've analyzed, if you use Clicky you should start seeing a huge influx of twitter.com referrals).

This may even be the reason that they're doing redirect this way, because they're sick of the uninformed media saying they have no influence. By having a ton of t.co hits suddenly showing up in referrer logs, maybe the media will consider what that means. Doubtful, but one can always hope.

JW Player tracking, and other recent changes

Tue, 16 Aug 2011 11:26:15 -0800

Since we released video analytics last year, the number 1 request has been to add a library for tracking JW Player. Well, we just did that. You can find it on our video help page.

We've also been fixing a bunch of bugs recently, and adding a few minor new features. We don't normally make a blog post just for this type of junk, instead preferring to piggyback on top of another post, so here we go:

- NEW: Checkbox on the tracking code page to disable affiliate badge (for paying members)
- NEW: Email reports are now in your selected language, instead of English for everyone.
- NEW: go.mail.ru is a search engine, not a web mail domain - yes, we hear you :)
- FIXED: Tracking code pings now end at 10 minutes as stated in the docs, rather than 8 minutes and 40 seconds (this was a math fail on our end)
- FIXED: The Flashy widget was broken.
- FIXED: After editing site prefs, you may have seen a message about your site being disabled.
- FIXED: Y-axis text in charts was getting truncated for values over 6 digits.
- FIXED: Loading a site, changing to a different site, then choosing Bigscreen would load that for the FIRST site.
- FIXED: Non-paying users who use our Wordpress plugin, viewing stats within the WP admin area didn't count as a login so they were getting disabled after 60 days of no logins. Now we count that as a login, even though technically it is not, to avert this problem.

clicky.com/blog

Local search now supports 'pretty' searches and multiple parameters

Tracking tel: URLs, custom heatmap objects, and other tracking code updates

New features

Bug fixes for sites using multiple tracking codes

Coming soon

Local / internal search support!

Tracking Youtube videos no longer requires PhD

Coming soon

API throttling

Monitage: Uptime monitoring beta

'GetClicky' no more!

On-site analytics update

Update on heatmaps

Please do not place third party javascript in your HTML head.

Heatmaps!

A difficult decision

Data retention

Internet Explorer, may I count the ways in which I love thee?

On-site analytics!

Visitors online

Visitors on this page

Heatmaps

Other notes/features

History of Spy (also, ZeroMQ rocks)

Initial implementation

Spy, evolved

Trimming the fat

Enter Spyy

Conspycuous benefits

Jigsaw company API integration is dead and we're looking for alternatives

Alternatives?

We're moving!

New custom data report

Cookies and you: An update

Database server migrations over the next couple of weeks

Android Widget (beta)

Handy dandy installation guide

blogspot tracking issue

Filter visitors by time spent online

Unique ID tags and filters (yes, finally)

Track your Facebook storefront with Clicky and Zibaba

Path analysis (beta release)

Simple path analysis

Real path analysis

Database maintenance this weekend

Custom outbound link pattern matching, and iframe tracking

iframe tracking

Long term metrics, and other goodies

New goal report

New visitor details page

New visitors report

Bonus

One more thing

Ok, one more

Count your chickens while they're still hatching

Load balancing the load balancers

Google search rankings :D

Here's what's been happening

The quest

But then...

The plan

The fix

In conclusion

iframe tracking, copying dashboards, Google search encoding, etc

Better iframe support

Copying dashboards between sites

Google search encoding

Black nav bar

More ways to view hourly data... and more!

24-hour time formatting and smarter defaults for new sites

utm_custom: a new URL parameter to attach custom data to visitors

SHUT. DOWN. EVERYTHING.

Multiple dashboards and custom dashboard modules

New option to anonymize IP addresses

BREAKING: TWITTER'S INFLUENCE IS NOW ZERO. FACEBOOK WINS. ALL HAIL THE ZUCK.

What to do?

JW Player tracking, and other recent changes