getclicky.com/blog

New segmentation features: landing/exit pages, number of actions

Wed, 04 Nov 2009 00:40:17 -0800

We're launching some really great new segmentation features today:

Filter by landing page
Filter by exit page
Filter by number of actions (exactly 1, greater than 10, etc)
View top pages, landing pages, and exit pages for any segment of visitors
This is of course all available in the analytics API too

Let's run through an example for getclicky.com's traffic. Note that some of these screenshots are edited to save space.

First, let's analyze our home page. Before this new update, all we could do was filter by visitors who had seen home page at any point during their session, like this:

But now we can filter down to only visitors who actually began their visit on our homepage. These are who we really want to look at. This is more likely to be new visitors, as they wouldn't be coming from bookmarks to pages within our site or anything like that:

Based on the average actions and time per visit for this second group, it's obvious these people aren't as engaged as the first group. Perfect, that's who we want. Ok, let's look at the top exit pages for these visitors, which will show us where we lost the most visitors:

But is this the data we really want? This includes people who came to our homepage, perhaps logged in and viewed 1,000 reports, then logged out and left. We don't want these people included here. So, let's add a number of actions filter and set it less than 5. This will now mainly be people who are not engaged with our web site at all.

As you can see, the average time and actions is now way less than it was before. Now let's look at the top exit pages. It's about the same, actually. The numbers have gone down on all of them, but the actual top exit pages, and their order, is about the same. But that's fine - we can still look at specific data for this segment of visitors, such as top goals completed, or goals started but incompleted, to get some more data from them. Or even view the individual visitor sessions that interest us.

I won't go any further with this example, but as you can see, the paths of exploration are endless from this point. I hope it gives an idea of the powerful filtering you can do with these new features.

Here's one other quick example: Top landing and exit pages for everyone who arrived via search.

The pricing page is still a big deal for exits though. Gonna have to work on that one :)

Updates to Wordpress plugin

Tue, 03 Nov 2009 00:19:39 -0800

The two biggest feature requests have been added to the 1.0.2 release of our plugin:

Field to enter in text that precedes the auto-tweet, e.g., New blog post:
New preference to always tweet new posts. Enabling this option should allow new posts that are created within third party interfaces (e.g., the iPhone) to be auto-shortened and auto-tweeted as well.

What are you waiting for? Download the new version now and tell us what you think!

Note: we are in the process of adding this to wordpress.org so that in the future, updates can be fully automatic from your WP admin page.

New tracking code format; please update your site if possible

Mon, 02 Nov 2009 14:49:57 -0800

We've recently changed the format of the tracking code you paste onto your site to take advantage of cross-site caching. Here is an example of what it used to look like (Javascript portion only):

[script src=http://static.getclicky.com/123.js][/script]

In this example, 123 would be the site ID for this site in our database. This is all fine and dandy, but the problem is that the URL is unique for each site that has our tracking code on it, even though the actual javascript code that is being downloaded is the exact same. So if someone browses to 10 different sites in a day that all have Clicky on them, they would be downloading 10 copies of the same code, instead of 1.

As of a few weeks ago, we changed the format to this:

[script src=http://static.getclicky.com/js][/script]
[script]clicky.init( 123 );[/script]

As you can see, the URL of the tracking code is now the exact same for every site that someone would visit with Clicky on it. We just added a seperate function call to set the site ID within the script after it's downloaded. Since any user's browser would cache this script, it doesn't have to redownload it. This means your site will load faster for anyone who has visited another site with Clicky on it in the last 7 days.

So, the point. If possible, please replace the existing tracking code on your site with a fresh copy (go to site prefs, then tracking code, to get a copy). As already mentioned, this will help speed up your site for the average visitor, as well as reducing bandwidth usage on our end.

Of course, the old format still works, and will indefinitely. We don't want to force you to do this, but if you have a few extra minutes, the web will be a better place. Thanks.

UPDATE: Yes, the new WP plugin has the new tracking code format. This only applies to people who manually paste the code on their site anyways, since using a plugin puts you at the mercy of the plugin developer.

Clicky API v4 released

Fri, 30 Oct 2009 03:18:27 -0800

In addition to the new wordpress plugin, today we're also releasing version 4 of our analytics API. We've got some really great new features outlined below. Full documentation is here.

Segmentation data!
Example: Summary data + top countries and web browsers for visitors from twitter.com over the last 7 days
Data for individual items (e.g. a specific country or search term). Note, this is different than the existing filter parameter, which is a wildcard (i.e., slow) match. This is an exact match string and hence very quick.
Example: Daily hits from google.com
Hourly data for supported data types (visitors and actions)
Example: hourly data for both visitors and actions over the last 7 days
Option to specify only the fields you want returned for type=visitors-list. This is a big one because we have a number of users who export their entire visitors-list every day. They may only want a few fields, but had to get all of them no matter what. If this applies to you, this new option will vastly speed up the response time for visitors-list queries.
Example: Only return IP, session ID, landing page, location, and browser
Proper error responses. It's hard to believe that up until now, a bad request would just return completely empty. How helpful of us! Now, if there are any problems with your request, an error will be output.
Example: Forgetting the required type parameter. XML PHP JSON
The XML parent element has been changed from items to the more standard response. This parent element also has a new attribute, status, whose value will either be ok or fail. If it is fail, there is an error with your request. The XML document will then have just one element, error, whose value will be the error message. For PHP and JSON, which both return arrays, the array will only have one item, whose key will be error and whose value will be the error message.
We've added a ton of inline examples throughout the documentation, so instead of perhaps being confused about what a certain feature does, you can just click a link and instantly see an example.

The two big features for us are segmentation, and being able to specify a individual items. With these, and everything else, we feel the API is much stronger than v3.

We like APIs. They're neat.

We're going to be adding some more APIs in the near future that let you get information about your account, as well as the ability to write data to Clicky, instead of just reading from it. For example, registering new sites, updating site preferences, creating goals, etc.

We're releasing one today as a sneak peak. This is something that those of you with lots of sites may find quite useful. It's called sites. (We're really creative). Pass it your username and password, and it will return to you all of the sites in your account, including their domain name, nickname, site_id, sitekey, sitekey_admin, and database server.

For example, here's a pretend account I just registered.
http://api.getclicky.com/api/account/sites?username=awesomeaccountpassword=awesomepassword

And with PHP output:
http://api.getclicky.com/api/account/sites?username=awesomeaccountpassword=awesomepasswordoutput=php

Happy Halloween weekend!

Don't drink too much! We'll see you in November.

New Clicky Wordpress plugin released!

Fri, 30 Oct 2009 02:44:08 -0800

Well that was fast! Thanks to Yoast for his efforts, he completed this in less than 12 hours from the time he started working on it. He's pretty good. In addition to the existing features (outputting tracking code, tagging your visitors, and ignoring admin visits), our new WP plugin boasts the following awesome new features:

View stats from within your WP dashboard
clicky.me URL shortener integration
Option to automatically post new stories to your Twitter account with a clicky.me short URL when new stories are published
Goal integration (bonus!)

This requires WP 2.8 or higher. It has been thoroughly tested by myself and Yoast, but, there may be random issues we didn't encounter, or compatibility problems with other plugins. If you have any problems at all, please post them here so we can try to fix them as fast as possible.

If you have the old plugin installed, delete it before installing this one!

Download the new Clicky Wordpress plugin here

$1,000 for a new Clicky WordPress plugin

Mon, 26 Oct 2009 13:10:05 -0800

Update: Thanks to all of the people who applied for this. We have selected our developer, and development is already in progress. Thanks for all of your interest.

Our WordPress plugin seems to have serious compatibility issues with 2.7 and beyond. It's also lacking a couple of features we wish that it had. We find the WP API very difficult to work with and wish to never lay eyes on it again. Therefore, we are offering $1,000 to a qualified WP plugin developer to make the plugin of our dreams.

You need to be very experienced with the WP API, and must have developed at least one major (semi-popular) plugin, or several more minor ones. You also must be the type of person who believes code is a beautiful piece of art, because if there's one thing that gets my goat, it's messy code.

The current plugin just outputs the tracking code in the footer, and has a couple of options - automatically tag visitors who have previously left a comment (by grabbing a cookie that WP sets for commenters), and ignore visits from admins of the site. The new plugin, that you will be rewriting from scratch, needs to have that same functionality, as well as the following new features:

A page to view stats within the WP admin page, via an iframe that points to our site. We will be creating a special page that this iframe will be pointing to, so we (Clicky) are in control of how this page looks and are able to update it as necessary.
clicky.me API integration. Whenever a new story is posted, we want the author to have the option to automatically create a clicky.me short URL with our API and post it to Twitter via their API. This means the user will need to have a space to enter their Twitter username and password, and if they have done so, we want this option (a checkbox at the bottom of the story creation page) enabled by default.

We also need you to support it up through and including WP 3.0, so we can ensure it works perfectly for at least the next 6-12 months. By support, we are not talking about supporting our users. We mean you will be willing to update it, for free, if any updates to the plugin API break compatibility up through and including WP 3.0.

If you are interested, and qualified, please send an email to Sean, titled wordpress plugin, with your credentials/experience. Don't be offended if you don't get a response. We will be looking over all submissions but will only be contacting the top few candidates to narrow it down to the best one and then start the process.

We prefer to pay via Paypal if possible, and if you are outside of the US, this is the only payment method we can provide. If you have a US address, then we can also pay by check, if you prefer.

Infrastructure upgrades nearly complete

Wed, 21 Oct 2009 11:39:27 -0800

Like we said, nothing exciting for a while. We've been working behind the scenes massively improving our infrastructure and updating some problem servers for greater reliability, and some old servers so they're much, much faster. We're not quite done, but here's the story so far:

Tracking servers

In each of our tracking servers, we doubled the RAM and added much faster drives to store the incoming traffic data. Initially there were a few problems but they were resolved.

As an update to that story, the problems we mentioned were related to the file system we were using, Ext3. The upgrades we initially made did help with performance, but load on the servers was still much higher than we thought it would be. After many hours of research, we discovered that this file system, which is the default for almost any Linux installation, isn't well suited to storing, updating, and deleting thousands of tiny files 24/7. It turns out the file system of our dreams is called ReiserFS. Article after article said check it, this file system is amazing for dealing with thousands of tiny files - use it if that's what you're doing. So we did.

We reformatted the drives that store our incoming traffic data to ReiserFS and the results were stunning. Load plummeted to levels we haven't seen for well over a year. So this was actually the biggest bottleneck of our existing setup, but that isn't to say our RAM and hard drive upgrades were fruitless. Before we discovered ReiserFS, the hardware upgrades still made a significant difference - just not as big as we thought they would, which is why we kept researching. Once we added ReiserFS into the equation, the results were what we were hoping for.

We also made a couple of very major efficiency improvements to the code that logs incoming traffic. The tracking servers are currently in a state of bliss and thanking us kindly for helping them work more efficiently.

Software to Hardware RAID migration

In the last 6 or so servers we built, we were using Linux's built in software RAID to mirror a pair of drives. Software RAID has served me well in the past but it doesn't seem to be quite as reliable for extremely heavy read/write drives. About once a month, we had a RAID failure which would almost always lead to one of our biggest database tables on that server having corruption. So we'd have to take that server offline and repair the 1 or more tables with corruption, which is a slow process to say the least.

A Redundant Array of Independent Disks is supposed to prevent this type of thing. A drive popping offline should be no problem - you either replace it or re-add it to the array and it rebuilds and nothing noticeable happens from the end user's perspective. But this wasn't the case with our Linux software RAID servers.

The main reason we went with software RAID was for cost savings. Not that hardware RAID is that expensive, but it adds about 15% to the cost of each server we build. So, no more software RAID. All servers that had this setup have been migrated to hardware RAID. All of our older servers use hardware RAID and they've never had a single problem.

Upgrades to old servers

As I just mentioned, none of our older servers have ever had any problems. On the other hand, they're all a bit slow, as they're not using drives meant for high performance. The database servers affected most by this were 2, 3, 5, 6, and 7.

We've migrated 2, 3, and 5 to much faster drives. If any of your sites are on these servers, you should notice very significant speed improvements when viewing your stats. We haven't yet migrated 6 or 7. We currently only have 1 spare server ready to host the data from another one. db7 seems to be slightly slower than db6 so that is the one that will be getting the upgrade first, this coming weekend most likely.

Next week, I will be at our data center again building some new servers, hopefully for the last time for a while! At this point, db6 will be moved to new hardware. db12 will also be moving, as it's also on slower drives. db12 is much newer than these others ones so it has less data, which means the speed is still acceptable - but that's only for now. Over time its performance will slowly degrade as well, so we're just going to move it now.

Once that is completed... we'll be done!!!

Well that was fun!

Actually, not really. This is the type of work that is opposite of fun. I've built so many new servers and installed Debian Linux so many times the last month, it's probably some kind of world record. But, that's ok - all of this needed to be done, Clicky is much better because of it, and we hope you have noticed the improvements.

Now, we can get back to working on the software, which is what we really live for. Look for some great new features soon!

Clicky crushes it!

Mon, 19 Oct 2009 23:17:47 -0800

Gary Vaynerchuk is one of our first customers. I don't know how he ever found out about Clicky so early in its life - he registered way back in Feb 2007, when we were absolute nobodies - but we've always been psyched to have him as a customer, because we're big fans of everything he does.

He's on tour right now for his new book Crush it, and tonight the tour hit Portland, Oregon, where we are based. We stopped by to watch him speak and take questions from the audience for about 90 minutes. And of course, we grabbed a couple copies of the book. It was awesome to meet him and his signature on my book made my day. To have the absolute king of social media be so passionate about our product means a lot.

Thanks Gary! Good luck with your book, although we know you won't need it.

Tracking server issues this morning

Mon, 05 Oct 2009 13:38:18 -0800

You probably notice a bit of missing data from this morning. Let me explain what happened.

This was not a database issue, which seems to be the story of our life recently, but an issue with our tracking servers. This means every single site was affected. The issue is very technical and related to Linux itself, but I'll try to explain it as simply as I can.

As part of our infrastructure improvements we have been making, we upgraded our tracking servers with twice the RAM and much faster hard drives. These two things combined should help eliminate most of the lag you may sometimes notice on your site during peak times, which is about 8am to 2pm PST.

However, a serious human error was made on my part when I formatted these new drives. I haven't had to manually format anything other than a drive meant for a database for quite a while. For our database drives, we use what's called largefile inode structure, which is optimized for disks that have very large files. Some of our database servers have individual files that are over 40GB. inodes store metadata about every individual file on a partition, including where exactly a file is on the actual physical part of the disk.

Unfortunately, without thinking about it, I optimized these new drives on our tracking servers the same way. It's habit at this point. The problem is that our tracking servers have hundreds of thousands of tiny text files on them that store all of the traffic coming in for all the sites we monitor. Each site has its own dedicated Spy file, and each database server has its own dedicated file as well, which is basically a Spy file times 8000. We also cache the javascript for each site seperately, for complex reasons. Including pMetrics and the version of Clicky for Webs.com, we're tracking over 500,000 sites, so this translates into a ridiculous amount of files stored on these drives.

I'm not an inode expert but I know what works well for different situations. With largefile, it creates an inode every 1 megabyte, which translates into about 143,000 inodes on the 150GB disks Raptors we put in here. With so few inodes for so many files, the percentage of inodes being used reached 100% within about 48 hours. This is a very bad thing for a heavy read/write disk with hundreds of thousands of files. Load skyrocketed to over 400 on each server, which is absolutely ridiculous. The tracking servers slowed down considerably and were timing out.

Normally I get pages within minutes of such an event. However, my stupid iPhone, which I'm about to throw out the window, was somehow stuck in headphone mode, which means the external speaker was disabled, which means it made no sound as these pages were continuously coming in. (Note - this is different than silent mode - it actually thought there was a headphone inserted, although there was most certainly not). It wasn't until I woke up at my normal time that I noticed that I had hundreds of new text messages that the servers were severely timing out.

Anyways. It took me a while to track down what specifically was causing the problem. But as soon as I found out, I knew exactly what I had done wrong. I took each tracking server offline individually and reformatted the drives that stores these tiny files with the news inode type. This creates an inode every 4KB, which translates into over 36 million inodes for these disks, which is exactly what we want for this type of usage. (This is how our old drives were formatted, and worked well except for the fact that the drives were quite slow. These servers were built when we were MUCH smaller.) When I brought each server back online, things returned to normal immediately.

We have been planning to change the javascript tracking code so it's global for all sites but it's not as easy as flipping a switch. If we had been using a global tracking file instead, this problem would not have occurred so soon. But as we continue to grow fairly quickly, it would have eventually reared its ugly head. Now it's fixed, so it should never be a problem again.

Please accept our sincere apologies. We have been having an abnormal amount of problems recently, but the quality of our service is our absolute top priority. You are upset, but know that we are 100x as upset about something like this. As we upgrade the rest of our servers over the next few weeks, we are hopeful the service will return to the stableness and quality you have been accustomed to for nearly 3 years now.

Nothing exciting for a while

Wed, 23 Sep 2009 16:44:52 -0800

We're working on massively improving our infrastructure for the next month or so, which we hope will greatly improve the speed and reliability of our service. During this time, there will likely be few, if any, new and exciting features.

There are so many awesome ideas we have for Clicky but we've reached the point where our existing setup isn't quite cutting the mustard anymore. Nothing is more important to us than the quality of our service, so we're going to be focusing on that for a bit to ensure we can continue growing well into the future with as few problems as possible.

We'll be upgrading our tracking servers with a bunch more RAM and super fast hard drives, which will help to eliminate the lag that occurs sometimes during peak times (8am-2pm USA PST) when these servers are getting blasted with over 1000 hits per second. We'll also be adding more redundancy to our main database and web servers, and splitting off Spy onto its own dedicated server to speed up the web servers even more. You wouldn't believe how much load Spy adds to our entire system - if you knew how much, you would probably cry.

We'll also be doing some more work on our database servers, as I mentioned in our last post. I didn't quite finish everything I wanted to last time I was in our data center, so some db servers may go offline here and there. The downtime should never be more than an hour or two, however. We always tweet live updates during server maintenance, so be sure to follow us on Twitter for up to the minute updates.

So that's what we'll be doing the next month or so. It's a lot more work than it sounds like, but when all is said and done, I think everyone will be really happy.

Server maintenance Thursday and Friday

Thu, 10 Sep 2009 12:28:35 -0800

I'm in our data center today and tomorrow, doing maintenance and replacing some hardware on a bunch of our database servers to help prevent issues like we had last week from happening again. We only take down servers during the week if absolutely necessary, and this is one of those cases. This is because we will be in San Francisco for most of next week, and then I will be taking another small trip unrelated to work.

The last thing we want is to have a problem while we're on the road, because it may take much longer than normal to resolve depending on what we're doing at the time of such an incident (not to mention it's a much more of a PITA to do that type of thing while traveling).

Any given server may be down for as long as 3 hours. Data will not be lost during this time, however when a server does come back online, it will take it a while to catch back up with real time.

Our #1 priority is to make Clicky as reliable as possible for you. Thanks for your patience and understanding.

iPhone updates (and we'll see you in San Fran)

Wed, 09 Sep 2009 23:30:00 -0800

September 8th marked 1 year since we launched our iPhone web application. We've been meaning to add some new features for a while, and what better time to do that than on its first birthday? So we started adding some new features yesterday and just launched them tonight. It's still not meant as a full replacement for the desktop version, but these new features really add to the experience and make it that much more enjoyable to check your analytics on the go:

Visitor segmentation

You can now analyze any segment of visitors like you can in the desktop version. Just click on any item when viewing popular data, and you will see data about that specific segment of visitors:

Historical data / graphs

You can now view the historical data for any individual item. Just click the red/green percentage next to any item, and you will see its daily history. We're using the pretty amazing Google Charts API to generate these graphs, since Flash doesn't work on the iPhone.

You can view history in landscape mode as well, which will widen the graph to see better detail:

Organized dashboard

We've organized all of the menu options on the dashboard into groups, which makes it much quicker to find what you're looking for. As you can see here, we also added the new Short URLs data (from clicky.me) to the iPhone app:

More dates

You can now select dates going back as far as 6 months, including individual months:

Lastly, we wanted to let you know that the entire crew at Clicky (yeah, all 2 of us) will be at the TechCrunch 50 conference in San Francisco this coming Monday and Tuesday, so if you're going to be there, we'd love to meet up. Also, because of our traveling schedule (we'll actually be in San Fran for 5 days), please be patient if you send us an email. We'll of course still be checking them on a daily basis but unless it's an emergency, it may take a few days before we respond. Thanks for understanding. We look forward to seeing you there :)

Problems this morning

Wed, 02 Sep 2009 08:44:29 -0800

At approximately 4am PST, two separate database servers (db1 and db16) had RAID failures that caused file system corruption. They kept trying to process traffic but Linux had switched part of the file system to read only, so no traffic data was actually being written to the hard drives. This problem lasted from approximately 4am to 7am PST. Unfortunately, this traffic data is gone and unrecoverable.

We have alert systems setup so that when a significant event occurs, such as a server going offline or a RAID failure, we are alerted immediately. Unfortunately, the RAID notifications on a few servers were recently disabled while we were performing some maintenance, and wouldn't you know it, db1 and db16 were among those servers. Because of this, we weren't notified of the problem, and didn't discover it until we woke up to a flood of emails in our inbox this morning.

There were no problems on other servers that we could find, but if you have a site on a server other than db1 or db16 and it's experiencing issues, please leave a comment here explaining what's happening. Be sure to include the site ID.

We apologize for this issue, which we take very seriously. The RAID notifications are all back online, and we will be sure to always re-enable them immediately after this kind of maintenance in the future. Leaving them disabled was just an honest mistake.

One final note, these RAID failures occurred at the exact same time on two different servers. This happened once before as well, although it was three servers instead of two, and it didn't cause any corruption last time. This seems like very strange behavior to us, and we're not sure what could possibly cause such a thing to happen to separate servers (that don't talk to each other) at the exact same time. If any sysadmins out there have any ideas, please share.

We just launched a URL shortener. Are we insane?

Wed, 26 Aug 2009 15:05:29 -0800

The last thing the world needs is another damn URL shortener. There is no money to be made in this business, yet a new one launches pretty much every day. They're all mostly the same, offering some basic analytics like how many people clicked your link, and maybe the top referrers and geolocation of these clickers.

That's all well and good but there's something lacking: you don't get any data on your visitors after they arrive at your site. And that's the data that really matters. Who cares how many people clicked if you don't know what they did when they actually got to your web site?

That's why we're launching our own URL shortener, clicky.me. Here's what makes this product special:

The best analytics, period

Every visitor who arrives at your web site via a clicky.me link will be tagged with that link. This means you can filter/segment your visitors based on this URL, just like you already do with countries, referrers, etc. This gives you the full spectrum of segmentation data that Clicky offers, providing valuable insight into the actual impact of any link you share.

We made a brief announcement on Twitter about this, asking people to click a link to help beta test clicky.me. Here are the results, about 1 hour later:

If we had used another shortener such as the ever popular bit.ly, the only data we would have is the number of visitors, and the top referring domains and countries, as are shown in our screenshot above. That's great, but look how much more data we get from clicky.me!

We can get all this extra data because Clicky Analytics continues to track these visitors after they have arrived. We can see the bounce rate, the average pages per visit, average time per visit, and more. As you can see from the drop down menu that's open above the countries box, I can also view things like which goals these specific visitors completed, what web browsers they use, what cities they live - anything I want. Plus, since this is Clicky, you can actually see every individual visitor who arrived at your site from this link, so you can study them individually as desired. No other shortener gives you anything even remotely close to this.

The most accurate data

Just like Clicky only tracks human visitors, so does clicky.me. This means the numbers you get from us will be much more accurate and meaningful. Other shorteners log every hit, regardless if it's from a search engine, bot, or otherwise. clicky.me only tracks humans, so you'll know exactly how many people actually viewed your link.

No, we're not relying on Javascript for this :P We're doing it internally with a supar seakrat algorism.

Tons of ways to get your data

You can view the stats of your Short URLs by going to Links, then Short URLs. You can add the new Short URLs module to your dashboard (coming soon!). You can grab the data from our stats API, with type=shorturls.

Tons of ways to create short URLs

You can go right to clicky.me and paste in a URL. You can use our bookmarklet (available on the main clicky.me web site) to prefill both the URL field, and the site that the analytics will be attached to. You can create a short URL right from the Short URLs stats page (coming soon!). And we'll be launching an API in the near future to create them as well. We didn't consider the API a make or break launch feature, because we don't expect any of the major twitter application developers to integrate our service with them, since it requires an account. But we'll still add one eventually.

An actual business model (gasp!)

While clicky.me isn't going to make us money directly, it adds a lot of value to the service we already make our money from: Clicky Web Analytics. Any time we add value like that, more people upgrade to our paid services. As long as Clicky is around - and it's not going anywhere - clicky.me will be as well, so you can depend on us to keep redirecting those links well into the future.

(clicky.me is free for all users of Clicky Web Analytics. However, free accounts are limited to shortening 1 URL per day. Paid accounts have no limitations.)

Semantics, baby

Who can resist clicking a link that says clicky me right in the URL? Your click-throughs are going to skyrocket.

Well, that covers that. Enjoy!

Large date range queries now 400-900% faster for the average site

Tue, 18 Aug 2009 14:10:45 -0800

One of the biggest complaints we receive is how long it can take to look up historical data that spans a range of more than a couple of days. I want to explain why this was and what steps we have taken to fix this.

Previously we had only been storing summary data on a per-day basis. So if you wanted to view an entire month's worth of searches for, say, July 2009, our system would have to do 31 queries (one for each day) to get all of the searches for the entire month, then sum all that data together before outputting anything. This is inefficient and could take quite a while.

To fix this, we have added systems that also summarize all of your data on both a weekly and monthly basis. So now if you do a query for July 2009 searches, it only has to do one database query, instead of 31. However, because there is more data in that single query output, the time it takes will be longer than just 1/31 of the original. For the average site it will be about 10-20% of the original size of data. (This is relevant because the more data it has to read off the disk, the longer it takes to get it). This means for the average site, large date range queries will take just 10-20% of the original time that they used to. That's a huge improvement!

The same thing applies to full week (Sunday - Saturday) queries. Instead of having to do seven queries per week, it only has to do one. And all of this works fine for spanning multiple weeks and months also.

(We've been running scripts in the background the past week or so that have been calculating summarizing this data into weekly and monthly chunks, all the way back to Jan 1. All servers are done with this, except db7, which should be done within 24 hours. At the end of each day going forward, it adds your daily totals onto the week and month that that date belongs to. This means that for data prior to Jan 1, these optimizations will not apply).

What if your date range doesn't fit exactly into full month or full week increments?

No problemo. We have designed this system to be very flexible, using the most optimized queries possible for whatever query you throw at it. Let's pretend instead of July 2009 (1-31), you queried for May 31 - July 15.

Here's how our new system deals with this query. This is a bit technical, but it's worth a read to learn how the system optimizes the queries, which will also allow you to create date ranges that are as optimized as possible.

First, it figures out all of the individual days in this date range. In this case, there are 46 full days. This is how many queries would have to be done under the old system.

Next, it checks if there are any full months in this date range. In this case there is one (June 2009), so it removes all of June from the daily queries, and adds that month to the monthly queries. We are now down to 17 queries, from 46.

It then checks if there are any full weeks (Sunday - Saturday) in the remaining single days. In this example, July 5-11 is the only full week. So it removes those 7 days from the daily queries, and adds that week to the weekly queries.

We are now at 11 queries - 1 monthly, 1 weekly, and 9 daily. This is down from 46 daily queries originally, which is ~75% less queries that need to be run, which will result in much faster response times. Hope you enjoy.

Major updates to Spy

Mon, 20 Jul 2009 16:25:50 -0800

We've just released some great updates to Spy. We made a 5 minute screencast explaining everything, so watch that, or read more below the video embed.

The biggest thing you'll notice is the new map, which adds a lot of life to Spy. Each dot on the map represents a visitor on your site right now. As visitors come and go, dots will appear and disappear.

When a visitor performs an action, a popup box will display next to them on the map with more details. The map can be zoomed in for greater detail, and the popup boxes will also have more detail in this view.

The popular data box is now on its own as well, instead of being part of the action stream. Previously you had to choose between seeing the actions or seeing the popular data. Now you can see both at once.

If you click any item in this box, it will apply a filter that applies to everything Spy, including the map. However, one change we made is that the visitors online figure at the top now always displays the global value, rather than relative to the visitors. We thought it was neat to have that value relative to the filters but it caused too much confusion, particularly when a filter for just one visitor was applied.

The visitors sidebar at the bottom now includes the time they arrived by default (before you could only see it when you zoomed in on them), and it also shows you how many actions they have performed so far, which helps find the most active visitors, which are typically more interesting to watch.

Let us know what you think!

Recent updates to Clicky

Mon, 22 Jun 2009 19:10:39 -0800

We've silently been releasing new features and bug fixes over the last 2 weeks. I wanted to let everyone know about the recent changes, because it's doubtful anyone has noticed all of them.

Goals

Official support for goals with outbound links or sub-domains
More than once goals. By default, any goal can only be completed once per visitor. For certain types of goals, this may not make sense. Now there is an option to allow a goal to be completed as many times as it actually happens for an individual visitor.
The main goals page can now be sorted by column (e.g. sort by best conversion rate, most revenue, etc)
Goal and revenue data now show up in the basics dashboard module, as well as when filtering visitors

Visitor filtering

More data available up front. When applying filters, we've always shown you summary data (the exact same data that's in the basics), but to get more data you had to choose a drop down menu to view e.g. top countries for the filtered visitors. Now we show you top referring domains and top searches by default, as we feel that's the most important data to most people, but you can choose from a menu to view other data as well. We'll add a preference in the future to permanently set which data you want to view by default.
Prettier additional data. When viewing additional data before, it was a custom coded, fairly generic table. Now it's being sent through the same functions that create most of our graphs so it looks a lot better:
Bug fix: Viewing top links for filtered visitors now excludes search engine referrers.
Bug fix: Comparing one visitor segment to another was broken, now it's fixed.

Other changes

We now track daily values for actions types seperately (page views, downloads, outbound links, and clicks). You can see the daily values in the basics dashboard module, click the expand link. (Note: Since we only track additional action types for paying members, this link will only show up if you have a premium account).
Our new homepage is up, including the testimonials you all gave us back in April.
There's always been an option to hide ISP's in the visitors list. There's now an option to also hide them in the main hostname/organization pages. This is available in your site preferences.
Tracking code now has automatic support for SSL pages. This probably doesn't affect many existing users as they have already installed the secure tracking code if needed, but we receive quite a few emails from people who ask why our tracking code generates warnings in MSIE when used on secure pages. This was always caused by using the normal tracking code, instead of the secure version. So, now we've changed it so it works out of the box with both secure and insecure pages without having to worry about using the right code, because now there's just one code! If you want to use this, grab a fresh copy of the tracking code from your preferences.
Fixed a bug with the API that created some data formatting errors when mixing different types together (e.g. tally and popular data, or visitors-list and popular data, in the same request).
API requests for visitors-list now include the landing page for each visitor.

Announcing Contenture.com, a new micropayment service from the creators of Clicky

Tue, 26 May 2009 12:49:12 -0800

We're just pleased as punch to announce the launch today of Contenture.com, a new micropayment service from the same team behind Clicky Web Analytics.

There have been countless micropayment systems developed over the years, and they all have one thing in common: Serious Fail. They are too complex, both for a site to integrate, and for a visitor to use. But micropayments have become a hot topic again, with many major sites closing down because their sole revenue source - ads - just doesn't pay the bills.

We saw an opportunity here and jumped at it. We already monitor sites with Clicky, and we got to thinking - what if there was a micropayment service that worked the same way? Users pay a monthly fee into the Contenture pool, then we monitor what sites they go to, and automatically distribute each user's money to those sites every month based on how often that user visited the site. This makes for a completely seamless experience, and we believe it's one that will finally bring micropayments to the masses.

So why would a user pay for Contenture? That's where the second feature of Contenture comes in - premium microservices. Contenture lets you offer exclusive features to paying Contenture members, such as removing ads, giving exclusive access to certain content (e.g. /archives/), giving priority access to new content, commenting privileges, and more. All of these features are optional though, which brings up the question: Why would you not install Contenture on your site? By just putting a piece of Javascript on it, you will automatically make money from every paying Contenture user who visits your site, even if you never enable any premium feature.

Every new paying user benefits the entire system. For this reason, Contenture has an affiliate system, like Clicky does. For every new paying user that signs up through your link, we'll pay you $1. But as a special promo to Clicky users, we're going to offer you double that affiliate rate for a full year. When you register, just enter in clicky in the promo code section. The first 100 accounts that sign up with this code will get the special affiliate rate.

We plan to integrate Contenture fully into Clicky in the very near future, as an alternative payment method. We will also be disabling ads on Clicky for free accounts that sign up for a paying Contenture account.

So what are you waiting for? Go register for Contenture, and let the Micropayment Revolution begin!

Do us a favor?

Tue, 19 May 2009 09:48:28 -0800

Kaspersky is an anti-virus and online security software suite. They have decided that getclicky.com is a malicious tracking service and have blocked access to it for anyone using their software. I have contacted them more than once and they refuse to acknowledge the problem and just send me generic form responses about the steps I need to take if I'm having problems accessing the internet. However, I have been contacted by at least 10 unique users who have pinned down the problem to their software. When they disable it, suddenly they can access getclicky.com. Hmm, amazing coincidence, that.

I figure, if they suddenly get a bunch of support requests specifically for problems with getclicky.com, maybe they'll fix it. It doesn't matter if you actually use this product, we're just asking you to submit a support request for this problem. Tell them you use their online security software and when it's enabled you are unable to access getclicky.com, but you can access all of the other sites you regularly visit, and when you disable their software, you can then access getclicky.com.

We're very thankful to any of you who are able to help us out with this issue. It is a serious issue, as this product has a fairly large install base.

Consider this a birthday gift from you to me. I turned 31 today and I'm feeling awfully old. :(

Twitter Mania - the best update we've ever released

Thu, 23 Apr 2009 02:27:05 -0800

If you are a Twitter user, or at least understand the value of the data that's available from their real time search engine, this update should blow you away.

We use Twitter to heavily monitor what our competitors are saying, and what other people are saying about them. Amongst other things. You can get any search result from Twitter via RSS, which is handy, but it's just a raw stream. I've wanted to summarize this data for a while now so I had it all in one place and could easily sort through it. I'm not a fan of any of the tools currently available, so I decided to bake one right into Clicky.

Twitter brand monitor

Hence, Twitter brand monitor is born. I don't think much explanation is needed. Everything in the familiar dashboard boxes summarizes all of the Tweets at the bottom, and clicking on any of them will filter the Tweets by that criteria. You don't need to run a business or have competitors to find this tool useful. Use it to monitor any topic you want!

Try out the demo here

Alerts!

One feature request we get quite a bit is some kind of alert system. The typical alert system is via email. Boring! I mean sure, we did that, but we also took it a step further and can send you alerts straight to your Twitter account. (You will need to follow getclicky to receive alerts, however, as they are sent via Direct Message). We use the shortest links possible to link you straight to the session ID for this visitor, so they fit perfectly within the 140 char limit.

The alert system is fairly basic right now (you can only setup alerts for searches, referrers, goals, and campaigns) but we plan to add more options in the near future. Namely, alerts on tweets, custom data, downloads, outbound links, and page views. You can setup alerts in your site preferences.

Hourly data

One thing you may have noticed from the first screenshot is hourly data. Yes, we are now tracking hourly figures for certain types of data: visitors, actions, and tweets. This has been a huge request as well. Any time you are viewing a single date, and the data you are viewing supports hourly data, it will default to showing you that date, as well as the day before for comparison's sake. We already programmed it to use your dashboard pref to compare vs 7 days ago by default if that is your preference, but we don't yet have 7 days of data - so this doesn't happen automatically just yet.

UPDATE There is now a dashboard preference to disable showing hourly data by default. I think defaulting to hourly data makes sense for most people though, since Clicky is more about the right now than anything else.

Other tidbits

- Twitter monitor is a Pro-only feature. It sucks up a lot of bandwidth and space, analyzing and storing all of those Tweets. Alerts are currently Pro-only as well although we do want to offer that to everyone. We're going to be monitoring the alert system to make sure its resource usage isn't too high. If all seems well, we'll release it to everyone. Currently, any alert will max out at 100 per day, as we don't want to spam Twitter or your email servers.

- The Twitter monitor has a dashboard module as well, although it only shows you the total values for each search you have setup. You have to click through to each of them to see all the details. This basic data is also available via the API (type=tweets) and email reports.

- There was a bug with the API that was returning the incorrect unixtime value. This has been fixed. Your application may require an update if you were relying on this value.

- Many of you informed us of this post on the Google Analytics blog. They are messing with referring URLs again. We've already implemented a fix, although it wasn't live until just now. Hopefully you won't be seeing many more referrals from just http://google.com anymore.

- Replying to Tweets directly from Clicky - this will be coming soon. It didn't make it into the initial release.

- When viewing your Twitter search results, keep in mind that the summary data is only for the date you are looking at, which defaults to today. So if the search you have setup is fairly low traffic (e.g. averages just a few hits a day), changing the view to 7 days or 30 days will give you a better overall view of activity.

UPDATE Reply and Retweet icons have been added in the tweets list, next to each one. However these simply link to Twitter.com with the appropriate params to do as requested. A future release, we'd like to integrate that directly into our service, but that will require a bit of work to make it sexy. It will also require us to store your password in our database, which we don't really want to do.

Also, when viewing one of your searches, if there are more than 100 tweets, you can now view next page. I don't know how I managed to leave that off the initial release :)

We're looking for testimonials for our new landing page

Tue, 14 Apr 2009 16:09:25 -0800

Hi everyone, we're working on a new landing page for getclicky.com, and we're looking for your personal testimonials to showcase to new/potential users. We'll be putting our favorite 5 or so on the new front page, and there will also be a link to view more of them on another page.

If you want to participate, please post a comment here about why you like Clicky, and/or why you pay for it when there are so many other free options available. Be sure to include your name and web site! 2-3 sentences is best, but feel free to say however much you want. We may make minor grammar changes to some of them, but otherwise they will be posted word for word.

Thanks for your time, we love all of you dearly.

Microsoft's spam bot has been blocked

Thu, 09 Apr 2009 12:27:47 -0800

We've been getting more and more complaints of how annoying Microsoft's crawler for search.live.com is. In case you haven't seen it yourself, what it does is pretend that it's a person and spams your sites with tons of hits and fake searches. Check out the screenshot below to see the kind of traffic we see in our own stats every single day.

But no more. Anytime we see a referrer from serach.live.com, we check if the IP is in the range of known IP addresses for their crawler, and if so, we just ignore it.

Microsoft ought to be ashamed of themselves for this type of behavior. This has been going on for a long time (just search Google about it) but it seems to have gotten a lot worse recently. I don't know if they're trying to falsely inflate their market share when people view their stats or what, but whatever the reason, one thing is certain - I'll never use search.live.com for anything, ever.

Mega spy out of beta + new features for free accounts

Wed, 01 Apr 2009 18:06:05 -0800

Mega spy has come out of beta. It is now just Spy, and the old Spy is now called Spy basic. This also means the new Spy is now only available to Pro members, as was originally announced.

We didn't have time to make as many improvements as we wanted but don't worry, we will be visiting this again in the near future. We did manage to squeeze in the top 2 most requested features though - linking to external searches, and showing the visitor's location in a tool tip when you hover over their flag in the right hand side.

For consistency we've been trying to make it so ALL external links in your stats are accessed via that little grey arrow you all know so well, and all other links (in your stats) are generally for navigation or to apply visitor filters. Spy's new layout was too cramped we felt to do this all with, so now all external links in the action stream (to your own pages, or to referrers/searches) just link directly to those external pages.

New features for free users

We've been wanting to offer some more features to our free users for a while now. We really wanted to offer the basic version of Spy to all of you - but it just sucks up a ridiculous amount of resources supporting that, so we can't do it. The load averages on our tracking serveres were averaging more than 4x their typical load during the last 2 weeks when all sites had access to both versions of Spy. So we can't give you that, I'm sorry.

But we did give you 2 more features: Bounce rates, and IP tags and filters (but you are limited to 10 max). Also most active visitors was for some reason a pay-only feature, which I don't ever remember deciding on, and it's not like it's the most amazing feature in the world that will get everyone to sign up for paid service. So that, along with bounces and ip tags/filters, are now available to free members.

Mega Spy now in public beta

Mon, 16 Mar 2009 23:53:24 -0800

Mega Spy is now in public beta, which means all users have access to it.

Go to your Spy page, then click the test the mega spy beta link to check it out. It still has some issues and is not yet feature complete, but we've drastically increased the performance since the original beta was released on Friday. It now runs very fast on all browsers, including MSIE. This will also really help you out if you have a slow computer.

By this coming weekend we plan to add in support for goals and campaigns, and maybe organizations and hostnames, if the performance hit on the server end isn't too bad. We'll also be trying to add a nice highlight effect on the popular data view, so you can more easily tell which data is actually changing when it refreshes.

Free accounts also have access to both versions of Spy during the beta period. Once you get a taste, we think you'll have a hard time resisting that upgrade. Mega Spy will be a Pro-only feature when the final version comes out. Blogger accounts will only have access to the old Spy. Sorry, but all the new stuff that the new Spy is doing is quite intensive on the server end. If we could offer Clicky for free to everyone - we would!

Please leave all feedback as a comment on this post. Enjoy!

Mega Spy preview + private beta

Thu, 12 Mar 2009 14:14:20 -0800

Mega Spy has taken longer than planned to develop but I promise the wait will be worth it. We're still not quite done, but we know how eager you are, so we're releasing a beta version to get feedback before we finalize the 1.0 release.

Mega Spy will be a Pro-only feature, but during beta it is open to all users. To be eligible for the private beta (we'll open it up to everyone else in the next 3-5 days), all you need is an existing Twitter account with at least 10 followers. To apply, simply tweet the following message:

http://getclicky.com is the best web analytics service on the planet!

Or, make up your own tweet, and tag it with #megaspy. For example:

http://getclicky.com - live web analytics FTW! #megaspy

We will be monitoring for these tweets via Twitter's search engine, and will send a direct message (tonight or tomorrow) to the first 50 eligible participants with instructions for accessing the beta.

UPDATE - private beta is now closed, invites have been sent out to everyone.

Mega Spy still has some bugs, and it needs some speed optimizations, so keep that in mind! It has been tested with Firefox 3, IE7, Opera 9.5, Safari 4, and Chrome 1.0, and works in all of them. Any lesser version may or may not work. A lot more things are going on in the background compared to the old Spy, so a browser with stellar Javascript performance will enrich the experience. MSIE is pretty bad, Firefox and Opera are decent, Safari and Chrome fly. Once Firefox 3.1 comes out though it should fly as well.

Here's a couple screenies to whet thy appetite. The second screenshot shows my absolute favorite feature of Mega Spy - seeing what pages on your web site people are looking at RIGHT NOW. As people move around your site, the values update on the fly. It's a blast to watch. You'll also notice that you can filter the data you want to see by many different criteria. When filters are applied, the entire view is relative to those filters. You can also filter down to a single visitor to watch just them navigate your site, but that's not shown here.

Filter visitors by page, better segmentation - and more!

Sun, 22 Feb 2009 19:46:31 -0800

We just released an update that's been two months in the making. On the surface, there are only a couple of major differences you'll notice, but they are very cool ones for sure. What took so long was designing and implementing an entirely new system for how we store page URLs, and converting all of the existing data into that new format (many hundreds of millions of database rows). This fixed a couple of annoying problems we've had with the system from the beginning, and let us build some awesome new features on top. Let's get to it.

Filter visitors by page!

You can now filter your visitors by a page they have viewed. This has been a big request. Works in the API too, just use the parameter href with the visitors-list type.. You should only be sending the path and query (e.g. everything AFTER http://yoursite.com), unless the page you want to filter by is on a mirror or sub-domain, or it's an outbound link. For example here's how we'd grab all visitors who saw our /users/ page, and all visitors who clicked our company's web site in our footer (http://roxr.net)

http://api.getclicky.com/...type=visitors-listhref=%2Fusers%2F
http://api.getclicky.com/...type=visitors-listhref=http%3A%2F%2Froxr.net%2F

New visitor segmentation!

When you filter your visitors by anything, the only type of summary data you used to see was basically everything from the basics dashboard module, but applied to these visitors. But now you can see anything you want about them - top searches, top countries, etc. This is particularly useful when filtering by page. Now you can see the top searches hitting any page on your site for example. We plan to add on to this quite a bit this coming week, so for now it's slightly rudimentary, but it performs its purpose well. Click on the red View these visitors... link (shown below) to choose what type of data you want to see. This feature is for Pro members only.

Links so hot you'll want to eat them!

Links to external content now hide the protocol by default, which saves precious space (particularly on the dashboard). We've also changed it so it highlights the domain name and the rest is slightly faded. This is inspired by the Google Chrome address bar - I really like how easy it is to pick out the domain when looking through links now.

Also, when viewing your visitors list, there is now an icon next to the referrer to indicate which type of referrer it is (search, social media, etc).

Links to external content have changed!

Used to be when you viewed your links or your content, clicking on an item would take you directly to the external page. That's useful but inconsistent with how the rest of the site works, where clicking on anything else (search, country, etc) would instead filter your visitors by that item. Links and content have been changed to match this behavior in all areas of the site. Now, anytime there is an external link, there will be this arrow next to it. Click that arrow to be taken to the external link. You can see this in the screenshot above.

Much better support for sub-domains and mirrors!

If you had multiple domains pointing to the same site, or had the same code on a bunch of sub-domains for one site, you couuld still track that fine, but the content we tracked was never linked to the right domain that it was actually viewed on. Now, we'll store the domain it was viewed on, if it's anything other than what you have in your hostname field in your site prefs. Example:

No more dupes!

The new pages system gets rid of the dupe problem for your content. Unique pages used to be determines by a combination of the URL and the title. This resulted in a lot of duplicate entries for the same page, because of reasons like the visitor having Javascript disabled (no title), the visitor using an old browser sending us the title formatted slightly differently, etc. This will happen no more, although it will only affect your stats from this point forward.

In conclusion!

That covers all the new features that are obvious on the surfage. We hope you love them like we do, but as always leave your feedback, good or bad, below.

Also, there are a few gotchas that might catch you off guard. Let me cover that. First, pages will not have any titles until they get a page view under the new system. This is related to getting rid of the dupes problem. Give it a few days and almost all of your content will have titles again. Obviously your popular content should already.

Second, filtering by page may not work right away for all of your pages, because there were so many duplicate IDs. If your ID for a URL was not the first ID for that URL in our system (on your database server), then they don't match up right in the database. So filtering by a page URL may only work for some of your pages from this point forward (as I seem so fond of saying). I do have an idea of how to fix this so filtering will work for all pages, for any date, but it'll have to wait until tomorrow so I can test it.

Last, the new segmentation drop down menu doesn't work right in MSIE. I could not figure out the problem before release, but I will try my hardest to get it fixed on Monday.

Ok, now go enjoy the new features!

What are your top 3 feature requests for Clicky?

Wed, 18 Feb 2009 20:11:10 -0800

We'd love to hear your top 3 feature requests for Clicky. This service has been in non-stop development for almost two and a half years, and there are still a billion things we want to do. Our todo list is 10 miles long, so it's hard to decide what to do next sometimes. That's where you come in.

Excluding all the stuff that's coming on our March roadmap - which includes things like filtering visitors by page URL, per-page analysis, and super mega ultra spy - what are you top 3 feature requests?

UPDATE This isn't a poll because we don't want to restrict your choices. We want to know exactly what you want, even if it's absolutely ridiculous.

Feedburner has been fixed

Mon, 16 Feb 2009 16:13:33 -0800

All of you who have moved to the new Feedburner platform have been noticing that Clicky can no longer access your Feedburner data. I'd like to rant about the embarrassingly awful job Google did with this migration, but I'll save my efforts for something more useful. What matters is that it should be fixed if you are on the new platform. If you continue to have problems, please contact us or leave a comment here - but it should be working for everyone now!

UPDATE One of our users asked about why their FB data from more than a few months ago (around when they migrated) had disappeared. I just want to clarify this is a side effect of your migration. When you move from the old Feedburner to the new one, your data doesn't carry over - you start fresh, as if you had just signed up. I mean, that would just way too convenient, wouldn't it, to have your data carry over? Surely you weren't expecting them to spend the 15 minutes it might take to write a script that exported your data from the old database and imported it into the new one?

Lots of new tracking features, including OnClick goals and internal script logging

Fri, 13 Feb 2009 02:42:35 -0800

Lots of new stuff related to tracking being released today.

OnClick goals

You can now declare goals from Javascript events, aka OnClick. This has been a big request. Docs.

Internal script logging

We have documented the API that our tracking code uses to log data, which means we officially support this now. You can log any data you want to it, anytime you want, from anywhere. Validation is done with a new sitekey, called the Admin sitekey. This is available in your site preferences. We know this will be insanely useful to all of you who use internal redirects for outbound/affiliate links. I get questions on that all the time, so you should love this! We also wrote this PHP function as a simple way to add this functionality to your own app. Use something other than PHP? Rewrite and send it to us, we'd love to offer it to our users.

I'm pretty sure that Clicky is the only service in the world that offers this type of functionality. Please tell me otherwise if this is incorrect.

Better customized tracking

There is now just a single Javascript variable called clicky_custom to deal with all aspects of customized tracking (adding custom user data, changing the pause timer, etc). Much cleaner and easier to work with, and a few extra features too. pageview_disable to disable the automatic logging of a page view on initial load, and href and title properties to customize the URL and page title that is logged for that page view. The docs have rewritten from scratch with all the new stuff added in. Backwards compatibility is built in so you don't need to change any of the existing features you're already using, such as custom data tracking. To use the new features, you'll need to update to the new methods though.

Update sessions after-the-fact

Using the same API mentioned above, you can update sessions well after they are expired, to add your own custom data to them, and declare goals and/or revenue. I get many emails from people who can't add tracking code and/or javascript variables to their shopping cart pages. Now you can update sessions later if you want, to declare the goal and revenue for that visitor.

It's 3am. Time for bed.

Google's new Ajax-powered search results breaks search keyword tracking for everyone

Tue, 03 Feb 2009 09:50:26 -0800

Do you run a web site? Do you use a service such as Clicky, Statcounter, AWStats, etc, to monitor the traffic to your web site? Do you find it useful how these services can tell you what search terms are leading visitors to your site? Have you ever wondered exactly how they are able to do that?

No, you probably haven't. But that's ok. Just know this: a major update that Google is testing has completely broken the ability for any external analytics service like Clicky to determine the search query used by a visitor arriving at your web site. Why would they do such a thing? Who knows. They aren't talking.

Knowing the search terms that are driving traffic to your web site is probably the #1 reason to use any kind of traffic monitoring service. And Google owns a huge chunk of the search market. If this update goes live for everyone, it effectively means that 2/3 of all searches leading to the average web site will be a complete mystery. This is huge.

So what is the problem exactly? Normally when do you a search on Google or any other search engine, the search term used become part of the URL. A search for Clicky, for example, gives you this:

http://www.google.com/search?q=clicky

When someone clicks a search result on that page, that URL above is sent as the referrer to the target site. An analytics app running on the target site can parse the referrer string and extract the word clicky, and store that as a search that occured for that site. This is obviously very useful.

Here's what the new search result URLs look like with the new Ajax feature:

http://www.google.com/#q=clicky

See how there's a hash mark # in there now, and the q=test is after it? The problem is that web browsers don't send anything after the # in the referrer string. This means organic searches from Google will now show up as just http://www.google.com/, with no search parameters. In other words, no analytics app can track these searches anymore. I started noticing lots of hits from just http://www.google.com/ recently in our own search logs. I thought maybe it was just a bug with Clicky. But then one of our users contacted me about this article, and my jaw about broke from hitting the floor so hard.

This change isn't live for everyone yet so if you go to google.com and do a search, you may not see it in action. But it is happening for me when I go there and do a search.

So what can we do about it? If you run a blog, write about this. Submit this story or your own story to large tech blogs like TechCrunch, CenterNetworks, ReadWriteWeb, GigaOm, etc - no large site has written about this yet, and one of them needs to. Post in Google's Web Search forums (there's no way to directly contact them about web search unfortunately). Do anything you can to spread the word and let Google know that this is not acceptable.

Update: Google is watching

Clicky roadmap through March 2009

Mon, 26 Jan 2009 09:06:23 -0800

We've got a lot of buns in the oven right now, many of which are waiting for that database transform to be completed. All database servers are done, except for the one, the only, db1, which is lagging badly at barely 33% done. It should be done by this weekend though. (If you have a site on db1 and are tired of it being laggy, read this).

It's been fairly quiet around here recently, but we have in fact been working like mad behind the scenes, as usual. So I wanted to share with you the features we have planned for the next couple of months, all of which we plan to have out by the end of March 2009.

In depth per page analysis, e.g. for your /contact page, view the top countries, top searches leading to it, top referrers to it, percent of views that were landing/exit (meaning first or last action of a session), average time spent on page - ETC!
Filter visitors by number of actions, including support for less than, greater than, and equal to (e.g. at least 10, exactly 1, less than 3, etc)
Filter visitors by amount of time spent on site (at least 5 minutes, less than 1 minute, etc).
Filter visitors by revenue (at least $1000, exactly $100, less than $100, any amount of revenue, etc)
Filter visitors by a page URL (e.g. all visitors who viewed /contact)
Goals: Support for javascript/onclick events to complete a goal
Goals: More up front details about revenue in views outside of the main goals page
Goals: View history (timeline graph) for conversion rate and/or revenue for any goal
API for adding data to a session after the fact (e.g. declaring a goal as completed, adding/changing revenue, etc)
Clicking an item under links or pages will by default invoke visitor filtering for that item, rather than going to it (e.g. clicking /contact would fitler your visitors by that page, instead of going to the contact page on your web site). To actually go to the link/page, there will be an arrow next to the item to click, if you want to do that instead.
Much better mirror/sub-domain support
Hourly data figures for visitors and actions
Track figures for page views, outbounds, downloads, and clicks seperately (dashboard module the basics will have expandable view to see this extra data for the total actions value)
Super Mega Ultra Spy (this will blow you away)
Visitor groups - kind of like a saved filter, e.g. all visitors from the US who arrived via Google and use Firefox. Groups will be able to be used as funnels for goals, which will be amazing.
Interface to update your credit card details, instead of forcing you to sign up for a new subscription when your card expires or you need to change it for whatever reason.

As you can see, that's quite a number of things coming soon. So be patient, and it will all eventually be yours!

The new user homepage

Mon, 15 Dec 2008 20:11:25 -0800

We've had all sorts of requests to improve the user homepage to give a better summary of all your sites with one glance. The screenshot above shows you what we have come up with, which you can see for yourself on your user homepage.

This is basically the Super Mega Uber Dashboard. The Dashboard of all dashboards. But hopefully not as controversial as when we launched the new site dashboard :) Our goal was to make the most amazing, user-friendly, as-much-data-as-possible-but-still-clean-and-efficient interface that could possibly be made to overview the traffic on all of your web sites on one page. There is nothing else like this on the market. Well, not yet anyways. Give Woopra and Piwik a couple of weeks - they'll soon have their own imitation of it.

Here's a quick run down of everything:

By default, for each site you will see today's visitor and action tally, and a graph on the right of the visitor count for the last 30 days.
You can customize the type of data shown for today. Options include visitors, actions, bounce rate, and goals (which includes revenue, if applicable).
Clicking on any of the links under today will refresh the graph with that data type. By default it shows visitors but if you click actions, for example, it will refresh with the daily action tally.
You can change the date range for this tally data with the same options on the normal dashboard - 30, 60, 90, or 180 days, and weekly (26 weeks) or monthly (12 months).
You can sort your list of sites by alphanumeric (default), or any of the data types you have enabled for today - e.g. most visitors, most actions, etc. Your sort preference is remembered until you change it at a later time.
You can also view popular data types for each site. By default the options are searches, links, and pages, but you can customize it to have any and all data types available in the drop down menu.
You can customize how many results are displayed for popular data. Default is 10, but available options range from 5 to 50.
There are global menus (in blue) at the very top. If you change the data type to searches for example, all of your sites will refresh to show your top searches. Very usefeul and amazing.
There is a global date menu shown in this screenshot, but there is a bug with a few of the major web browsers that we haven't been able to fix yet (Surprisingly enough, MSIE is the only browser that actually works 100% correct for this menu. Is it just me or are pigs flying?). Until that is fixed, the global date menu is disabled for all types. But we know you'll find it uber awesome once we get it working right. Being able to, for example, refresh all of your sites with just a few clicks to show the monthly visitor tally, or the top searches from the last 30 days, is insanely amazing.
Option to hide the forums from your user homepage, if you find them distracting.
Customizaton, sorting, dates, and global menus are only available for premium members. Since free members are limited to 1 site and 30 days of history anyways, this really isn't much of a limitation.

Thoughts? We're really happy with it from our own internal use, but we fear our page view limit is going to drop, with so much data available on one page. We know, the page view is dead and all, but still, it's a number that's hard to let go of.

Update

By popular demand, you can now disable the line graphs. This has the unfortunate side effect of disabling the popular data options as well, since they shared the same space previously. When disabled, you will see yesterday and last 7 days instead, as shown below. This brings it back to being quite similar to the old userhome, except it looks a lot cleaner, and you still get the sorting options and the ability to choose a few additional types other than just visitors and actions.

Keep tabs on your twitter followers from Clicky

Mon, 24 Nov 2008 23:31:57 -0800

We've partnered with TwitterCounter.com to integrate their data into Clicky, via their just released API. This service monitors the number of followers you have on a daily basis. Your Twitter account isn't necessarily tied directly to your web site, but we have so many bloggers and other people who use both Clicky and twitter (us included) that it made sense to integrate this.

All we need is your twitter username, which you can enter in on the site preferences page. Then your follower count will be integrated into the basics dashboard module (you'll notice we also added your FeedBurner count here too):

We also created a simple dashboard module for twitter, if you want to always see the graph by default. (Hot tip: you can also click the red/green percentage next to any item on the dashboard to see its history, including twitter and feedburner):

Check out the Clicky demo to see this all in action!

Anyone have ideas for other third party services that it would make sense to integrate? The two we thought of recently:

- Google PageRank - unfortunately there is no official API to get this data and Google frowns upon any sort of hax0rz to obtain it. Also this number only changes a few times per year at the most for any given site.

- Technorati authority - this service used to be huge but I don't think anyone takes it seriously anymore.

Max date range is now 1 year for all Pro members

Sat, 08 Nov 2008 13:21:44 -0800

By far one of our most requested features has been to increase the maximum date range limit of 31 days. We've been delaying this because of the performance penalty our servers take when accessing such a large range of data, but we can ignore your requests no longer. As of now the maximum date range has been increased to 1 year. However, because this does take a lot of extra processing, we can only offer it to our Pro members. And be patient. If you do an entire year, it can literally take several minutes to get the data!

We'll be monitoring how this affects the performance of traffic processing. If it's too big of a hit, we may have to decrease it to 180 days. I don't want to do that, 1 year is kind of a magic number, but we'll see what happens.

Also, the dashboard is still limited to 31 days. Pulling up an entire year's worth of data for potentially ALL data types might cause our servers to explode, so we can't offer you that. But on all individual pages other than the dashboard, and including the API, the max range for all Pro users is now 1 year. When you click the Custom data range link in the date drop down menu, it will tell you the max date range for that page. On the dashboard it will always say 31 days, but on all others pages it will say 366 (1 extra day for leap years).

Clicky is now available in 12 languages

Tue, 04 Nov 2008 16:08:01 -0800

The translations are live! Clicky is now available in Danish, Dutch, English, Finnish, French, German, Hungarian, Portuguese, Russian, Spanish, Swedish, and Turkish. You can set your global language from your user homepage by clicking the edit link in the top right corner. This preference will apply throughout the entire getclicky.com. You can also set this preference on a per site basis, from any site's preferences page, if for any reason you want one or more of your sites to be in a language other than your main selection.

Note that none of the languages are fully complete yet, although a few of them are really close. Any phrase that is not yet available in your selected language will simply be displayed in English instead. Also note that we haven't fully implemented the translations throughout the entire site yet. It is about 95% on all pages having to do with your stats, but I haven't gotten it into any of the /user/ pages yet. I'll be working on further integration throughout the week.

Now that you can actually use some other languages, you may be feeling more motivated to help us translate them. We'll soon be rewarding the top contributors with free Pro accounts.

We think this system is going to work really well. It auto-updates every night with any new phrases that have crossed the approval threshold, based on user voting. As new phrases are needed, or as we need to change the copy in certain areas, these will be immediately available on the main translations page and will make it into the live service automatically once they have received enough positive votes. This will help ensure all languages stay up to date.

We definitely expect there to be some inconsistencies within languages, and some things that are just plain wrong. Please contact us with anything you notice that needs fixing. One thing we need to add is the ability to assign admins to certain languages, so they can add, edit, or change phrases as needed, without users having to vote on them. This will make them more accurate and consistent. If you are interested in being a language admin, please contact Sean.

Translations launching by the end of this week - this time, for reals

Tue, 28 Oct 2008 11:15:01 -0800

We launched our crowd sourced translation framework over 6 months ago. Since then, you've submitted over 6,000 translations, and what have we done with them? Nothing. I apologize. When you run a service like this, things just keep piling up, and it's hard to prioritize.

But this time is different. We are focused 100% on getting this functionality launched by the end of this week. Since April some of the copy on our pages has changed, and we've added quite a bit of new things too, like goals and campaigns. So there are about 150 new phrases to translate. And we'll probably be adding some more as the week goes on.

If you want to help out, the translations page is here. As we mentioned before, we'll be rewarding the top contributors with free Pro accounts, after the system has launched. Thanks for your patience!

Goal and campaign updates

Wed, 22 Oct 2008 14:30:59 -0800

We just released some updates to the recently announced real time goal and campaign tracking.

But first, an unrelated announcement. Custom data can now be logged incrementally, instead of all at once. Previously when you wanted to attach custom data to a visitor, we only let you do it once. This wasn't a huge issue for most people but we certainly received complaints about it. So now this restrictions is removed. You can log custom data to visitors incrementally as needed throughout a session, which I know will be really helpful to some of you.

Ok, onto the goal stuff:

New features

Goals and campaigns have been added to the API, the mobile/iPhone services, email reports, and Spy.
Conversion rates for goals without funnels are now calculated using the total number of visitors from that date/range
Unique icons for goals and campaigns! The icon you choose is shown in the dashboard modules, in the main goal/campaign pages, in the visitors list (where it is most useful), and in individual visitor details.
In the visitor's list, we were using a blue flag to indicate a completed goal, and a red flag for an incompleted goal. Adding unique icons made us change this however. Now an incomplete goal has the same icon as the completed goal, but it's faded out. Example:

Bug fixes

Case insensitive matching was only working for wildcard matching, and not exact matching.
Trend calculation on main goal page was always -100%
Removed dollar sign next to monetary values (not all of you use dollars after all)
Revenue totals were incorrect for total amounts greater than 1000.

Two features that didn't quite make the cut: First I was trying to make a better dashboard module that showed you more info about goals, but I just couldn't make it look good. I'm a big believer in aesthetics and until I can get it up to my standards I'm not going to release it. Second, we've had a few requests for supporting manual goal completion via ajax/onclick events. This proved more challenging than I thought because our tracking script on the receiving end always expects there to be a page view or other action in there somewhere, which wouldn't be the case here necesarrily. It's a bit complicated to explain.

I really wanted to get these other fixes and new features out though, especially the new icons, so we went ahead with this release anyways. We'll try to add these two things into the next update!

Real time campaign and goal tracking

Thu, 09 Oct 2008 01:29:21 -0800

The wait is over. I locked myself up for 2 weeks and did nothing but work on goals and campaigns. Seriously, I'm about to pass out here. We're really happy to have them out, but we're also really excited because Clicky is the ONLY service that offers either of these features in REAL TIME!

These features are for Pro and trial account holders only. We made them as efficient as possible to process, but there is still a good amount of overhead involved. Until we can gauge how much this affects performance when we have a bunch of sites using it, we are limiting each site to a maximum of 7 goals and 7 campaigns. This limit will be raised in the future. Quick breakdown:

Campaigns

Real time campaign tracking!
Can be triggered by landing page or referring domain.
Wildcards supported for matching
Campaigns can be funnels for goals
Visitors who are attached to a campaign will have this icon next to them in your visitors list
Dashboard module and dedicated tab
More information (excuse the marketing speak)

Goals

Real time goal tracking!
Multiple funnel types: page URL, campaign, external search query (e.g. clicky), external referring domain (e.g. google.com)
Track revenue and costs: per site, per goal, and per visitor
Can use Javascript tags to force a goal to be completed (good for complex scenarios), and customize the revenue for an individual visitor (documentation is on the goal setup page)
Visitors who have completed a goal will have this icon next to them in your visitors list
Visitors who have gone through a funnel but NOT completed a goal will have this icon next to them in your visitors list
A visitor can't go through a particular goal more than once per session, but they can go through multiple unique goals per session
Wildcards supported for both the goal page, and the funnel
Dashboard module and dedicated tab
More information (excuse the marketing speak)

Ooh, added bonus: We've widened all /stats/ pages by an extra 100px, so there's now quite a bit more rooom for your data. Yay progress!

Campaigns and goals aren't yet in the API or iPhone/mobile services, but will be by week's end! Enjoy everyone, and please give us your feedback!

Updates to iPhone service

Thu, 25 Sep 2008 14:15:32 -0800

We just released some updates to the iPhone service. First, we added some new types: regions, clicks, and referring domains. Second, we removed some excess padding and margins so there is more room for the data. Last, we now properly support landscape mode, so the strings aren't as truncated. Changing between portrait and landscape mode on the fly works really well. I prefer browsing in portrait mode normally, but for some things, being able to change to landscape mode for just that page to see more data is super great. Hope you like it, this has been the biggest request we've had.

Up next for Clicky is campaigns and goals, which are coming along real well. They should be available by the end of next week!

We got a map

Fri, 19 Sep 2008 15:46:42 -0800

We re-organized things a bit, you may notice a new Locale tab. This is where all the country, city, language etc stuff has been moved to. We also added a new sub-tab, called Regions, but data only started tracking yesterday for it. When you first click the Locale tab, you will see a new map that we've added to the system:

The darker a country is, the more visitors you have had from that country, for the selected date range. You can mouse over any country to get the number of visitors. You can also click on a country which will zoom into that region:

We wanted to add zooming into individual countries, but the API for this product (FusionMaps) is so terrible and inconsistent that it would literally take about 5000 man hours of work to do that. Unless we wanted to spend a billion dollars, there are surprisingly few products of this nature. This one was has the most features and is by far the best looking so it's what we chose. If only the API was a bit more friendly...

We did however add in support for zooming into the United States, as we do have a lot of US customers. And there was actually consistency here, between the Maxmind database and the API, so it was easy to do. Looks pretty good, although I'm not so sure our international customers will be happy with us only supporting our own country. Sorry :P

States are part of the Regions data mentioned earlier, so you can't view data for this prior to Sep 18.

For everyone dying from anticipation for goals and campaign tracking, they are already in progress, don't you worry about a thing now.