Open Juice Blog

External Interface, Internet Explorer and Time Wasted

2010-04-15T10:52:56Z

Dear Googlers,

I leave this message for you in the case that you encounter a similar obstacle to your productivity. Internet Explorers 6, 7 and 8 do not like uppercase letters (or hyphens) in the object ID when embedding a swf into the page if you then plan on using the code with ExternalInterface. They do not like it with SWFObject or with the simple old fashioned object tag. The solution was elusive, but there is some help on the ghettocooler.net Treasure Trove.

The following code will work in Firefox and Safari, but will mysteriously have a problem in Internet Explorer:

<script language="javascript">
function submitLemon() {
   var swf = document.getElementById("myContent");
   swf.iCanHasParameter('lemon');
}

swfobject.embedSWF("movie.swf", "myContent", "800", 
                   "100%", "9.0.124", null, {}, {bgcolor: '#ffffff'});
</script>

It will however work if you replace the upper case letter with its lowercase equivalent:

<script language="javascript">
function submitLemon() {
   var swf = document.getElementById("mycontent");
   swf.iCanHasParameter('lemon');
}

swfobject.embedSWF("movie.swf", "mycontent", "800", 
                    "100%", "9.0.124", null, {}, {bgcolor: '#ffffff'});
</script>

Depth and Discovery: Powering Visualizations with the Google Analytics API

2009-11-17T13:49:05Z

At Juice, we work with web analytics APIs large and small, from Google, comScore and Omniture. The Google Analytics API is our favorite. It powers the world's best, most widely deployed analytics site. And it powers Juice products like Concentrate (innovative search analytics) and Vasco de Gapi (a tool for exploring the Google Analytics API).

We were approached by the Google Analytics API team to find ways to explore new ways of looking at data with the API, and we were excited by the possibilities. We've been working on our own visualization framework, JuiceKit, that integrates the power of the Flare Visualization Library with Adobe Flex.

The result is Analytics Visualizations, two visualizations powered by the Google Analytics API that are free to use. You just need a Google account with access to Google Analytics data to explore your own data.

Referrer Flow

Curious about what sites are linking to you and what content is benefitting the most? Referrer Flow answers those question and shows how results change over time. Here is a brief video introduction:

Referrer Flow is a stream of daily treemaps showing pageviews and bounce rates for various groupings of your website's pages. You can group by combinations of page title, referrer and url. Clicking on the treemap will filter all the data by the page, referrer or url that you clicked on. Click again to clear your filter.

Keyword Tree

A list of top keywords isn't enough to really understand how people are searching and finding your site. Keyword Tree visually displays the most frequently used search keywords and how they are used together. Here's a video overview:

You'll see a frequently used search term at the center and the words and phrases that are most often used in combination with that word. Pick a different starting word by typing into the box in the upper right or selecting from the top word across the bottom of the screen. The words are sized by their frequency of use and colored by bounce rate (or % new visitors or average time on site). Roll over a word to see details about that combination of connected words.

Depth and Discovery

In designing these visualizations we focused on the question: how can we let users uncover the unexpected? That means designing targeted visualizations focused on limited well-defined issues. The Referrer Flow monomaniacally focuses on a single question "What pages are people viewing on your site and where are they coming from?" The Keyword Tree is laser-focused on word ordering and what that means for keyword performance.

The Google Analytics reporting tool is a great general-purpose reporting solution. It gives the advanced users everything they need to answer specific questions. However, its generality means it has limited ability to focus on two issues; depth and discovery.

The Google Analytics API is Google's solution to this problem. It's an opportunity both for businesses like ours that can create new ways of analyzing data, and for large sites that can use the API for integration, custom analytics, and more.

Thanks to Nick Mihailovski at Google for his gracious support, help and encouragement and Avinash Kaushik for inspiring this idea.

Enhanced Google Analytics: Firefox Plugin

2009-04-13T14:10:09Z

There is new life in the tool that shows change in Google Analytics. A year after releasing our Greasemonkey script, we are pleased to release an updated version of the Enhanced Google Analytics script as a free Firefox Plugin. For those already using the older Greasemonkey script, you can skip ahead to the What's new? and How do I get this plugin? sections of the page. For the rest, you may be wondering: Why does my Google Analytics need change?

Change, and why it is important

When I first started working at Juice Analytics, my boss Zach showed me a part of his daily Google Analytics routine. He would open up the Referring Sites page, glance at all of our 942 referrers. Using his superior intellect and capacity for remembering random urls, Zach would discover interesting deviations in the traffic from sites linking to our blog.

Our top referrers looked more or less similar day to day. Even once you get past the more recognizable top sites such as Twitter and Google, the various somethingblog.com pages, without context, often look a lot like somethingelseblog.com. To top it off, most of the information is not even specifically interesting. Our chartchooser.juiceanalytics.com domain sends us consistent regular referrals, but so what? Day to day, I don't even really care about Google or Twitter unless something changes. With change, I know whether someone has posted something new about me, sending valuable traffic. A good read on the topic is Avinash's rant about "actionable analytics".

Our Firefox plugin is designed to allow analysts to get more action out of what changed in the Referring Sites and Keyword Reports. Here are a couple examples of the plugin in action from our Google Analytics account:

What's new?

Our focus for this release has been to improve functionality, to reduce the barrier to entry for new users, and to allow automatic updates for the plugin. The new version of the script works nearly instantaneously, and the installation involves only two clicks (in contrast to the 7 clicks of the Greasemonkey version). As a Firefox plugin, updates are now automatic and require no reinstall. Keyword sensitivity has been raised to 50% for consistency. As a slight bonus, the design and layout of the form and buttons is now sleeker and the table stands out in a pretty Google blue.

Greasemonkey itself is no longer required for the plugin, but you may want to keep it around for any of the other cool scripts available from the community. If you ever find yourself wishing that something about the web looked different, acted different or had different functionality, there may be a Greasemonkey script to ease your pain.

How do I get this plugin?

First, you need Firefox 2.0+.

If you are a user of the equivalent older Greasemonkey version of this script, you may want to go ahead and uninstall it. Go to Tools=>Greasemonkey=>Manage User Scripts..., select Google Analytics Downloader, and uncheck the Enabled box.

If you never had the script installed, or once you removed it, simply click here to go the mozilla addon site, select the checkbox and click the button. Once installed, navigate to Google Analytics, and go to either the Referring Sites or Keyword pages, and click the blue button.

Happy analyzing!

US Economic Census Treemap

2009-03-20T13:47:18Z

Now that I’ve got treemaps on the brain, I keep noticing how many things could be better understood using this visualization technique. A few examples:

We thought it would be a nice demonstration to use data from the 1997 and 2002 US Economic Census (unfortunately 2007 isn't out yet) to see what kind of stories bubble forth. The demonstration was built using a component from JuiceKit™, our recently open sourced Software Development Kit (SDK) for building Information Experience™ applications. The SDK can be used by web designers and developers to build graphically rich and interactive information displays. JuiceKit currently integrates with Adobe Flex to create components that are easy to implement and aesthetically pleasing.

Check out the treemap here.

Here are a few of the macro-trends that I found:

The rise of CostCo, Amazon, and Home Depot: This time period saw strong growth in warehouse clubs and superstores, online retailers (“electronic shopping”), and home centers.
From manufacturing to services economy: Most of the growth was in service sectors (financial services, healthcare, professional services) while manufacturing was shrinking.
Productivity gains, even in adversity: For struggling sectors, the employee declines almost always outpaced the sales declines — squeezing more sales per employee.
Demographic shifts: Homes and services for the elderly were among the strongest areas of growth in the category of “healthcare and social assistance.”

And there were lots of little insights as well:

No wonder hospital TV shows are so popular: Hospitals are the largest single employer as a business-type.
Starbucks and Krispy Kreme steal the unhealthy food dollar: Cookies and frozen yogurt retail saw a rapid decline while coffee and donut shops flourished.
Goodbye stand-alone pump: Gas stations with convenience stores overtook the just-plain gas station.
It can’t last, can it?: Mortgage broker payroll up 177%.

Once you understand how to read treemaps, they are great for exploring data like this: hierarchical with both quantity and quality-type measures. In a true testament to their power, my wife admitted this visualization was “kinda interesting.”

Announcing: JuiceKit™ SDK Open Source

2009-02-21T09:23:30Z

As our followers know, for the past few years Juice has been creating software applications that solve customers' real information visualization problems in purposeful, understandable, and beautiful ways. In doing this, we have found ourselves reusing quite a few components over and over again - which has made our jobs a lot easier. It occurred to us that others might like to benefit from using these components to achieve great results too.

We're proud to announce the open source release of Juice Analytics' JuiceKit SDK.

The JuiceKit is a toolkit built on Adobe's Flex SDK to make it easier for web designers and software developers to build visually compelling Information Experiences™. It contains a wide variety of development components from individual data renderers such as a single "small multiple", to a large visualization component such as a treemap or US Map, to fine grained "helpers" that provide handy capabilities such as copying data to the computer's clipboard. These components can be used independently, within other applications, or assembled together to create full applications.

What can I do with it? (Show me the money)

Because we've been using the JuiceKit for quite a while, we have a number of customer proven applications based on the SDK that we thought you'd be interested in seeing.

Here is a screenshot of an application that we built to help our client see trends in their internet search and traffic activity. We used the JuiceKit™ to create the small multiples data visualization component of this application.

We've also frequently used JuiceKit to create dashboard prototypes. If you haven't seen our recent application of our treemap component to the incomprehensible Federal Stimulus Plan, here is a nice example (click to explore):

And here is a very quick one we did for an IVR monitoring application where we assembled multiple different components together into one view:

Finally, we've used JuiceKit many times to build full enterprise applications such as this sales pipeline tracking dashboard:

How do I get it?

Now it's time for you to have a go. Here's how you do it:

Go to the JuiceKit SDK web page at juicekit.org and catch up on the current status of the project
Check out the JuiceKit discussion group on Google Groups
Download the JuiceKit library from github
Contribute back to the JuiceKit community to make the JuiceKit even better

While Juice continues to focus on designing and providing software solutions (as opposed to toolkits) for our clients, we believe offering the JuiceKit as open source will benefit the information visualization community we try to serve. In the future we will continue to extend the JuiceKit with other components and technologies.

Good luck, and make sure you share how you're using the SDK so we can continue to drive it in the right direction not only for us, but for you as well.

Delivering Data in Excel: The DTP Framework

2009-01-27T10:44:41Z

Here at Juice we build fewer Excel dashboards than we used to. Excel itself is a decidedly imperfect vessel for any serious development--it's simply too easy to veer off of the disciplined track onto the underbrush.

Even so, Excel remains a playground where we can do surprising things. For instance, check out our Excel lightbox and an Excel tagcloud. We could appropriate everything that you find on the webbiest of Web 2.0 websites and build our Uruk-hai equivalents.

The key to staying on the rails when building Excel tools--either dynamic dashboards or simply to explore data--is discipline. At Juice, we use a methodology that we call "DTP" (Data Tansform Present). The foundation of DTP is the rigorous separation of data from presentation. This is similar to a well-known approach when building computer user interfaces called Model-View-Controller. I'm going to cover some of the key principles and we'll follow up with an example later on the blog.

Data

Data is the raw material of any visualization or report. It needs to be easy to add data or change data without having to change anything else about your dashboard.

We store raw data with dimensions preceding metrics in blocks in separate worksheets. If you want to sound pretentious, you can call this "first PivotTable normal form". Key points:

Have one worksheet for each data source.
Call these sheets "Data", or "{Title} Data".
Place them at the end of your workbook.
Data is snug to the top left of the spreadsheet. This allows us to use dynamic ranges. Dynamic ranges let you add data and have it automatically incorporated in all PivotTables.
Ensure that column names are in the first row.
Place your dimensions before metrics.

Transform

We use PivotTables to transform the data into the structure we need.

Call these sheets "Transform" or "XXXXXXX Transform".
Create one sheet for each issue that you are exploring. This doesn't mean that you will only create one PivotTable. You may have multiple PivotTables to support different views or perspectives on an issue.
Turn on "show items with no data" for row and column dimensions.
We are seeking predictability, we want to the PivotTable to always be the same size regardless of what the PageField filters are.
Place all the dimensions that aren't used as rows or columns in the PivotTable as page fields. Every dimension should have a home.
- Set all PivotTables to not store data and refresh on open.

Present

The Presentation page copies data from the Transform page(s) and formats it for display. It also allows users to control what data is being displayed.

Build a user interface to interact with your data. There are many ways to let people interact with your data, but one of the easiest is to use a PivotTable as your interface. This is described below.
We use an in-house style guide for graphs that you can see in our Chart Chooser.
If the Presentation page is likely to be printed, preset the print range.
When copying data from the transformation page to the presentation page, blank values will come out as zeros. We use a simple formula, =if('Transform!A2'<>"",'Transform!A2', ""), to ensure that blanks remain blanks.

Using a PivotTable as your interface

A simple way to let people manipulate your data is place a PivotTable containing only PageFields but no data on the presentation sheet. A Visual Basic macro triggered to run whenever the PivotTable changes then pushes out any changes to the master PivotTable to all the PivotTables on your Transform sheet.

Here is the code to make this happen.

This drives our PivotTables in concert and ensures they stay in sync.

That's a basic overview of our DTP technique. You can try a simplified version of DTP here.

DTP Example.xls

We'll be back soon to talk through this example.

Bubble, bubble toil and trouble

2009-01-15T12:22:53Z

Recently we wanted to show how Concentrate, our new long-tail search analytics tool, could give you a view of search patterns across travel websites. As political junkies, we were inspired by this chart from our friends at the NY Times.

The first tool we tried, simply on principle, was Excel 2003. As expected, making a NY Times quality bubble chart in Excel 2003 is a hard problem. Here's a draft of how far I got before giving in to label fatigue.

The bubbles themselves aren't tough, but getting the labels right is hard. I'd love to see a solution, so if any reader wants to tackle it eternal fame can be yours. Here is a CSV if you want to try.

travelpatterns.csv

Another of the tools we use at Juice is NodeBox, which we used to make this:

Here's the code that made the graph.

The power of a programmatic approach like this is that by changing a line or two, you can get the following. Click for a larger version. Click the text for the code..

Vary formatting by value

Drop the labels, tighten the spacing

Display the largest item in each row as a red square

With great power comes a great need to exercise restraint. Otherwise you end up like these poor chaps. Must... flex... restraint... muscles...

Introducing Concentrate for Long Tail Search Analytics

2009-01-09T13:01:17Z

We are ~~pleased~~ thrilled to introduce Concentrate™, an innovative long-tail search analytics tool. Concentrate is for SEO and paid search professionals who want to make sense of search keyword data and make the most of search investments.

Check out the demo here. Or try out the free version here (you’ll need admin access to a Google Analytics account).

We built Concentrate because we saw a fundamental conflict in the world of search analysis: On the one hand, search keyword data is terrifically interesting and valuable. It can tell you what your visitors and customers want and how they think about you and your products.

Unfortunately, search query data is also big, messy, and hard to get your hands around. In a typical month, the Juice site gets over 10,000 visits from over 7,000 unique keywords.

Even if I could somehow wrap my head around our top 100 keywords, I’d only understand 25% of the visits. For people spending money on search engine optimization or paid search campaigns, that’s a big blind-spot to accept.

We want you to understand and act on all your search data. Concentrate ingests data from sources that most sites already have available (e.g Google Analytics, Omniture, Coremetrics, Hitwise, Compete, etc.), enhances this data by finding common patterns and query types, and visualizes search phrases for exploration and analysis.

Over the next couple of weeks, we will share examples of some of the interesting things you can do with Concentrate, including:

Pattern identification to condense the long tail into keyword phrases with similar structures. For example, here are some common search patterns from a cooking web site (the “[x]” represents a wildcard).

Keyword visualization to show the connections between keywords and the relative performance of phrases. This wordtree shows the frequency of words within phrases (size) and average time spent on site (color).

Congratulations to Chris, Pete, and Sal for all their hard work, diligence, and creative problem solving to launch this solution.

Jquery History Plugin With Regular Expressions

2008-11-04T15:45:40Z

Over the past couple months, I have delved into the world of jquery, which, for the uninitiated, is the framework for developing javascript in today's world of web development. Jquery shortens or eliminates many of the more tedious aspects of HTML's dom framework, while allowing elegant access to the fancier effects of javascript. One of the other coolest parts of this framework is the growing number of plugins built by the community.

The plugin that I want to bring up is the history plugin by Klaus Hartl, the original of which can be found here. Javascript usually has problems with the back button, but here it is integrated smoothly, and without forcing the page to do an ugly reload. As an added bonus, the plugin also allows the creation of links that are tied to an internal javascript state. Basically, javascript would now have the abilities of a full-blown 90s style html-only webpage, with the smoothness and efficiency of not requiring any page reloads. A good example of a similar javascript history module in action is Gmail, where the back button works smoothly, and you are able to copy and paste the url to a different window.

My slightly modified version of the script can be found here. This potentially interesting enhancement allows the use of a regular expression when defining the javascript internals for the history plugin. It is useful when the site has additional user-specific variables, such as an id or an object name or number, and the programmer has no desire to explicitly code every possibly imaginable scenario. Regular expressions are very powerful tools.

To use the modified version of the module, use a snippet like this one, substituting a regex for where the history module would normally require a string:

    $.ui.history('add', '(\\d+)/analyze/queries', function(url, pg) {
        set_phrasegroup(pg);
        //do something something
    });

Note the slight irregularity where you need to escape each slash with another slash.

The first parameter, url as passed into the function is the entire string that the user inputed. It would look something like 3/analyze/queries, while pg, any other further passed parameters refer to any of the matched groups within the regex. In this case, we only have one, (\d+), a number.

Programmatic Google Trends API

2008-06-11T17:43:44Z

Updated October 21, 2009

Yesterday, Google released an update to their popular Google Trends tool. There are improvements over the previous version, but the biggest new feature is a new shiny button that lets you download all your data in the format of a CSV. This is a very cool enhancement. Where Google Trends was a geeky toy, it now takes the leap to integrate into analysts' reports and with that, edge its way onto managerial desks.

This python module is a quasi-API to make it easier to authenticate into Google Trends for those who want to squeeze the extra level of functionality out of their data. The advantage of programmatic access is that the data can be automatically trended and merged. It can be snuck into a 9:00 AM daily email to the VP of Marketing so that she knows to ramp up Google Adwords campaigns for some specific keyword. Also, by programatically pulling multiple reports, it is possible to create a wealth of data not visible in a single report. Using one keyword as a benchmark to merge multiple reports, we can do a meaningful comparison on tens or hundreds of relevant keywords.

To use the pyGTrends, the quasi-Google-Trends-API, you can download the latest version from github.

Here is an example of the most basic basic report that you can pull down from Google Trends. The connector function needs authentication info, and download_report needs to be passed a list of keywords.

from pyGTrends import pyGTrends

connector = pyGTrends('google username','google password')
connector.download_report(('keyword1', 'keyword2'))
print connector.csv()

You can, however, use pyGTrends to get any slice of data that you can pull down from Google Trends. To see the exact parameters that you should use, go to Google Trends, and navigate to the specific sufficiently-narrow report that you are interested in. Then, right-click on the CSV download, and save the link location. The different parameters should be discernible from the link. The following code downloads a report for banana, bread, and bakery keywords from April 2008, originating from the magnificent nation of Austria, and scaled using fixed scaling (aka the second download link).

connector.download_report(('banana', 'bread', 'bakery'), 
                          date='2008-4', 
                          geo='AT', 
                          scale=1)

By default, the csv() function downloads the main part of the report, but there are a few additional parts stuck to the bottom of the CSV file. If you are interested in those, pass the section parameter to the csv() function. The following will return the Language section.

print connector.csv(section='Language')

Full recommended usage includes using either the csv.reader or csv.DictReader module.

from csv import DictReader
print DictReader(connector.csv().split('\n'))

Here is a snapshot from the new Google Trends to add some eye-candy to the post: