We are happy to say that we have been developing a powerful, simple to use script that will analyse your webserver log files (Apache, Nginx, IIS, Akamai, etc.) and will import visits, page views in Piwik.

We hope that in the next few months, Piwik will become the best alternative to Urchin and AWStats (and others).

Piwik Features when used to Import Log files

Piwik normally uses Javascript code to track visits and pages. This new script will also make it easy to track visits by importing one or many web server log files in Piwik. This is useful if you are not able to add the JS code to the websites, or if you wish to import large amount of historical data at once, or if you are looking for a software that does the same thing as Urchin, AWStats, Webanalyzer or Webtrends.

Some features of the Piwik Log Import script include:

• Great performance, we have tested to track several millions of log lines per day with success. See Piwik for high traffic check list.
• Bot traffic is automatically excluded: to keep your Web Analytics report clean and useful, and increased performance.
• Piwik can track websites with the standard Javascript code, and other websites could be tracked by importing the access logs. For example Javascript tagging for website 1 and 3, and Log import for site 2 and 4. We expect these hybrid Piwik servers to become a common configuration among the community.
• File downloads appearing in the logs will be automatically tracked as "Downloads" in Piwik
• Because Logs will be imported via the Tracking API, all Piwik features will be supported (Goal tracking based on URL, IP Anonymization, Visitor log, etc.)
• Some reports will have no data because the log data is more limited that data obtained via Javascript. For example: screen resolutions, Supported Browser plugins, Custom variables, Ecommerce Analytics will not work.
• This script will effectively replace Apache2Piwik, the new tool providing more features and better performance.
• In later versions we are planning to support Log reprocessing, Error code tracking, Search engine & spam Bot tracking, Feature to use the logs to enhance existing JS tracked pages, and more (based on user popularity and feedback).

This script will be written in Python and will be released under the GPL license, for Free (just like Piwik!)

Perfect for Web hosts company and Web agencies, but also for a one-off log import

The script will have 2 modes:

1. Web Host – Web Analytics Provider user
   This mode is ideal for Web hosts, where new websites are often added in the access logs, but the Piwik admin does not wish to create manually each website. The script will automatically detect the Piwik website ID to track based on the URL being parsed: it will look for any Piwik website registered with a URL or "Alias URL" set to this page view host. If a website with the hostname doesn't exist, a new website is automatically created for this URL.
   A summary is then emailed to the Piwik Super User so he/she knows which websites are automatically created by the Log import script, so you can create users or assign permissions to view these new websites.
2. Simple Log Import for one or a few websites only
   This mode is ideal if you import only a small number of websites or if you wish to control exactly in which websites requests are tracked.
   When a line contains a URL to an unknown Piwik website, Piwik can either ignore all these pageviews, or you can choose to record these unknown pages in a specified "catchall" Website id, to double check they are not legitimate pageviews.
   If these unknown URLs turn out to be legitimate pageviews, you can either create a new website manually, or add an Alias URL to an existing website, so the page URLs are directly tracked in this website the next time you import similar logs.

Join the beta testing group

To be part of our beta testing group, please email us at hello@piwik.org and mention the Testing of the Urchin/Awstats log import script. Please also mention the number of websites to track, how many pages per day, and if you are willing to test the script and report bugs or feedback.

Featured sponsor: a Web Host that tracks millions of log lines with Piwik

This work is sponsored by Alwaysdata, a French web hosting company. They provide Piwik as the Web Analytics package of choice, deprecating AWStats, for thousands of their users. They have been using Piwik for a few years and we are finally integrating this Log Import Analytics key feature in Piwik, and ensuring good performance for the script. We want to make it easy for Web hosts and large Web Agencies to use it as their Web Analytics platform.

Goodbye Urchin + Scale of Google Analytics in 2012

The Google Analytics team have decided to focus on the privately hosted Google Analytics (GA) service and discontinue the Log Analysis version (Urchin). At Piwik we are quite simply amazed at the scale and reach of Google Analytics in 2012: GA is used by over 55% of all Internet websites (source). At least 15 million websites use Google Analytics! (source). In comparison Piwik is used on 1% of the Internet (cheers!) and 250k+ websites.

Million of pings (page views) are tracked by GA per SECOND. It is hard to imagine such scale which would make any software developer speechless. We can only congratulate Google engineers and product designers for the work they are doing to track and aggregate so much data, while allowing to slice this data in real time across dozens of dimensions. This is an amazing technical milestone. We also hope that Google users privacy will be respected and privacy standards will improve in the future.

Regarding the end of Urchin, we at Piwik will do our best to provide to existing Urchin users a good experience to upgrade to Piwik, and try the leading Free software platform. If you are a Urchin user and would like to try Piwik, send us an email us at hello@piwik.org with your current setup. We will help and check similar functionnality is do-able with Piwik and the log import script.

Privacy & Security implications of self hosting your Web Analytics data

Ensuring the full control over your customers Log files, Piwik database are important requirements if you are a Web agency or a Web host providing web analytics to hundreds or thousands of users.

The tips in the Privacy page will help ensure that you make the changes to data collection and data retention required by your Privacy Policy. We also focus on Code security and recommend to all Piwik users to spend some time securing the Piwik server.

Piwik also an alternative to AWStats, Webalizer, : modern UI, better performance, and more!

We hope that Piwik will become the leading alternative to Urchin and to AWStats. AWStats was a great tool but we hope to modernize the Log analysis open source software world and make use of all the great Piwik features and capabilities in terms of data analysis and graphing. Users in 2012 and beyond will need a modern interface to access the data gathered from their Web server Access logs.

We expect a release in 1-2 months. Stay tuned…

Happy Log files import with Piwik!

Everyone loves stickers, we can use 'em to jazz up your car, laptop, books or even lamp posts! Therefore, we would love to offer beautiful Piwik stickers to the community.

So, we are launching a Design Contest to come up with a funky design that we will use to print stickers!

Contest & Sticker Design guidelines

• Themes: we would prefer if the sticker included at least the Piwik logo or stylised Piwik text. Otherwise, you can get creative! As an idea, you could add in text about related topics such as: Open source web analytics,  Data privacy & protection, private cloud, Freedom, open data, real time analytics, etc. You are free to design how you like!
  
• Shape: sticker designs can contain transparency and can be of any shape: oval, rectangular or square
• Sizes: maximum reasonable size for a sticker would be around 10*6 centimeters. Small stickers are also possible, up to you. We will do our best to find the right sticker printing company to print the winning designs
• Duration: we will award prizes for the first 5 winning designs! Piwik team will pick the winning designs, but we will also post them on this page and listen to community feedback :)

YOU COULD WIN!!

The prize for the winning sticker design is $137 for each design we end up printing on stickers – payment by Paypal. You will also get free stickers with your design(s) shipped to your preferred address wherever you are in the world so you can show your cool design off to your friends!

The stickers will then be available in the Piwik shop for the smallest price possible: we want all Piwik users to enjoy and put their stickers wherever they like.

Submit your logo designs

Send your designs to design@piwik.org in png or jpg format, with a max width of 1200 pixels.

We hope to receive many cool sticker design submissions, and will update this page with the best ones in the next few days.

Happy designing!

Order your very own Piwik t-shirt or hoodie here:

About the T-shirts

ooShirts.com is a reputable Custom T-shirts company, dedicated to providing a great service and products. The T-shirt quality is great (we have tested many washes and it still looks good!). Also, the shirts comply to standards of Fair labor, which is very important to us. Shipping anywhere in the world is just $5 USD. More details on the tshirts page.

Other Piwik news!

We are currently working on the next version of Piwik with some of the new features already in SVN: new reports in Piwik, graphs in PDF reports, cool new features in the user interface: we hope to release Piwik 1.7 in a few weeks time, so stay tuned!

We are also working on posting new content on the Piwik blog and will publish a "How to" series. The tutorial series will for example explain: how to track various pages and sections of a website effectively, how to track facebook activity, how to track clicks on slideshows… and more! We will use a Piwik community member site, divezone.net scuba diving guide, holidays… as an example and show how to track and analyse relevant user interactions on your website!

If you have any suggestions or request for tutorials, how to, please submit us your ideas.

Thank you for your support & we hope you enjoy wearing your Piwik gears!

We are happy to announce that Piwik is ranked #2 in the top of web analytics tools in Germany. A total of 13% of German websites .de that are using an analytics tool have chosen the open source alternative Piwik.

To celebrate this amazing news, we would like to invite the Piwik community to meet for our first meetup in Munich, Germany!

Location: Mannhardtstrasse 6, 80538 Munich - Map & address
Date: Oct 22nd, afternoon from 2pm
Cost: Free!
Register now on the event page

We would love to see you there for an afternoon of talks, fun, meeting other Piwik community users and core team members. We also hope to discuss the roadmap and generate new ideas and suggestions to improve Piwik.  Please register on the event page for a free ticket (limited to 25 seats).

More info: Piwik ranked #2 Web Analytics tool in Germany

Piwik is ranked #13 worldwide with a global marketshare of 1.5%. Piwik is the most popular open source software and the only Free software to have a significant marketshare. However in Germany Piwik has 13% marketshare in Web Analytics tools, which is an incredible achievement!!

Below is the screenshot of Web analytics Germany marketshare (Source W3techs)

Germany is leading Privacy issues awareness with open political discussions and decisions. In particular, internet businesses with servers in foreign countries are restricted in the way they collect Personally Identifiable Information (PII) from German visitors. Piwik has become a building block of the Free Internet and Open data movement. We believe that user privacy should always be protected, and more websites will make use of open source technologies to keep control over this sensitive data. Work is always ongoing to make Piwik better and faster. We also hope to keep developing our active community on the forums and organise more meetups in the future.

Piwik meetup: Munich, Saturday October 22nd

Germany is the most represented country in the Piwik community, and half of the core dev team is German. So we are excited to meet community members and meet other Piwik users in Munich! We will announce the agenda (all talks will be in German language) in a few days on the event page. Food and drinks will be provided.

Click here to register quickly, limited entries (first come first served).

For any questions or feedback about the meetup, please contact us at hello@piwik.org. Thank you to Mayflower for hosting the meetup.

We hope to see you there!

Thomas, Stefan, Timo, Peter and André
Piwik team

Updated: Piwik slides can now be found below!

Piwik Roadmap 2.0 – Oct 2011, by Timo Besenreuther

Piwik Roadmap 2.0 – October 2011

Piwik Mobile Overview – Oct 2011, by Thomas Steur

Piwik Mobile Overview – October 2011

In particular, we are planning to work on a website redesign for piwik.org website, which contains the various documentations, FAQs, the download counter, Participate in the project, and many other pages.

What are your ideas on the website redesign? What look and feel should it have, etc.

Please leave your best suggestions/ constructive feedback, in a comment in this blog post, or in the Facebook fan page, or RT & Reply to us. Thank you to all for leaving your ideas!

Happy life & analytics,
Piwik team

While tracking and reporting ecommerce data is hard, it is equally hard to implement ecommerce analytics tags on your web site. Just think about the integration – each product page, shopping cart page, confirmation page needs specialized tags, which hook into data (e.g. price, quantity, tax, shipping, etc). And this is not taking into account Javascript 'add to cart' functions, returns, and other features!

We pride ourselves on making Piwik INCREDIBLY simple to install, setup, and use. We also want to make the process of implementing ecommerce tracking even simpler. To this end, we are working closely with the folks from Jirafe to implement plugins in ecommerce platforms (such as Magento, Prestashop, etc.) that automate setting up tags in all of the right places, and to create a hosted service so that there is an automated registration, installation, setup, and reporting inside of the ecommerce platform.

We are really excited about the potential developments here. By using an open source tool like Piwik, with our huge community to help provide feedback on plugins on various platforms, we have a truly unique way to bring enterprise level reporting to all ecommerce businesses, big and small.

If you run an ecommerce store, and would like to participate in a beta of a hosted Piwik service for ecommerce platforms, please fill in your details below.

→ Sign Up For Ecommerce Analytics

Log files are known to contain a wealth of information about activity on a website, and are usually analyzed with tools such as AWStats or Webalizer. Being able to transfer it to Piwik, a powerful web analysis tool, can greatly enhance data mining and presentation. This, in turn, means more control over your web property, better informed decisions and greater potential for optimalization.

Importing visits, pages, Goal conversions from logs is very fast, processing thousands of log lines per second and can also read & process your log files in real time. Piwik reports after import have a few missing data points compared to the standard Javascript code and standard Piwik reports. However, compared to older log analyser softwares such as Webalizer or AWStats, Piwik reports are sharp, easy to understand, and lets you focus on your analysis goals!

This page contains the following sections: Apache2Piwik requirements, How to use guide, List of missing reports when using log files, Performance of the script, and Credits.

Apache2Piwik Requirements

• access to Piwik installation
• access to Apache logs with read privileges (you can specify log format in settings.py)
• Python 2.6 with MySQLdb, GeoIP for Python and httpagentparser

How to import Apache logs in Piwik?

Follow these steps for a test export with Apache2Piwik:

• Important: create backup of your Piwik MySQL Database.
• create `settings.py` as a copy of settings.py.sample and edit MySQL Piwik Database configuration
• execute apache2piwik.py – see examples below

Example 1 – importing log file, all settings set in settings.py file:

$ python2.6 ./apache2piwik.py
Started processing /path/to/file/logfile1 file...
Finished in 2m16s.
Started processing /path/to/file/logfile2 file...
Finished in 2m59s.


$ python2.6 ./apache2piwik.py start

$ python2.6 ./apache2piwik.py stop

$ python apache2piwik.py -g

• Images files are automatically ignored. You can customize ignored extensions in settings.py file. You can also ignore specific logs with regular expressions there
• Search bots are not excluded at this stage. We might add a feature to exclude bots in a future version.
• When you import data in the past, or when you want to reprocess your reports from the logs, you can delete piwik_archive_* tables. See more information in this FAQ.
• Apache2Piwik imports data into the idsite specified in settings.py. You can override this by "-i [idsite]" command line parameter

Reporting differences between Server Logs and Javascript code

Piwik 1.5 is a major upgrade, bringing amazing new features, better scalability and fixing a security issue affecting a minority of Piwik servers:

• Download link
• How to update Piwik?
• List of all tickets closed: Changelog

New features in Piwik 1.5!

• Ecommerce Analytics
• Custom Variables for each page view
• Graphs in Canvas+Jquery – no more Flash graphs!
• Upload your Custom Logo
• IP Anonymization available in the Privacy Settings
• Automatically Delete old logs to keep database size small
• Security bug fix affecting only minority of users when Piwik is open to anonymous (see below)
• many bug fixes
• and more…!

Critical security issue fixed

Piwik 1.5 also contains a fix for a security issue that was disclosed to us as part of our Security bug bounty program.  The security bug is rated critical. In the default configuration of users logging in to see their reports, Piwik is not affected by this issue. But when the anonymous user has some access to reports that the Piwik server could be compromised using arbitrary code execution..

We would like to thank Neal Poole for his responsible disclosure of the security bug. Neal is the first researcher to win the $500 bounty. Thanks to Neal for making Piwik safer and better!

Ecommerce Analytics for FREE!

The e-Commerce functionality in Piwik v1.5 came as a result of consulting with Jirafe, an ecommerce analytics company, over the last few months. Jirafe uses Piwik as their analytics engine, and deeply integrates their analytics service inside of popular ecommerce platforms like Magento and provides an ecommerce dashboard to these platforms. They have been sponsoring all of the ecommerce functionality that you now see in Piwik. We hope that they continue to sponsor new features for the Piwik community!

Check out some screenshots

Ecommerce overview

Ecommerce Best Products

and MORE Ecommerce goodness!

Spread the word

The support of the community, activity in the forums, downloads are going strong…. We hope that Piwik stays on course and becomes one of the major Analytics solutions, bring advanced Privacy features and more control over your data. Please spread the word about Piwik in your blog, website and let your friends know.

We hope you enjoy the new release, please report any feedback, error or suggestion in the Forums.

Happy analytics!

Sites are Big and Small

The Piwik community seems to span across the spectrum of website sizes, without a skew toward large or small sites. The ‘sweet spot’ of website sizes seems to be between 1000 and 100,000 visitors per month, with 50% of you fitting in this category.

Diverse Site Categories

There is a great mix of websites that choose Piwik.  While there are plenty of community sites, web publishers, and ecommerce sites (together about 50% of sites), the other 50% are ‘other’.  It seems like web sites today do not fit in a particular category – and possibly have many different functions.  For example, the Piwik sites include a community site (forums), web publisher (blog), development site (dev), among many other sites.  One of the strengths of Piwik is to customise and integrate our technology sites that do not fit a straightforward category.

Word of Mouth Advertising

The majority of you do not spend money on advertising to bring people to your web site (word of mouth, eh?).  Those that do, use many programs, with Google Adwords, Banners, Links, Affiliate Programs, and Email are each used by over 10% of respondants.
Note: Piwik provides advanced Campaign tracking, you can bookmark the Piwik URL Builder which will track all campaigns links in your Piwik!

Vocal Community!

The majority of you want real time analytics, and half of you want a hosted service (no surprises there!).  We also received some EXCELLENT new feature suggestions in the comments – the Piwik community are serious about their analytics!  We are constantly using your feedback to decide on what to work next.

If you have any other feedback or ideas to share about Piwik, please do so in the comments below.

In the meantime, the Piwik team is still actively testing Piwik 1.5 RC which include many really useful new features:

• Custom Logo Upload
• Better Privacy plugin, IP Anonymization
• Delete old logs automatically
• Analytics Ecommerce
• Custom variables per page

If you have a low risk Piwik server, you are safe to upgrade already, Piwik 1.5 is already very stable and will never corrupt your data.

Thanks for the ride,

Happy analytics

Timo has finished his first major project as a Piwik member: moving from flash graph to Javascript & Canvas graphs!

This guest blog features a JQPlot description & general answers, by his creator Chris Leonello, and a word about Timo

Screenshot of the new Graphs

Pie chart

Evolution charts

What is jqPlot?

jqPlot started in early 2009 after trying to find a suitable interactive browser based charting package for web based apps. Flash based alternatives were either too expensive, had restrictive licensing or didn't have the needed features. Flash based alternatives also brought another programming language and set of programming tools into the development process, which wasn't desired. Many fine javascript based packages (such as Flot and HighCharts) were investigated. While these were more developer friendly, they still lacked some needed features or had some licensing restrictions.

The decision was made to create a new jQuery based plugin and jqPlot was born. In an effort to give back to the community which made jqPlot possible, jqPlot was created as an open source project.

Piwik now uses jqPlot to render charts. jqPlot is a pure JavaScript plotting plugin for the jQuery framework created by Chris Leonello. jqPlot renders charts using the html5 canvas tag.

What is canvas?

The canvas tag creates a 2D bitmap canvas for drawing graphics and text. Drawing on canvas is like painting on paper, quick, efficient and easy. The canvas element doesn't have any extra memory overhead other than the element itself. Unless you are interacting with the element, it doesn't consume any extra cpu resources either. You control the canvas and draw shapes through simple javascript properties and methods such as "lineStyle" and "fillRect".

jqPlot Browser support
The canvas tag and jqPlot work well on most browsers. All canvas features used by jqPlot are supported on Firefox 3.5+, Safari 4+, Chrome 8.0+, Opera 10+ and IE 9+. IE 6 – 8 are supported through Google's excanvas.js library. On mobile, canvas is supported by iOS Safari 3.2+, Android Browser 2.1+ and Opera Mobile 10.0+.

jqPlot in active development

The project has grown to include 597 members on the mailing list. jqPlot source code on the issue tracker are on bitbucket. Of over 55,000 projects on bitbucket, jqPlot ranks 35th in number of followers, 31st in number of forks and has been downloaded 80,000 times.  While jqplot is developed primarily by Chris, many members of the community have helped improve it.

Piwik contributions in jqPlot

Timo Besenreuther, of the Piwik development team, made significant contributions which helped to greatly improve memory performance. Timo refactored the code to reuse canvas elements instead of recreating them. Timo's enhancements are especially noticeable during rapid plot redrawing, such as while dynamically resizing a chart.

In true open source fashion, Timo forked jqPlot to develop the improvements. These were then merged back into the main jqPlot code. His fork has been merged in the main jqPlot trunk! He also built extra plugins such as easy to read Pie chart legends and Canvas Export as Image.

The Piwik Development Team also created specific use cases that were key in tracking down jqPlot DOM memory leaks.All in all, the Piwik/jqPlot relationship has led to what are probably the most significant improvements in core jqPlot performance. Good job Piwik Dev Team!

Future of jqPlot

jqPlot is currently in beta on it's way to a 1.0 release. Further post 1.0 enhancements under investigation include tweaks for mobile devices and more axes features. In the long term, changes may include a significant overhaul of the core code for improved dynamic response and efficiency, a clean up of internal data structures, a more unified API and streamlined build process.

– Chris Leonello

Piwik innovation

Timo is a talented website developer and software engineer located in Munich, Germany. His expertise in JavaScript was collected over years of JavaScript frontend development. He also offers server side software development and content management in Java and PHP – you can contact Timo via his company website EZdesign.de. Timo has also worked on Inline help (quick help about most reports and columns), a new feature in Piwik 1.5.

Piwik team members and contributors are open source enthusiasts, and it is always a pleasure for us to use latest open source technologies and get a chance to contribute to some of the projects we depend on.

For early birds, enjoy the 1.5 RC release

Consider using jqPlot (see demo) when you need to draw charts,

Thank you for spreading the word about Piwik (check out this page for more information)