by michal-frackowiak

Below I will tell you how we do it.

Assumptions:

1. We have a working Chef server.
2. All nodes run Ubuntu, but it's not really important.
3. All nodes run chef-client periodically (in our case, every 3 minutes).
4. All web nodes have role web.

As a result will develop a minimal haproxy cookbook to be run on the HAProxy node that:

1. Will set up HAproxy service.
2. Will discover and connect to all backend servers.

I will skip the parts related to setting up Chef, nuances in HAProxy configuration, multiple roles and cookbooks and concentrate just on the auto-configuration part. If this in any way encourages you to set up a similar environment, it's good :-)

Here are all important files in our haproxy cookbook:

haproxy ├── metadata.rb ├── recipes │ └── default.rb └── templates └── default └── haproxy.cfg.erb

default.rb

# Install HAProxy repo and package itself apt_repository "haproxy_repo" do uri "http://ppa.launchpad.net/vbernat/haproxy-1.5/ubuntu" components ['main'] distribution node['lsb']['codename'] keyserver "keyserver.ubuntu.com" key "1C61B9CD" deb_src false end package 'haproxy' package 'socat' template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "haproxy" group "haproxy" variables({ backend_nodes: search(:node, "chef_environment:#{node.chef_environment} AND role:web").sort_by{ |n| n.name } }) notifies :reload, 'service[haproxy]' end service "haproxy" do supports status: true, restart: true, reload: true action [ :enable, :start ] end

What it does is that it adds the Vincent Bernat's HAProxy repo first. You might skip it if you are fine with HAProxy in your distribution. We want 1.5 badly.

Then it installs haproxy and socat (useful for admin stuff) packages.

The next lines create the config file from an ERB template. The important line is:

backend_nodes: search(:node, "chef_environment:#{node.chef_environment} AND role:web").sort_by{ |n| n.name }

We can verify that the query actually returns the nodes we expect:

$ knife search node 'chef_environment:production_2 AND role:web' | grep -E '(Name|Roles)' Node Name: i-d15eda3b Roles: web Node Name: i-6ec4088f Roles: web Node Name: i-6dc5098c Roles: web Node Name: i-3491e1de Roles: web Node Name: i-2d0e81cc Roles: web Node Name: i-ce5fdb24 Roles: web Node Name: i-d4c40835 Roles: web Node Name: i-cd5fdb27 Roles: web

One more thing — whenever the config file changes, we need to reload haproxy service. It's done by the line:

notifies :reload, 'service[haproxy]'

This way, whenever list of backend servers change, or we provide a new version of template which affects the config file, haproxy gracefully reloads it's config.

The service… part defines the haproxy service and makes it run by default.

haproxy.cfg.erb

global # Global config goes here defaults # Defaults go here frontend http-in bind *:80 default_backend http-backend backend http-backend balance roundrobin http-check expect status 200 option httpchk GET /ping.php <% @backend_nodes.each do |node| %> server <%= node.name %> <%= node.ipaddress %>:80 check fall 1 <% end %>

I have simplified the config a bit, but the essentials part are here. In the real config we have stuff like SSL termination, stats, several backends, throttling rules for abuse etc.

What's critical is that the list of backend nodes are created from the @backend_nodes variable. And you know, that's it! The only thing left is add the cookbook to the haproxy node and drive traffic to it.

Now it's important to run chef-client periodically on your HAProxy node. This way your list of backend servers would stay up-to-date. One thing I have not mentioned — it helps if you periodically remove dead nodes from Chef server.

We have been running this setup for 2 weeks now on AWS and it works really, really well. We have 2 HAProxy nodes listed in DNS for *.wikidot.com with Route53 health check.

If you have any questions, feel free to ask!

by michal-frackowiak

I have just returned from my summer family holidays in Pieniny. Spending 100% time with my wife and 3 kids was definitely a very needed refresh after a challenging year. I had enough time to think about plans for the next year and gather enough motivation for upcoming tasks.

(BTW: For those interested in the uncensored stream of holiday photos check out my Flickr Gallery)

So what is up for the next year? Actually, quite a lot, and I expect the next 10 months to be crucial for Wikidot and my other projects. It does not necessarily mean there will be a revolution of any kind, but Wikidot is a perfectly mature project and business right now, and it needs smart management and clear way for the future.

Also, other projects I am part of (more or less involved) are going to surface this year. Either hit or miss, those projects are interesting and I will definitely keep my thumbs up when they go live.

Last but not least, I will write more (I mean this blog too). Last few months were kind of difficult the work required a lot of my attention, leaving very little time for blogging.

If you want to follow me closer, the best way is to follow me on Twitter. Thanks!

by michal-frackowiak

After completing the steps below, you will be able to press a combination of keys and ZendCodeAnalyzer will analyze your PHP script and render its output like on the screen below:

And here is how.

1. Download Zend Studio (demo)

Go to http://www.zend.com/en/products/studio/downloads-prev and download a 30-day trial of Zend Studio 7.0.

You can skip this option if you already have Zend Studio, but since you are using TextMate — I bet you don't. You might need to create an account at Zend, but this is all free. Note that the newest Zend Studio 7.1 does not have a standalone ZendCodeAnalyzer binary, and cannot be (yet) used in TextMate.

2. Copy the binary file

One way to get ZendCodeAnalyzer is to install Zend Studio and look for the file in the installation path. But I do not like to waste my time installing stuff I will not need. Besides, we are interested just in one single executable file.

Open (mount) the downloaded dmg image. If you had just downloaded this, it should be mounted. Now paste those commands:

mkdir /tmp/zca cd /tmp/zca unzip /Volumes/ZendStudio\ 7.0.*/ZendStudio.app/Contents/Resources/Java/Disk1/InstData/Resource1.zip \ '$IA_PROJECT_DIR$/resources/Mac/plugins_zg_ia_sf.jar' unzip '$IA_PROJECT_DIR$/resources/Mac/plugins_zg_ia_sf.jar' '*ZendCodeAnalyzer' sudo cp `find . -name ZendCodeAnalyzer` /usr/local/bin/zca sudo chmod a+x /usr/local/bin/zca cd rm -rf /tmp/zca

By doing so you end up having a zca executable in /usr/local/bin. Verify whether it works:

zca --help

4. Edit TextMate PHP bundle

Now you need to add a new command to the PHP bundle.

• Select Bundles » Bundle Editor » Edit Commands…
• Add a new command — I called mine Analyze Code
• Paste the following code:

#!/usr/bin/env ruby require ENV['TM_SUPPORT_PATH'] + '/lib/textmate' result = `/usr/local/bin/zca #{ENV['TM_FILEPATH']} 2>&1` puts result.gsub /^Analyzing.*?\n/m, '' if result =~ /line (\d+)/ TextMate.go_to :line => $1 else puts "No problems found" end

• Set other options as on the screenshot:

(choose any other keyboard shortcut if you wish)

• Edit PHP Bundle menu and move the Analyze Code below Validate Code (pure esthetics)

5. Try it!

Let us try the analyzer on a simple code:

<?php function foo($bar) { $a = 3; return "123" . $c; } echo foo("bar");

The code itself is broken in many places, although it does not have any syntax errors. Now try the code analyzer and you should see something like in the first image up there.

The command automatically moves the cursor to the first line that contains a problem.

Still missing

ZendCodeAnalyzer has been replaced by Semantic Analysis in Zend Studio 7.1 and completely integrated into Eclipse framework. It does not provide a standalone version. Perhaps in the future such a version will be available from Zend, but not now.

ZendCodeAnalyzer has some tuning options, mainly: --enable <warning-name>, --disable <warning-name>, --recursive for including required files and —include-path. Perhaps it would be interesting to play with those to provide more in-depth code analysis.

Show as Tool Tip might not be the best way to display output of analyzer, but I could not figure out anything smarter.

Adding code analysis to TextMate is IMHO a game changer, it can potentially save you many headaches caused by various coding mistakes one can make. Although the above installation uses only basic functions of ZCA, it proved to be a very useful addition to the PHP bundle.

If you also find it useful — please add your comment! I will highly appreciate any suggestions and improvements!

by michal-frackowiak

Anyway, today I have accidentally found a cool game written in JavaScript, using canvas. It really surprises me how much you can actually do with the browser nowadays. The game is:

Another World

— something I was playing on my old Atari 520ST long time ago.

It is not a full game, as author says: Q. What's missing in this version?
A. Only one level is supported, and that one is broken as well. There are graphical glitches, movement bugs, and no music or sound effects. As said before, it's only for messing with <canvas>

But still, as a proof of concept, this is simply great.

For more "experiments" with browsers, visit http://www.chromeexperiments.com. Most of them should also work with Safari.

As a side effect: isn't position of Adobe Flash a bit in danger? Oh, yes it is!

by michal-frackowiak

If you want ruby and irb to link to 1.9.1 version, look at the following piece of code that change the default ruby version. Applying this to a plain and fresh Ubuntu 9.10 Karmic Koala gives you full Ruby installation:

aptitude -y update aptitude -y install ruby1.9.1 ruby1.9.1-dev \ rubygems1.9.1 irb1.9.1 ri1.9.1 rdoc1.9.1 g++ # If you already have 1.8 installed update-alternatives --install /usr/bin/ruby ruby /usr/bin/ruby1.8 500 \ --slave /usr/share/man/man1/ruby.1.gz ruby.1.gz \ /usr/share/man/man1/ruby.1.8.gz \ --slave /usr/bin/ri ri /usr/bin/ri1.8 \ --slave /usr/bin/irb irb /usr/bin/irb1.8 \ --slave /usr/bin/rdoc rdoc /usr/bin/rdoc1.8 # Install ruby1.9.1 update-alternatives --install /usr/bin/ruby ruby /usr/bin/ruby1.9.1 400 \ --slave /usr/share/man/man1/ruby.1.gz ruby.1.gz \ /usr/share/man/man1/ruby1.9.1.1.gz \ --slave /usr/bin/ri ri /usr/bin/ri1.9.1 \ --slave /usr/bin/irb irb /usr/bin/irb1.9.1 \ --slave /usr/bin/rdoc rdoc /usr/bin/rdoc1.9.1 # choose your interpreter # changes symlinks for /usr/bin/ruby , # /usr/bin/irb, /usr/bin/ri and man (1) ruby update-alternatives --config ruby # now try ruby --version

I am also installing g++ and ruby1.9.1-dev packages because some of gems need those.

BTW: Gemcutter.org is now the default gem source, out of the box.

Good luck!

by michal-frackowiak

processor : 0 vendor_id : GenuineIntel cpu family : 6144 KB model : 23 model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz stepping : 6 cpu MHz : 2659.998 cache size : 6144 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm bogomips : 5323.56 clflush size : 64 cache_alignment : 64 address sizes : 38 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz stepping : 6 cpu MHz : 2659.998 cache size : 6144 KB physical id : 1 siblings : 1 core id : 0 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc up pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca lahf_lm bogomips : 5323.56 clflush size : 64 cache_alignment : 64 address sizes : 38 bits physical, 48 bits virtual power management:

The availability zone is: us-east-1c.

Is Amazon moving away from Opteron-based servers? I hope so. Xeon machines so far provide better performance, so I guess they decided to expand their farm with those. Great move!

I could not find any confirmation or other reports on the subject, so please confirm!

by michal-frackowiak

By looking at popular Twitter link aggregators like Tweetmeme.com or twitt(url)y you might get the impression that Twitter is full of useful news, links to interesting stuff, a truly user-driven next-level communication.

Below is the result of one-day data analysis, and it lists top links people were tweeting about on Friday:

1. Entrepreneurs Wanted: http://oneyearplan.net/tmiller/?site=MyOwn2&t=AutoTwitter
2. Mark McCulloch, The Network Marketing Expert: http://www.secretsofwealth.info/
3. Success University, Mark McCulloch.Com, Home Business, Home based business success coach: http://www.coachofsuccess.com/
4. monetize: http://monetizeyourdream.com/
5. Malaria No More - Donate Bed Nets: https://give.malarianomore.org/SSLPage.aspx?pid=382
6. XXX BlackBook: (adult content, sorry, no link)
7. Generate Hot New Leads Using Ebay!: http://www.getmegaleads.com/
8. andy dick test on USTREAM: http://www.ustream.tv/channel/andy-dick-test
9. The Twitter Tag Project - Mr. Milestone!: http://thetwittertagproject.com/milestone.php
10. Shocking Report' Finally Reveals How To Earn A REAL Six Figure Income Without Building A Huge Downline…: http://www.forestmarie.com/DefeatMLM/at.html
11. Proactol | Official Proactol Store :: The Best Natural Slimming Pill: http://www.proactol.com/
12. Introducing MetaMorph: http://www.unemploymentkiller.com/
13. 7 Chakras: http://www.astral-projection.org/7chakras.htm
14. Entrepreneurs Wanted: http://oneyearplan.net/tmiller/?site=MyOwn5&t=100tweet
15. http://twitpic.com/3wzdm
16. Wideo Twisted Helena (autor: Labor) - MySpace Wideo: http://vids.myspace.com/index.cfm?fuseaction=vids.individual&videoid=56342462
17. http://twitpic.com/3xs5n
18. arthquake Tracking Maps - WebPartner Channel: http://webpartner.com/?wp=chroom&chid=jH6xAICdoU91za44hD
19. PTWITTYTV on USTREAM: PTWITTY TV IS LIVE!!! twitter.com/iamdiddy. R&B/Soul:
20. World-mall: http://world-mall.org/free_targeted_traffic

…

The list goes and goes. What is shocking is the popularity of "Twitter-based marketing" (a combination of spam, sales, marketing, ebooks).

If you have never seen any of those links before, this is probably because you stick to your friends' timelines, and this is what most people do. But the increasing number of users trying to actually earn money by moving the well-know internet-based sales schemes to Twitter is at least surprising.

The top links from above easily get thousands of tweets per day, leaving organically-created links far behind.

by michal-frackowiak

The nature of the project requires really fast data backend, capable of storing and retrieving a few thousand items per second. The dataset would be approximately 5GB, average item size: 0.5KB.

When it came to tools selection, after short considerations I have chosen Sinatra for web interface, and Redis as a memory-only (with disk dumps) key-value datastore. It should be capable of handling 100 000 requests per second and deal well with large datasets, so fits perfectly. It also differs from Memcached or MemcachedDB because it has great higher-level structures like Lists and Sets, basic sorting and selection commands.

Recently there is a lot of hype about (distributed) key-value storage, for more info I recommend a nice article by Richard Jones Anti-RDBMS: A list of distributed key-value stores. HighScalability blog also has a lot of references and articles.

Redis looks like the way to go. The only problem with it is that the whole dataset (database) must fit in the RAM, otherwise performance might degrade terribly (because of swapping). Performance itself is not an issue, and you would need several concurrent clients to actually face this as a limit.

Anyway, initially I wanted to deploy the project at Amazon EC2 - because of hyped scalability, price etc. But here comes a surprise — the performance simply sucks. I guess this is because the instances share common hardware and you might have actual memory bandwidth limited.

Here are my results of running

./redis-benchmark -n 100000

Amazon Small instance ($0.10/h)

====== PING ====== 100042 requests completed in 11.95 seconds 50 parallel clients 3 bytes payload keep alive: 1 8369.61 requests per second ====== SET ====== 100023 requests completed in 12.13 seconds 50 parallel clients 3 bytes payload keep alive: 1 8247.28 requests per second ====== GET ====== 100004 requests completed in 14.26 seconds 50 parallel clients 3 bytes payload keep alive: 1 7010.94 requests per second ====== INCR ====== 100000 requests completed in 14.40 seconds 50 parallel clients 3 bytes payload keep alive: 1 6945.89 requests per second ====== LPUSH ====== 100000 requests completed in 12.24 seconds 50 parallel clients 3 bytes payload keep alive: 1 8171.27 requests per second ====== LPOP ====== 100000 requests completed in 14.22 seconds 50 parallel clients 3 bytes payload keep alive: 1 7033.83 requests per second

The small instance is a no-go if you want to use it for Redis. Keep in mind it is AMD-based and in general the High CPU instances (with Intel Xeons) outperform their AMD brothers dramatically.

Amazon High CPU Medium ($0.20/h)

====== PING ====== 100007 requests completed in 6.52 seconds 50 parallel clients 3 bytes payload keep alive: 1 15333.79 requests per second ====== SET ====== 100006 requests completed in 2.22 seconds 50 parallel clients 3 bytes payload keep alive: 1 44986.95 requests per second ====== GET ====== 100009 requests completed in 2.21 seconds 50 parallel clients 3 bytes payload keep alive: 1 45252.94 requests per second ====== INCR ====== 100000 requests completed in 2.35 seconds 50 parallel clients 3 bytes payload keep alive: 1 42625.75 requests per second ====== LPUSH ====== 100009 requests completed in 2.24 seconds 50 parallel clients 3 bytes payload keep alive: 1 44686.78 requests per second ====== LPOP ====== 100011 requests completed in 2.28 seconds 50 parallel clients 3 bytes payload keep alive: 1 43787.66 requests per second

This is much better, but still sucks. For a similar price you could get a dedicated box at SoftLayer, our current provider, with more than a double performance AND good upgrade options.

Surprisingly, more expensive EC2 instances could not deliver any much higher performance, being in every respect less performant than any decent dedicated box. You could find more benchmarks at the Redis website. Our office quad-core server was also able to get about 100 000 inserts per second.

I know the power of Amazon is not exactly the "inexpensive hardware", but rather flexibility, range of added services, probably easier administration… but there are kind of services you really do not want to put in virtualized environment. Talking to "bare metal" is extremely important when running Redis, and probably any memory-intensive software.

Also, since Redis datasets must fit in the memory, it would be nice to be able to get cheap boxes (slow drives are ok) with lots of ram. Still, it is worth considering if using Amazon EC2 is the best option.

Still, I am considering running the project on EC2 in the initial period, but you really need to be careful about the choice.

How it refers to Wikidot?

When I was testing EC2 instances with PostgreSQL installed, populated with a copy of Wikidot.com database, I was getting only 50% of the performance of the dedicated server for queries that for sure all used only cached data, even on the fastest instances. So it looks like moving our database server to EC2 would significantly decrease our performance. At this moment it is not acceptable. This post on Amazon forums would suggests memory bandwidth problems in EC2 instances.

Previously I have been presenting a possible migration to Amazon EC2 services. After a while it looks like our whole database / webserver infrastructure would need to be reconsidered to benefit from EC2 architecture. In the end we will need to partition our datasets (sharding) and probably modify storage for uploaded files, but honestly I would rather move this moment in time as far as I can, and as long as we still have plenty of options within our current setup.

BTW: A weekend (short) project is a kind of project that should take only a few days to complete, or at least to build a reasonably working and functional prototype. It should be fun and educational, give a chance to explore new solutions and technologies. Perfectly I would welcome more people on-board.

by michal-frackowiak

Two things today:

Unicorns, Rainbows On Demand

As a vast majority of geek-oriented people out there I hate unicorns, capricorns, little ponies, magic rainbows and all that stuff. But the site we have discovered, Cornify.com - Unicorns and rainbows on demand is so brilliant I could not resist.

Click the button (several times) below to see what I am talking about ;-)

A bit more description how to cornify your wiki is available on our Snippets page

3D Tag Cloud

So what is so cool about rotating cloud tag? Well, I can impress people with it. It does not add any usability improvements, but obviously a rotating tag cloud catches attention and affect users' behaviour on your site. From the usability point of view — totally useless, but still cool ;-)

Again, description and howto comes from Snippets wiki

by michal-frackowiak

So here are a few facts:

• In max 6 months from now we need to reorganize our server infrastructure anyway— simply because demand for Wikidot is growing.
• We want to create a more self-configuring (self-healing) infrastructure, removing as many single-points-of-failure as possible.
• We want to be able to scale just by throwing additional servers into the cluster, possibly within minutes, and be able to throw them away when we do not need them.
• Managing hardware is a pain and we would rather move our effort into higher-level management.
• We want better separation of various services we are running (like daily tasks, log analysis etc.). Some of them deserve separate servers.

Some of the solutions that more or less comply with the above requirements:

• Simply add more hardware — it would work. it works for most applications. But is costly and not elastic. Difficult to add resources on-the-fly. With SoftLayer server provisioning is really good, we can have a new server within 2-3 hours.
• Virtualize our own hardware — SoftLayer has an offering here, we would need to look deeper into this. But is it really what we need? Our hardware would still be a point of failure.
• Use virtualized instances — we could get virtual instances from a 3rd party provider (is SL going to offer virtual servers?). The problem is however that we need some good performance from our boxes too, and thus we would need a good degree of control over it.
• Use a "pure cloud solution" like Google App Engine. GAE is out-of-question because we would need to rewrite our code, and I am not sure it can run something as complex as Wikidot, with a lot of background services etc.
• Use Amazon EC2 — guess what, it looks like an optimal solution.

There is more info about EC2 here. Basically you can rent instances (virtual servers) on a per-hour basis ($0.10 - $0.80 per hour), there is a nice API to manage your instances, storage, IP addresses etc. EC2 deserves a separate article obviously. I can only say that:

• I am using EC2 + S3 + SQS from one of my other project and it rocks in terms of performance and scalability. A properly-designed application can handle millions of visits per day without much magic.
• Pricing is nice, but does not necessarily mean we could save any $$ moving to AWS. You pay only for what you are using, no up-front fees or plans. Good for small startups that grow over time.
• AWS meets most of our above requirements.

One of the nice things about EC2 is that you can get a new server within 2-5 minutes, use it as long as you wish, and terminate it. Everything is automated. There are dozens of Linux images available, tons of documentation and support from the community and Amazon itself (this one is paid extra).

Since we already had experience with AWS, we decided to run a couple of tests last Friday, just to get a glimpse of the situation. So what we did was:

• We set up a simple cluster configuration (1 front-end web server + 1 database server)
• We installed Wikidot on it using a fresh database dump.
• We tried to simulate read-only traffic by taking access logs from Wikidot.com and throwing the requests to the test server in parallel.

The only thing we are still concerned about is database performance in the virtualized (Xen) environment, over network-attached drives. Although in our tests the database was doing really fine, we need to do more read+modify tests.

It looks like Wik.is (although a bit less popular than Wikidot) is using quite an interesting cluster on EC2.

Amazon AWS is getting a lot of attention recently, and the recent changes and future enchancements look really promising.

After performing the tests we terminated the servers — Amazon charged us about $3 ;-)

BTW: This opens one more interesting case for us. SaaS with Wikidot — would you like to get your very own Wikidot installation within minutes, hosted on a virtual server instance? Yes, we know how to do this, and we might automate the whole process someday.

by michal-frackowiak

First of all, when designing current servers' configuration, we concentrated on having each piece of hardware as reliable as possible, which means:

• we are using only high-quality servers from SoftLater (they are using SuperMicro servers and pack it with really great hardware)
• our servers have redundant power supplies, in case one PSU dies (which is one of the most common problems), second one works and the tech support can replace the faulty one without any interruption
• we are using only mirrored redundant disks (in RAID1 or RAID10) with hight quality components: controllers from Adaptac and high-quality SAS 15k disks for system and database + large SATA disks for file uploads.
• all the services are being monitored by external resources — we are using Pingdom to monitor services availability + our own monitoring stack to check health of hardware.
• we automate as much as possible, this includes health checks, error logs etc.
• we are having a few autoreplication mechanisms for database and filesystems.
• we also have periodic backups stored on remotely on S3.

So far, we are really, REALLY happy with SoftLater, our hosting provider. And this includes everything: great available hardware options, devoted support, datacenters with fast internet connection etc. If someone asks me about a datacenter, I always recommend SoftLayer. And as far as I know at least a few people moved there and are really happy with the choice.

OK, so back to the subject. What we were discussing yesterday can be summarized as: "we should not scale vertically, we should scale horizontally". What this means is that instead on relying on just a few servers we upgrade from time to time, we should create a more self-healing structure by adding more (less powerful) nodes and assume "by design" that some of them can fail without affecting availability of the service.

Why is it important? Because we need to avoid single points of failure. So far our setup works perfectly, giving us a very good overall uptime (it is still 99.9%). But the question I am always asking myself is: if something goes very wrong, how much trouble will it cause for us and our users? How much work before we bring service up after unrecoverable hardware failures? Or filesystem errors? Sure we eventually will fix things and all the data will be available, but how quickly? How fast can our hosting provider provision new servers to replace failed ones?

Although answers to the above questions can be still considered as satisfactory, with the growth of Wikidot and larger and larger datasets, current solutions simply become less and less effective. We would rather prefer another case: "What do you need to do when one of your servers fail? — NOTHING."

Just to give you a glimpse what my next blog post will be about and how we could improve our infrastructure: virtualization and cloud computing. Enough said for today ;-)

by michal-frackowiak

• I stayed at home, Lukasz was at the office. All of sudden network went down. Later we learned that half of our city (that get internet from TP S.A.) had problems.
• Later Piotr called that our office server is down and cannot boot up. It was one of the disks in RAID 10 array that failed and for some reason GRUB could not boot. It booted later after Piotr did some magic, now we just need to replace one drive asap.
• At 15.30 local time I got an alert email that Wikidot.com is down. Immediately i tried to log-in to the server - nothing. Ping - yes. Alive. But all services went down.
• After a few minutes we knew we must act. Piotr started re-assigning IP addresses of the web server to a backup server. Failed. Looks like the router could not handle this in real-time this time.
• Main server restart - nothing helps. We had a similar issue some time ago, we started the rescue mode (server boots from a rescue linux image, this is greatly automated by SoftLayer). Server is up. A year ago what prevented the system from booting was a forced fsck on one of the drives and this required a key pressed or so (as told by the SoftLayer support team). So we started disk checks. And this took almost an hour! S#*t!
• Meanwhile my friend called me as his car broke just 20 meters from our parking lot and he could not move it, so I went to help him.
• Server got up, everything was back to normal. Situation under control.

I am not afraid of fatal Fridays any more. I fear of Wednesdays.

by michal-frackowiak

After looking at the screenshots, watching short introductory video, I dashed to http://bespin.mozilla.com to try this out. I do not say this very often, but this is quite an excellent and amazing piece of work. Although I have been following the web development and new technologies and trends quite closely for the last couple of years, this is something that really convinces me that browser-based applications can once replace the native ones. Simply great.

See for yourself:

http://bespin.mozilla.com

PS. I was so impressed with this development that I had to stop my other important work and blog about it ;-)

by michal-frackowiak

PHP is extremely popular. Zend is putting a lot of money and effort to move PHP from the commodity market into enterprise and… it seems to work. PHP has very nice learning curve, although to produce good code in PHP it takes as long time and effort as in other languages. PHP is easy to deploy, runs well on commodity hardware…

But PHP sucks. And it really sucks badly for certain tasks, which we from time to time find out. So here is a short list, which by no means is complete, but compiles my thoughts of the language I have been using for the last 5 years:

Functional + Object-Oriented

PHP started as a purely functional language. The initial easiness was tempting for many web developers, therefore PHP exploded in popularity. The problem is that Zend wanted it to be a object-oriented. They tried twice. PHP 4 was the first attempt and it was bad. Terrible language constructs, mixing functional and OO features in a very complicated and unclear way.

In PHP 5 the OO part was mostly redesigned and right now PHP 5 follows the Java-style - it has Interfaces, Classes, proper constructors, inheritance… Even syntax follows the one of Java.

The pity is that PHP is still a functional language mixed with OOP. Come on, why is Array not a real object? Or a String? Why do we still need to use complicated built-in functions to manipulate them?

Quality of the open source code

Quality of the open source projects varies tremendously. Because PHP does not enforce you to use good programming style, developers tend to use a lot of shortcuts, homebrewed solutions and a lot of hacks. Most of the available code is PHP4-compatible, which means it uses different OO approach than PHP5.

What is worse, libraries often pollute global namespace with their own variables (and constants) and functions which is a potential source of conflicts.

Most importantly, when you are trying to use

error_reporting(E_ALL|E_STRICT);

which will be default in PHP6, you will quickly find that most of the open-source libraries produce TONS of warnings and notices.

What I have learned so far and is a general rule of thumb:

• Avoid PHP4-compatible libraries
• Look very carefully at PEAR stuff, it might be broken, see all pending bugs, see if it is maintained
• Test any library with the strict error reporting

Duality of errors/warnings + exceptions.

PHP5 uses Java-like Exception classes with try{} catch{} clauses. But it also uses legacy errors warnings, with all their handlers etc. This really does not play nice together. Especially if any of your external libraries tries to handle such errors by itself.

What you should do in a perfect world is to map all the errors and warnings into Exceptions, like we do. (Except for Fatal errors that cannot be handled by an application). But then, you can be really unpleasantly surprised when you use the above error_reporting(E_ALL|E_STRICT).

More like Java

As I am trying to point out, PHP tries to resemble Java. This is both good and bad.

Good:

• Introduces clear language structures that we all know, classes, interfaces, exceptions etc.
• Easy to design application by people experienced in Java
• Yes, you can make your application behave like in Java with a set of custom error mappings, directory structures, autoloaders, pseudo-packages etc. But it is hacking.

Bad:

• Several other conventions still apply: functions, warnings/errors, PHP4-like OO
• Very different underlying architecture from Java (share-nothing)
• Still an interpreted language
• It is not Java

OO features in Standard Library

Did you hear about the Standard PHP Library? Probably not. Me neither before I had to use iterator interfaces to create a well-behaving array-like objects or use chained class autoloaders.

• Why is this called a standard library?
• It only covers a small tiny portion of the language.
• Documentation is almost non-existant, looks like reserver for advanced programmers.

Zend Framework

Zend Framework is an attempt by Zend itself to create THE way of using PHP. It is a really good project and we are using parts of it extensively. It is PHP5+ only, encourages good programming practices (including testing), have a nice set of classes. It somehow tries to fix the lowish quality of PEAR packages. But:

• It is late. Too late.
• Most parts on the Zend Framework should be coded in fast C modules, not in PHP itself. Try the performance of date/time/calendar operations and you will see what I mean.
• ZF tries to fix broken features and add new ones to PHP, but they are often sub-optimal.

Share nothing

The point I made about writing large constructs in PHP itself is very important. PHP is a "share-nothing" type of solution, every web request is handled separately and after it the PHP engine is reset into the default state. All the classes need to be re-loaded and initialized with every single web request! The larger your codebase libraries are, the less performance you get.

The share-nothing architecture has some benefits: you do not need to care about freeing system resources, returning db connections to the pool etc. PHP does it for you, because once the request is processed, all resources are properly closed and released.

Garbage collection

PHP does not resolve circular references when performing its simple reference counting and copy-on-write garbage collection. Several times I had to rewrite the code because the garbage collector was unable to properly release memory from database model objects.

Web-only

Poor garbage collection also prevents any long-running programs to be written in PHP. Every program I have written in PHP that was running longer than a few days ended up by using all its available memory and crashing eventually.

In fact PHP has been designed only to handle web requests. ONLY. It is quite good at it, but at nothing else. You will not write a FUSE project with PHP,daemon, mail server, irc server, ANYTHING persistent.

PHP is not a standalone platform. Many projects combine PHP with Java, so that PHP handles web + logics and Java fits where persistance is required.

There are a lot of other problems I see with PHP: missing namespace support, no real unicode (try strlen('ąśćŁ'), debugging etc.

I am really looking forward to PHP alternatives. Java is IMHO out-of-question, but Python and Ruby are tempting solutions that eventually I might migrate to.

by michal-frackowiak

A few days ago I was listening to some 43folders podcasts, I decided to give GTD a try. Although there are countless resources and tutorials for GTD, I decided the best way to start is the David Allen's book itself. So here I have it.

GTD is a religion for some, methodology for others. I myself am looking for better way to organize my time, my projects and tasks. Especially since I want to spend as much time with my family as I can (work is not the only thing in life), I am exploring ways to optimize productivity and efficiency.

Every person is different and can achieve productivity balance in different situations. I am certainly a kind of a person that can easily focus on particular problem. Creating Wikidot in just a few months is a good example. During the initial development, having only limited time to produce a working service, I developed a really strict and effective way of working, solving problems, and move forward. Studying theoretical physics and doing a PhD in astrophysics were quite interesting and demanding experiences too.

The problem I (and probably vast majority of people) have is distractions. And believe me, there are more and more of them. Sure it would be great to spend a few days solving a single problem, firewalling myself from the rest of the world, but work, family, daily duties, company management — all those important stuff need their place too. They are important and I want them to be important, but sometimes they are in apparent conflict with each other.

My other reason is: I want to have more time to do interesting things. Which means I need to be more efficient with things I am doing right now.

For the last couple of years I have implemented elements of GTD, some of them come naturally and some were based on tips found on the internet. Most of them, especially Zero Inbox, worked really well. Hopefully I will be able to approach project management more systematically now.

So here is my first project: implement more GTD in my workflow. Next action: read the book.

by michal-frackowiak

From my estimate, when creating initial version of Wikidot, making it work in IE 6 took extra 30% of total development time. Today, when adding functionality to Wikidot, special care must be taken to make sure it works with outdated browsers like IE6.

Come on, why do we have to support browser that was released in 2001 and never really evolved since then? The problem is that many people, with older hardware and older versions of Windows, simply do not bother to upgrade their browser, "because it still works". Also, in many corporate environments internal policies lock employees to use specific versions of software, including web browsers.

Hopefully, as I look at Dec 2008 statistics for Wikidot usage, only 15% of our traffic comes from IE6. IE7 accounts for 25%. This is good. Do not take me wrong — IE7 is not that much better than IE6 when it comes to standard compliance, but it is much much better.

Here is the full browser usage stats for Dec 2008:

So, Firefox is the clear winner. This is good. Chrome does surprisingly well as for a new browser in the market.

So with IE6 accounting only for 15% of the traffic, when can we drop support for it? This is the question that a lot of people ask.

I would say — for Wikidot.com, not anytime soon. But if we decide to create any new product, IE6 will not be supported for sure.

And the reason I am writing about IE6 today is that I discovered a nice website, http://idroppedie6.com/ :

It is a list of sites that actually dropped support for IE6. I was quite surprised to see http://gmail.com on the list. Still the website is not extremely popular, I wish them good luck.

As a conclusion I think that in 2010 IE6 support among new websites will be marginal. Web evolves, there are so many great new technologies, solutions and patterns that would simply take too much time to make them work with IE6. Would developers spend 30% more time on their project trying to reach extra 10% of users?

Here is a nice chart created by Asa Dotzler:

by michal-frackowiak

The feature is really great (we have it enabled for a few testing accounts) and allows you to be instantly notified about page edits, forum discussions, page comments etc. for pages, categories or whole sites.

So far our favorite feature is page comments by email. If someone adds a comment to a page you are wathing, you are almost instantly notified about it. Here is an example. Let us say that James commented on my blog post. Within minutes I got an email with full content of his comment.

The feature should be available for all users later this week. What we also think would be very nice is the "reply by email" option where you can simply hit a "Reply" button in your email program and your reply automatically gets posted as a comment too. But this will come later since it requires a bit more work.

Obviously email notifications solve an important problem of users not being notified what is going on in their sites. We have seen many contributions to both public and private wikis left unnoticed by other members. Although we do provide full activity stream by RSS already, sadly RSS is still not common and easy enough to use to watch changes at Wikidot.

So far we have seen this feature alone enormously improved our online activity at our own wikis. Hopefully this will also work for many other users.

by michal-frackowiak

What really happened?

From what we were able to reconstruct, one of the file operations related to live backup of file uploads went crazy and started consuming loads of memory. Usually we are duplicating all files belonging to each of the sites in real-time to another server to have a hot-backup for both backup purposes and to have a hot-swap server in case the main server dies.

The problem started at about 2 AM and we could see radical drop in memory used for disk caching. This is usually a bad sign and we have a monitoring set up for this, so when this happens we are getting warning emails. The problem with tracing memory-related issues in Linux is that utilities do not show memory consumed by the kernel itself, and this include memory for handling filesystems. (Try doing ls */*/* a couple of times on a really large filesystem and you will see memory vanish quite quickly).

When memory went low, other services that rely on kernel disk caches (like the PostgreSQL database) started making a lot of disk operations, increasing system load and decreasing responsiveness. Under high load PostgreSQL is often unable to accept new connections form the PHP application.

Then our PHP backend died, also because of memory issues, producing increased "500 Internal Server Errors". From that moment nothing dramatic happened, besides the prolonged unavailability of the service.

At this point please allow me to say: this is not the kind of situation that happens often. In fact, such large memory outage happened for the first time. Suddenly about 7 GB of RAM disappeared. For most part of the day we were trying to trace the issue and are quite sure it was partly an unfortunate mixture of kernel operation and method of live backup of filesystem.

So far however we have never had experienced similar issues.

How we reacted

Wikidot is fully monitored from external servers for web availability, memory usage, other services. Monitoring services automatically send us emails and SMSes when anything suspicious happens. What failed was the human factor and the fact all those reporting methods could not wake us up in the middle of the night.

The problem was easy to fix once I got my hands on a computer. After stopping filesystem replication and restarting core services (www server) things got back to normal. Other services were restarted too to make sure things are ok.

For most part of the day we were analyzing logs and trying to figure out what really happened. I guess Piotr has a better and more detailed opinion on the internal source of the problem, but we mostly agree on what I have written above.

How can we protect Wikidot for the future?

Random things always happen, it is more important how we react to them and how we try to avoid them. We really put a lot of effort into eliminating situations like this one. The obvious solution is to make our SMSes wake us at night. So far we had very little monitoring alerts (servers are running very even) and I assumed this would work.

What is also important for our users, we have several safety measures to keep your data intact, including data replication and periodic full backups to external servers (to a different datacenters).

Also we are analyzing server usage reports every day to prevent such events from happening.

Until the current issue is fully resolved, we have removed the live file replication from the service and we will probably reimplement it in a different way.

Apart from situations like that, I need to admit that our servers are working very reliably, with a very high uptime. According to external measures (from Pidgin), even including the today outage, we still have over 99.8% for the last couple of months.

I really hope today's outage did not cause much inconvenience.

Best regards

by michal-frackowiak

My problem with blogging is that I wanted to write a frequently updated blog with interesting and deep articles. And every day I had one or more great ideas, obviously worth writing about. But at the end of the day I never have time or strength to write long articles.

But it appears (thanks for feedback) that many of my readers would rather just read things more frequently instead of well-polished articles. And I think I will follow this direction.

From now on, my new resolution is to post every day. Not necessarily more. Just every day. A few ad-hoc rules are:

• No minimum number of words in the article.
• If I have nothing to say, I will not try to try. Instead I will link to resources I find interesting, but with at least a few words of comment.
• I will try to keep this blog on topics: wikidot, programming etc., but anything else that catches my attention is also OK.
• I will try to care more about proper English (thanks for help, Mom!), being a non-native speaker makes it more difficult.

I will start today.

So send me a bunch of nasty emails if you do not see at least 28 blog posts this month and at least one per day :-D

by michal-frackowiak