Google ♥

Well, Siege and Sproxy serve this exact purpose!
Sproxy is a proxy that we can use to collect a list of urls to feed Siege with, let’s see how.

First, we obviously need to install all the software we will need.

Make sure you have make, g++, perl-URI and perl-libwww-perl installed (or liburi-perl and libwww-perl on debian based distros).

1) Sproxy:

wget ftp://ftp.joedog.org/pub/sproxy/sproxy-latest.tar.gz
tar xzvf  sproxy-latest.tar.gz
./configure
make
make install

2) Siege:

wget ftp://ftp.joedog.org/pub/siege/siege-latest.tar.gz
tar xzvf  siege-latest.tar.gz
./configure
make
make install

We know need to collect some urls using Sproxy.

Launch sproxy and take note of the port it binded to, we will use this information to connect to it as a standard proxy. Here it comes the cool thing, every url we will visit through sproxy will be written to a file in a format Siege understands.

The url list will be written in ~/urls.txt, you can override this using he flag -o: for example:

sproxy -o /home/user/benchmark/urls.txt

Collecting urls manually using a browser (e.g: Firefox) could be a long process, especially for a forum or for websites with a lot of content. Wouldn’t it be cool to let an automatic tool do the hard work for us? Of course, so wget comes to the rescue!
Yes, you heard (read?) right, wget is a really powerful program even if you’ve always used it for downloading files only, so here’s an example of how we can crawl a website through sproxy using wget:

wget -r -o verbose.txt -l 0 -t 1 --spider -w 1 -e robots=on -e "http_proxy = http://127.0.0.1:9001" "http://www.example.com"

-w: specify the seconds to wait between requests to avoid overloading the webserver
-l: the number of maximum levels to descend, 0 means infinite.

For other options such as excluding some directories/file extensions please refer to “man wget”.

After some minutes/hours of crawling we should have a lot of urls in our sproxy file, let’s filter out all the duplicates:

sort -u -o uniq_urls.txt urls.txt

Finally, we can launch the actual siege test that will simulate real users visits across the urls we previously collected:

siege -v -c 50 -i -t 3M -f uniq_urls.txt -d 10

-v will give you more information of what siege is doing during the test.
-c specifies the number of total users simulated.
-i indicates to hit the urls in a random fashion.
-t specify the running time of siege, you can use S, M or H
-f  is pretty obvious.
-d specify the maximum delay between a simulated user requests.

Maybe the last one is a little tricky to understand, I’ll try to explain it better: siege in order to simulate a real users load randomize the time wait between page requests of a user, so -d indicates the maximum seconds between A user’s “clicks”. 30 seconds is a good value if you think of users behavior on a website, but of course you can set whatever you think is appropriate.

That’s all, feel free to drop a comment if you are having troubles, there’s something wrong with the article or you just want to say thanks.

The documentation (that you should definitely read) is well written and  easy to follow if you have any experience in working with json API and OAuth, but if you haven’t, and you don’t know Ruby (the programming language used for their sample application) it’s possible to find it tricky.

So, here’s the sample PHP application:

First, install the PECL OAuth library

pecl install oauth

And add the extension to your php.ini file

extension=oauth.so

The first file is the configuration file (misoconfig.php)

define('OAUTH_CONSUMER_KEY',"yourconsumerkey");
define('OAUTH_CONSUMER_SECRET',"yourconsumersecret");
define('OAUTH_REQUEST_TOKEN_API', 'https://gomiso.com/oauth/request_token');
define('OAUTH_ACCESS_TOKEN_API', 'https://gomiso.com/oauth/access_token');
define('OAUTH_AUTHORIZE_API', 'https://gomiso.com/oauth/authorize');
define('OAUTH_TMP_DIR', function_exists('sys_get_temp_dir') ? sys_get_temp_dir() : realpath($_ENV["TMP"]));
define('MISO_USER_AGENT', 'youruseragent');

The next file requests a token, saves it in a tmp file and redirects the user to Miso for the authentication, if the user allows your application, he will be redirected to the callback url

require 'misoconfig.php';
$req_url = OAUTH_REQUEST_TOKEN_API;
try {
$o = new OAuth(OAUTH_CONSUMER_KEY,OAUTH_CONSUMER_SECRET,
OAUTH_SIG_METHOD_HMACSHA1,OAUTH_AUTH_TYPE_FORM);
$o->enableDebug();
$resp = $o->getRequestToken($req_url, "http://www.yourwebsite.com/callback.php");
} catch(OAuthException $E) {
echo "Response: ". $E->lastResponse . "\n";
}
file_put_contents("/tmp/last_request",$resp['oauth_token']."\n".
$resp['oauth_token_secret']."\n");
print_r ($resp);
header("Location: ".OAUTH_AUTHORIZE_API."?oauth_token=".$resp['oauth_token']);

The last file (callback.php) lets us acquire the access token by using the previous request token and the oauth verifier:

require 'misoconfig.php';
$acc_url = OAUTH_ACCESS_TOKEN_API;
list($token,$secret) = file('/tmp/last_request');
$token = trim($token);
$secret = trim($secret);
$verifier = $_GET['oauth_verifier'];
try {
$o = new OAuth(OAUTH_CONSUMER_KEY,OAUTH_CONSUMER_SECRET,
OAUTH_SIG_METHOD_HMACSHA1,OAUTH_AUTH_TYPE_URI);
$o->enableDebug();
$o->setToken($token,$secret);
$resp = $o->getAccessToken($acc_url, NULL, $verifier);
file_put_contents("/tmp/access_token.txt",serialize($resp));
} catch(OAuthException $E) {
echo "Response: ". $E->lastResponse . "\n";
}
print_r($resp);

We’re done! We can now access Miso’s API using the access token, you can test it with something like this:

try {
$oauth = new OAuth(OAUTH_CONSUMER_KEY,OAUTH_CONSUMER_SECRET,OAUTH_SIG_METHOD_HMACSHA1,OAUTH_AUTH_TYPE_AUTHORIZATION);
$oauth->setToken($oauth_token,$oauth_token_secret);
$oauth->fetch('http://gomiso.com/api/oauth/v1/users/show.json', null, OAUTH_HTTP_METHOD_GET, array('User-Agent'=>''.MISO_USER_AGENT.''));
$response_info = $oauth->getLastResponseInfo();
header("Content-Type: {$response_info["content_type"]}");
echo $oauth->getLastResponse();
} catch(OAuthException $E) {
echo "Exception caught!\n";
echo "Response: ". $E->lastResponse . "\n";
}

Please note that this is just an example that should let you understand how to access Miso API, it won’t work out of the box just by uploading this code snippets into your FTP.

Here’s how to perform the upgrade in 3 simple steps:

First, make sure you have the update-manager-core package installed:

sudo apt-get install update-manager-core

Now we use nano to edit /etc/update-manager/release-upgrades and set Prompt=normal

nano /etc/update-manager/release-upgrades

Last step, launch the upgrade tool and follow the instructions:

sudo do-release-upgrade -d

Let’s see how: imagine you need to run a PHP script every 10 seconds to check whether a webiste is responding, here’s what you would need to put in crontab:

* * * * * /usr/local/bin/php /var/www/uptime.php
* * * * * sleep 10; /usr/local/bin/php /var/www/uptime.php
* * * * * sleep 20; /usr/local/bin/php /var/www/uptime.php
* * * * * sleep 30; /usr/local/bin/php /var/www/uptime.php
* * * * * sleep 40; /usr/local/bin/php /var/www/uptime.php
* * * * * sleep 50; /usr/local/bin/php /var/www/uptime.php

What we’re doing here, is starting 6 processes altogether every minute with progressive sleep times and that results in running the PHP script every 10 seconds. As simple as that.

Fortunately FeedBurner also provides a ping service that let you manually update your RSS Feed; but visiting a website every time you need to update your feed isn’t very practical, instead we can use cURL to do this automatically every time we publish a new article or something.

Here’s the PHP code (you need to make sure that your hosting has cURL enabled):

$ch2 = curl_init();
curl_setopt($ch2, CURLOPT_URL, 'http://feedburner.google.com/fb/a/pingSubmit');
curl_setopt($ch2, CURLOPT_POST, 1);
curl_setopt($ch2, CURLOPT_POSTFIELDS, 'bloglink=http://www.example.com');
$result = curl_exec($ch2);
curl_close($ch2);

You need to place this right after the code that publishes a new item on your website, so that the RSS Feed is immediately updated with the new articles (or whatever you are publishing).

Another way to do this is by putting the code in a cronjob and run it every X minutes, but use this only if you can’t get  the other method working, it’s way better.

Here’s the code to use cURL directly from the command line (use this if you are planning to go with the cronjob method):

curl -d 'bloglink=http://www.example.com' http://feedburner.google.com/fb/a/pingSubmit

Also you don’t need to provide the full RSS Feed URL, the website base URL will do just fine.

The purpose of this article isn’t explaining what mysqlnd is and what it does, for that you can start by reading the official documentation and this nice blog post.

I did these benchmarks mostly because I couldn’t find any proper comparison between the two drivers and I wanted to know which driver I should be using on all the servers I manage (performance-wise), I then figured that making this public could be useful to someone else, so here it is.

Test System
VMware Virtual Machine (VMware Workstation 7.1.0 build-261024)
Host RAM: 8GB
Host CPU: Intel Q8400
Host OS: Windows 7 Enterprise x64
VM RAM: 4GB
VM CPU: 1 CPU – 4 Cores
VM OS: Debian 5 x64

Software Utilized
nginx 0.7.67
php-fpm (the one included in php 5.3.3)
MySQL 5.5.5-m3 (with the malloc lib provided by google-perftools)
APC 3.1.4-dev
ApacheBench 2.3 (installed on another PC, connected to the Host with gbit ethernet)
Wordpress 3.0.1

nginx compile options: ./configure –sbin-path=/usr/local/sbin –with-http_ssl_module –without-mail_pop3_module –without-mail_imap_module –without-mail_smtp_module –with-http_stub_status_module

PHP compile options (mysqlnd): ./configure –enable-fpm –with-mcrypt –with-zlib –enable-mbstring –with-openssl –with-mysql=mysqlnd –with-mysql-sock –with-pdo-mysql=mysqlnd –with-gd –with-jpeg-dir=/usr/lib –with-freetype-dir=/usr/local/lib/ –enable-gd-native-ttf –with-curl=/usr/include/curl –enable-soap

PHP compile options (libmysqlclient): ./configure –enable-fpm –with-mcrypt –with-zlib –enable-mbstring –with-openssl –with-mysql –with-mysql-sock –with-pdo-mysql –with-gd –with-jpeg-dir=/usr/lib –with-freetype-dir=/usr/local/lib/ –enable-gd-native-ttf –with-curl=/usr/include/curl –enable-soap

MySQL compile options: ./configure –with-plugins=innobase

The first test measures the number of requests per second targeting a fresh install of WordPress that can be handled using the two different drivers. The benchmarking tool used is ApacheBench launched through this script, as you can see the script registers the results of 5 different runs, I then calculated the average.

Amount of CPU utilized by 3 concurrent requests: ~75%
Amount of CPU utilized by 100 concurrent requests: 100%

libmysqlclient

mysqlnd

As you can see with APC enabled libmysqlclient can handle more r/s than mysqlnd, but if you disable APC the behavior completely changes.
Comparing only the pure numbers is not very useful, so I converted the results in percentage:

Gain achieved by using libmysqlclient over mysqlnd with APC enabled (3 concurrent requests): 2,09%
Gain achieved by using libmysqlclient over mysqlnd with APC enabled (100 concurrent requests): 1,01%
Gain achieved by using mysqlnd over libmysqlclient with APC disabled (3 concurrent requests):  3,05%
Gain achieved by using mysqlnd over libmysqlclient with APC disabled (100 concurrent requests): 5,94%

So if you only care about pure performances and you are using APC you should probably stick with libmysqlclient, but if you aren’t using APC or you care about RAM consumption too, you should read on.

Now, imagine you need to host a php/mysql based website on a VPS that has only 128MB of RAM, what would be the best driver to choose (RAM consumption-wise)?

For this test I used 20 php-fpm child and I measured the RAM consumption after 30 seconds of multiple requests generated by ApacheBench, this is the simple script I made to register the exact amount of RAM used by every PHP process:

sleep 30
pmap $(pidof php-fpm) | grep total | awk '{ print $2 }' | sed s/.$//

In order to get the most accurate results possible I run all tests three times and then I calculated the average:

libmysqlclient

mysqlnd

RAM saved by using mysqlnd over libmysqlclient with APC enabled: 0,41%
RAM saved by using mysqlnd over libmysqlclient with APC disabled: 1,18%

Oh well, if your biggest concern is RAM occupation you should definitely use mysqlnd.

Of course which MySQL driver you use is only a tiny part of what can be changed on a server in order to achieve better performance, but when it comes to optimizing things, every little detail counts.

ab is a tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server. It is designed to give you an impression of how your current Apache installation performs.

In order to make sure that a benchmark involving the use of ApacheBench is run without any distortion due to, for example, wait times that could lead to cache expiration I needed a very simple bash script, so here it is:

#!/bin/bash
NUM=1250
CONC=3
SLEEP=15
FILENAME=Benchmark
URL=http://www.example.com/
echo "a started"
ab -n $NUM -c $CONC $URL > $FILENAME-$CONC-$NUM-a.txt
echo "a completed"
sleep $SLEEP
echo "b started"
ab -n $NUM -c $CONC $URL > $FILENAME-$CONC-$NUM-b.txt
echo "b completed"
sleep $SLEEP
echo "c started"
ab -n $NUM -c $CONC $URL > $FILENAME-$CONC-$NUM-c.txt
echo "c completed"
sleep $SLEEP
echo "d started"
ab -n $NUM -c $CONC $URL > $FILENAME-$CONC-$NUM-d.txt
echo "d completed"
sleep $SLEEP
echo "e started"
ab -n $NUM -c $CONC $URL > $FILENAME-$CONC-$NUM-e.txt
echo "e completed"
exit 0

Cosa scegliere quindi? Sembra facile, ma non sono poi molti i CMS PHP creati specificamente per l’amministrazione di un blog. In attesa che Antonio sviluppi la piattarforma di blogging ideale mi sono messo alla ricerca di qualcosa che facesse al mio caso e, dopo essermi imbattuto in cessi intervallati da CMS aggiornati al 2005, pensavo di averlo trovato: Serendipity.

Beh inutile dire che è stato un gigantesco FAIL: dopo averci perso una giornata a preparare il tutto, installare i plugin necessari (ne servono un po’ dato che le funzioni “out of the box” sono davvero basilari), modificare un po’ il tema, ed altre cosucce; vado finalmente a spostare i files nella root del sito per mettere online il tutto (avevo installato il CMS in una sottocartella per i test), modifico di conseguenza la configurazione e… puff! Non l’avessi mai fatto!

Ecco il simpatico errore apparso:

serendipity error: could not include /path/serendipity_config_local.inc.php - exiting.
Please create the file yourself or check permissions

Inutile dire che il file c’era, aveva i permessi giusti, ecc. Dopo altro tempo perso tra infinite prove e googlate inutili (sembra che questo problema, una volta presentatosi, nessuno sia mai riuscito a risolverlo) ho optato per un bel “rm -r serendipity/” e ho, a malincuore, (re)installato WordPress.

Ecco, avevo bisogno di un primo post per mettere quantomeno online il tutto, pur essendo io il re dell’incostanza e il peggior nemico della grafica cercherò di lavorare al blog per renderlo decente e addirittura(!) pubblicare qualche contenuto interessante.