niden.net

Downgrading PHPUnit from 3.6 to 3.5 [PHPUnit][Linux][HowTo]

2012-01-15T15:44:00.000-05:00

Recently I had to rebuild my computer, and decided to install Linux Mint 12 (Lisa), which is a very lean installation - for my taste that is.

Going through the whole process of reinstalling all the packages that I need or had, PHPUnit was one of them. Easy enough a couple commands did the trick

sudo apt-get install php-pear
sudo pear upgrade PEAR
sudo pear config-set auto_discover 1
sudo pear install pear.phpunit.de/PHPUnit

I wanted to run my tests after that, only to find an error in the execution:

PHP Fatal error:  Call to undefined method PHPUnit_Util_Filter::addFileToFilter()

 in /home/www/project/library/PHPUnit/Framework.php on line 48

At first I thought that it was a path error, so I included the /usr/share/php/PHPUnit and others in the php.ini file but with no luck. With a little bit of Googling I found out that there have been some changes in the 3.6 version of PHPUnit and things don't work as they did before.
Effectively, 3.6 had some refactoring done and thus the line:

PHPUnit_Util_Filter::addDirectoryToFilter("$dir/tests");

changed to

PHP_CodeCoverage_Filter::getInstance()->addDirectoryToBlacklist("$dir/tests");

Since I didn't want to change my whole test suite, I had to find a solution i.e. downgrade PHPUnit to 3.5.

Unfortunately specifying the version directly did not wok

sudo pear install phpunit/PHPUnit-3.5.15

since it would pull the latest version again and I would end up with the 3.6 files.

So I went one step further and installed specific versions of the relevant dependencies to satisfy the 3.5.15 version.

Uninstallation of 3.6

pear uninstall phpunit/PHPUnit_Selenium
pear uninstall phpunit/PHPUnit
pear uninstall phpunit/DbUnit
pear uninstall phpunit/PHP_CodeCoverage
pear uninstall phpunit/PHP_Iterator
pear uninstall phpunit/PHPUnit_MockObject
pear uninstall phpunit/Text_Template
pear uninstall phpunit/PHP_Timer
pear uninstall phpunit/File_Iterator
pear uninstall pear.symfony-project.com/YAML

Installation of 3.5.15

pear install pear.symfony-project.com/YAML-1.0.2
pear install phpunit/PHPUnit_Selenium-1.0.1
pear install phpunit/PHP_Timer-1.0.0
pear install phpunit/Text_Template-1.0.0
pear install phpunit/PHPUnit_MockObject-1.0.3
pear install phpunit/File_Iterator-1.2.3
pear install phpunit/PHP_CodeCoverage-1.0.2
pear install phpunit/DbUnit-1.0.0
pear install phpunit/PHPUnit-3.5.15

I hope you find the above useful :)

Change the encoding of a MySQL database to UTF8 [How-To][PHP][MySQL]

2011-12-19T22:00:00.000-05:00

Overview
As applications grow, so do their audiences. In this day and age, one cannot assume that all the consumers of a web based application will live in a particular region and use only one language. Even if the developer assumes that one country will be served by the particular web application, there are instances that the latin1 character set will not suffice in storing data.

Therefore, developers and database designers need to implement an encoding on their database that will safely store and retrieve any kind of data, not only latin1 based (i.e. the English alphabet).

For MySQL this encoding is utf8_general_ci.

The problem
MySQL usually comes with the latin1_swedish_ci encoding as a default. This encoding will allow the developer to store data of course but when non latin1 characters need to be stored, there will be a problem. Effectively latin1 encoding will store data in 8 bits but some languages like Japanese, Thai, Arabic, even French or German have special characters that need more space in the storage engine. Trying to store a 16 bit character in a 8 bit space will fail all the time.

Latin1 based database:
Input: abcdef...ABCD...#$
Output: abcdef...ABCD...#$

Input: 日本語 ภาษาไทย Ελληνικά
Output: ??? ??????? ????????

To combat this, all you have to do is change the encoding of your database to utf8_general_ci and the character set to utf8.

The solution
I wrote a script in PHP to analyze a database server and produce ALTER statements to be executed against your database(s). The script needs to run from a web server that supports PHP.

First of all, the encoding of the database will change with the relevant SQL statement. Following that each table's encoding will change, again with the relevant SQL statement. Finally, each TEXT/VARCHAR/CHAR etc. field's encoding will change towards the target encoding you specify in the configuration section (see below).

The safest way to transform data this way is to first change the field to a BINARY field and then change the field to the target encoding and collation.

Configuration
There are a few configuration variables that need to be set prior to running the script.

$db_user       = 'username';
$db_password   = 'password';
$db_host       = 'hostname';
$output_folder = '/home/ndimopoulos'; // Do not include trailing slash
$db_name       = 'mysql';             // Leave this one as is

set_time_limit(0);

/**
 * The old collation (what needs to be changed)
 */
$from_encoding = 'latin1_swedish_ci';

/**
 * The new collation (what we will change it to)
 */
$to_encoding = 'utf8_general_ci';

/**
 * The new character set
 */
$new_collation = 'utf8';

/**
 * Add USE <database>; before each statement?
 */
$use_database = TRUE;

The $output_folder is a folder that is writeable from your web server and it is where the .sql files will be created filled with the ALTER statements. The script will output one file <hostname>.sql which will contain all the ALTER statements for all databases. It will also create files for individual databases <hostname>.<database>.sql. You can use either the big file or the individual database files. The choice is yours.

The $from_encoding is what the script will check. In my script I was checking 'latin1_swedish_ci'.
The $to_encoding is what we need the encoding to be while the $new_collation is the new character set.

The $use_database is a flag that will allow you to generate statements such as:

USE <database>; ALTER TABLE <table>.....

if it is on, and if off, the statement will be:

ALTER TABLE <table>.....

Databases loop
The script opens a connection to the server and runs the 'SHOW DATABASES' command. Based on the result returned, it populates an array with the database names.

The script ignores two databases 'information_schema' and 'mysql', but editing the $exclude_databases array will allow you to ignore more databases if you need to.

mysql_connect($db_host, $db_user, $db_password);
mysql_select_db($db_name);

$dbs = array();

exclude_databases = array('mysql', 'information_schema',);

/**
 * Get the databases available (ignore information_schema and mysql)
 */
$result = mysql_query("SHOW DATABASES");

while ($row = mysql_fetch_row($result))
{
    if (!in_array($row[0], $exclude_databases))
    {
        $dbs[] = $row[0];
    }

The database names are stored in an array, so as not to keep the database resource active all the time. Had I not done that, I would have had to use three different resources (one for the database, one for the table and one for the field being checked - three nested loops).

Tables loop
The script then loops through the $dbs array and selects each database in turn. Once the database is selected, the 'SHOW TABLES' query is run and a $tables array is populated with the names of the tables in that database. At the same time the ALTER DATABASE statements are being generated.

mysql_select_db($db);

$db_output = '';

$statement  = "\r\n#-------------------------------------------------\r\n\r\n";
$statement .= "USE $db;\r\n";
$statement .= "\r\n#-------------------------------------------------\r\n\r\n";
$statement .= "ALTER DATABASE $db "
            . "CHARACTER SET $new_collation COLLATE $to_encoding;\r\n";
$statement .= "\r\n#-------------------------------------------------\r\n\r\n";

$db_output .= $statement;
$output    .= $statement;

$tables     = array();

$result = mysql_query("SHOW TABLES");

while ($row = mysql_fetch_row($result))
{
    if (!in_array($row[0], $exclude_tables))
    {
        $tables[] = mysql_real_escape_string($row[0]);
    }
}

Fields loop
The script then loops through the $tables array and runs the 'SHOW FIELDS' query so as to analyze each field.

$fields_modify = array();
$fields_change = array();

$result = mysql_query("SHOW FULL FIELDS FROM `$table`");
while ($row = mysql_fetch_assoc($result)) 
{
    if ($row['Collation'] != $from_encoding)
    {
        continue;
    }

    // Is the field allowed to be null?
    $nullable = ($row['Null'] == 'YES') ? ' NULL ' : ' NOT NULL';

    if ($row['Default'] == 'NULL') 
    {
        $default = " DEFAULT NULL";

    } 
    else if ($row['Default']!='') 
    {
        $default = " DEFAULT '" . mysql_real_escape_string($row['Default']) . "'";

    } 
    else 
    {
        $default = '';

    // Alter field collation:
    $field_name = mysql_real_escape_string($row['Field']);

    $fields_modify[] = "MODIFY `$field_name` $row[Type] CHARACTER SET BINARY";
    $fields_change[] = "CHANGE `$field_name` `$field_name` $row[Type] "
                     . "CHARACTER SET $new_collation "

                     . "COLLATE $to_encoding $nullable $default";

The two arrays generated ($fields_modify and $fields_change contain the MODIFY and CHANGE statements of each field. Using implode, we can easily construct the ALTER statement.

$statement .= "ALTER TABLE `$table` " . implode(' , ', $fields_modify) . "; \r\n";
$statement .= "ALTER TABLE `$table` " . implode(' , ', $fields_change) . "; \r\n";

Notes
You can use as mentioned earlier the $exclude_databases array to not allow certain databases to be processed. You can also use the $exclude_tables array to not allow certain tables to be processed.

The $exclude_tables_fields array allows you to exclude a field from being processed. However this is not tied to a database so any database/table that has a field with that particular name will not be processed. With a bit of refactoring you can make the script best work for your needs.

If you set the $use_database variable to TRUE then each line in your .sql statements will be prefixed with a 'USE <database>;' statement. This is to help the accompanying bash script to execute each statement in the respective database. If you intend on not running this process one statement at a time, you can set this to FALSE. You can then run each database .sql file (or the one that contains all of the statements from all databases) as one single command.

Server processing
Now that the relevant .sql files have been created, all you have to do is upload them on your web server. There are three ways of actually running the statements against the database.

Please make sure you backup your data first!

Single file processing

mysql -h<host> -u<username> -p<password> < /path/to/scripts/<host>.sql

The command above will run all the commands generated in the <host>.sql file for all databases. It is going to be taxing for your database server and there is no error handling or reporting. You can always pipe the results to an output file (just append "> /path/to/output/output.txt" at the end of the command). If this method fails for some reason (MySQL has gone away), it will be difficult to resume; you will need to edit the <host>.sql file to remove the statements that have already been processed.

Per database

mysql -h<host> -u<username> -p<password> <database_name> < \

       /path/to/scripts/<host>.<database>.sql

The command above will run all the commands generated in the <host>.<database>.sql file for that particular database. This method is similar to the one above. Again you can always pipe the results to an output file (just append "> /path/to/output/output.txt" at the end of the command).

Single file processing (per statement)

/path/to/scripts/process.sh

All you need to do is edit the process.sh script and change the relevant parameters to match your environment and upload it to your server. The source file that the process.sh script will read has to be generated with $use_database set to TRUE.

The process.sh script is:

#!/bin/bash

DBUSER=root
DBPASS=1234
DBHOST=localhost
SOURCE="/home/ndimopoulos/host.sql"
LOG="/home/ndimopoulos/conversion.log"

while read line
do
    TIMENOW=`date +%Y-%m-%d-%H-%M`
    echo START $TIMENOW $line
    echo START $TIMENOW $line >> $LOG
    /usr/bin/time -f "%E real,%U user,%S sys" -v -o $LOG -a mysql -h$DBHOST -u$DBUSER -p$DBPASS -e "$line"
    
    TIMENOW=`date +%Y-%m-%d-%H-%M`
    echo END $TIMENOW 
    echo END $TIMENOW >> $LOG

done < $SOURCE

exit 0

The script above will start reading the source file (<host>.sql) and execute each statement in turn, using time to measure the time taken to execute that command. The output ends up in a log file which can easily be tailed to view the progress and used later on for analysis. The results of the processing are also sent to the screen. You can change the parameters for the time command to match your needs.

The output will look something like the block below:

START 2011-12-08-23-46 USE mydatabase; ALTER TABLE `tablename` DEFAULT CHARACTER SET utf8;
 Command being timed: "mysql -uroot -p1234 -e USE mydatabase; ALTER TABLE `tablename` DEFAULT CHARACTER SET utf8;"
 User time (seconds): 0.01
 System time (seconds): 0.00
 Percent of CPU this job got: 0%
 Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.16
 Average shared text size (kbytes): 0
 Average unshared data size (kbytes): 0
 Average stack size (kbytes): 0
 Average total size (kbytes): 0
 Maximum resident set size (kbytes): 8192
 Average resident set size (kbytes): 0
 Major (requiring I/O) page faults: 0
 Minor (reclaiming a frame) page faults: 610
 Voluntary context switches: 11
 Involuntary context switches: 5
 Swaps: 0
 File system inputs: 0
 File system outputs: 0
 Socket messages sent: 0
 Socket messages received: 0
 Signals delivered: 0
 Page size (bytes): 4096
 Exit status: 0
END 2011-12-08-23-46

Conclusion
In order for a database to be best prepared to support localization, you need to make sure that the storage will accept any possible character. You can start by creating all your tables and fields with utf8_general_ci encoding, but for existing databases and data, you will need to run expensive processing queries on your RDBMS. Ensuring that the data does not get corrupted when performing the transformation process is essential so make sure you backup your databases before trying or running the output statements produced by the db_alter.php script.

PHP script (db_alter.php)

$db_user       = 'username';
$db_password   = 'password';
$db_host       = 'hostname';
$output_folder = '/home/ndimopoulos'; // Do not include trailing slash
$db_name       = 'mysql'; // Leave this one as is

set_time_limit(0);

/**
 * The old collation (what needs to be changed)
 */
$from_encoding = 'latin1_swedish_ci';

/**
 * The new collation (what we will change it to)
 */
$to_encoding = 'utf8_general_ci';

/**
 * The new character set
 */
$new_collation = 'utf8';

/**
 * Add USE <database> before each statement?
 */
$use_database = TRUE;

mysql_connect($db_host, $db_user, $db_password);
mysql_select_db($db_name);

$dbs = array();

$exclude_databases     = array('mysql', 'information_schema',);
$exclude_tables        = array('logs', 'logs_archived',);
$exclude_tables_fields = array('activities');

/**
 * Get the databases available (ignore information_schema and mysql)
 */
$result = mysql_query("SHOW DATABASES");

while ($row = mysql_fetch_row($result)) 
{
    if (!in_array($row[0], $exclude_databases))
    {
        $dbs[] = $row[0];
    }
}

$output = '';

/**
 * Now select each db and start parsing the tables
 */
foreach ($dbs as $db)
{
    mysql_select_db($db);
    $db_output = '';
    
    $statement  = "\r\n#-----------------------------------------------\r\n\r\n";
    $statement .= "USE $db;\r\n";
    $statement .= "\r\n#-----------------------------------------------\r\n\r\n";
    $statement .= "ALTER DATABASE $db "
                . "CHARACTER SET $new_collation COLLATE $to_encoding;\r\n";
    $statement .= "\r\n#-----------------------------------------------\r\n\r\n";
    
    $db_output .= $statement;
    $output    .= $statement;
    $tables     = array();
    
    $result = mysql_query("SHOW TABLES");
    
    while ($row = mysql_fetch_row($result))
    {
        if (!in_array($row[0], $exclude_tables))
        {
            $tables[] = mysql_real_escape_string($row[0]);
        }
    }
    
    /**
     * Alter statements for the tables
     */
    foreach ($tables as $table)
    {
        $statement = '';
        if ($use_database)
        {
            $statement  = "USE $db; ";
        }
        $statement .= "ALTER TABLE `$table` "
                    . "DEFAULT CHARACTER SET $new_collation;\r\n";
        $db_output .= $statement;
        $output    .= $db_output;
    }
    $statement .= "\r\n#-----------------------------------------------\r\n\r\n";

    $db_output .= $statement;
    $output    .= $statement;
    
    /**
     * Get the fields for each table
     */
    foreach ($tables as $table)
    {
        if (in_array($table, $exclude_tables_fields))
        {
            continue;
        } 
        
        $fields_modify = array();
        $fields_change = array();
        
        $result = mysql_query("SHOW FULL FIELDS FROM `$table`");
        while ($row = mysql_fetch_assoc($result)) 
        {
            if ($row['Collation'] != $from_encoding)
            {
                continue;
            }
            
            // Is the field allowed to be null?
            $nullable = ($row['Null'] == 'YES') ? ' NULL ' : ' NOT NULL';
            
            if ($row['Default'] == 'NULL') 
            {
                $default = " DEFAULT NULL";
            } 
            else if ($row['Default']!='') 
            {
                $default = " DEFAULT '"
                         . mysql_real_escape_string($row['Default']) . "'";
            }
            else 
            {
                $default = '';
            }
            
            // Alter field collation:
            $field_name = mysql_real_escape_string($row['Field']);
            
            $fields_modify[] = "MODIFY `$field_name` $row['Type'] "
                             . "CHARACTER SET BINARY";
            $fields_change[] = "CHANGE `$field_name` `$field_name` $row['Type'] "
                             . "CHARACTER SET $new_collation "
                             . "COLLATE $to_encoding $nullable $default";
        }
        
        if (count($fields_modify) > 0)
        {
            $statement = '';
            if ($use_database)
            {
                $statement = "USE $db; ";
            }
            $statement .= "ALTER TABLE `$table` "
                        . implode(' , ', $fields_modify) . "; \r\n";
            if ($use_database)
            {
                $statement = "USE $db; ";
            }
            $statement .= "ALTER TABLE `$table` "
                        . implode(' , ', $fields_change) . "; \r\n";
            
            $db_output .= $statement;
            $output    .= $statement;
        }
    }
    
    $bytes = file_put_contents(
        $output_folder . '/' . $db_host . '.' . $db . '.sql', $db_output
    );
}
    
$bytes = file_put_contents($output_folder . '/' . $db_host . '.sql', $output);

echo "<pre>$db_host $bytes \r\n$output</pre>";

Downloads
You can use these scripts at your own risk. Also feel free to distribute them freely - a mention would be nice. Both scripts can be found in my GitHub.

Fast serialization of data in PHP [How-To][Performance]

2011-11-22T17:34:00.001-05:00

Serializing/Unserializing data

Serialization is the process of converting a data structure or object state into a format that can be stored and "resurrected" later in the same or another computer environment. source

There are a lot of areas where one can use serialization. A couple are:

in a database (storing an array of options specific to the user),
in an AJAX enabled application (call to get a status update and display to the user without refreshing the whole page), etc.

Based on the the application, serializing and unserializing data can be a very intensive process and can prove to have a big performance hit on the overall system.

Options
The most obvious option for serializing and unserializing data are the serialize and unserialize PHP functions. A bit less popular are json_encode and json_decode. There is also a third option, using a third party module that one can easily install on their server. This module is called igbinary.

In this blog post I am comparing the three options, in the hope that it will aid you with your selection of the best option for you so as to increase the performance of your application.

I created a test script that used several arrays of data (strings, integers, floats, booleans, objects, mixed data, all of the data types) to test the speed and size of the serialization and speed of unserialization of each of the three candidate function pairs. I run the same function to serialize or unserialize the data respectively for 1,000,000 times so as to produce the results below.

The script that I have used is listed below:

$_testStrings = array(
   'AK' => 'Alaska', 'AZ' => 'Arizona', 'VT' => 'Vermont',
   'VA' => 'Virginia', 'AZ' => 'West Virginia',
);
 
$_testIntegers = array(0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 84, 144,);

$_testBooleans = array(TRUE, TRUE, FALSE, FALSE, FALSE, TRUE, TRUE,);
 
$_testFloats = array(
  0, 1.1, 1.1, 2.22, 3.33, 5.55, 8.88, 13.13, 21.2121, 34.3434, 
  55.5555, 84.8484, 144.144,
);
 
$_testMixed = array(
  'one', 13 => 'two', 0 => 25.46, 'four' => 0.007, 'five' => TRUE, TRUE => 42,
);
 
$_objectOne             = new stdClass();
$_objectOne->firstname  = 'Leroy';
$_objectOne->lastname   = 'Jenkins';
$_objectOne->profession = 'Gamer';
$_objectOne->status     = 'Legend';
 
$_objectTwo         = new stdClass();
$_objectTwo->series = 'Fibonacci';
$_objectTwo->data   = $_testIntegers;
 
$_testObjects = array($_objectOne, $_objectTwo,);

$_maxLoop = 1000000;

$_templateEncode = "%s [%s]: Size: %s bytes, %s time to encode\r\n";
$_templateDecode = "%s [%s]: %s time to decode\r\n";

set_time_limit(0);

$_output = '';

/**
 * Set the source arrays
 */
$_allTestData = array(
   'str' => $_testStrings,
   'int' => $_testIntegers,
   'bln' => $_testBooleans,
   'flt' => $_testFloats,
   'mix' => $_testMixed,
   'obj' => $_testObjects,
);

$_testSources = array(
   'strings'  => $_testStrings,
   'integers' => $_testIntegers,
   'booleans' => $_testBooleans,
   'floats'   => $_testFloats,
   'mixed'    => $_testMixed,
   'objects'  => $_testObjects,
   'all'      => $_allTestData,
);
 
/**
 * ENCODE DATA
 */

/**
 * Start each test
 */
foreach ($_testSources as $_area => $_source)
{
 /**
  * Start the timer
  */
 $_serializeStart = microtime(TRUE);
 
 for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
 {
  serialize($_source);
 }
 
 $_serializeEnd = microtime(TRUE);
 
 $_serializeOutput = serialize($_source);
 
 $_output .= sprintf(
  $_templateEncode,
  'serialize()', 
  $_area, 
  strlen($_serializeOutput), 
  $_serializeEnd - $_serializeStart
 );
 
 /**
  * JSON
  */
 $_jsonStart = microtime(TRUE);
 
 for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
 {
  json_encode($_source);
 }
 
 $_jsonEnd = microtime(TRUE);
 
 $_jsonOutput = json_encode($_source);
 
 $_output .= sprintf(
  $_templateEncode,
  'json_encode()', 
  $_area, 
  strlen($_jsonOutput), 
  $_jsonEnd - $_jsonStart
 );
 
 /**
  * igbinary
  */
 $_igbinaryStart = microtime(TRUE);
 
 for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
 {
  igbinary_serialize($_source);
 }
 
 $_igbinaryEnd = microtime(TRUE);
 
 $_igbinaryOutput = igbinary_serialize($_source);
 
 $_output .= sprintf(
  $_templateEncode,
  'igbinary_serialize()', 
  $_area, 
  strlen($_igbinaryOutput), 
  $_igbinaryEnd - $_igbinaryStart
 );
 
 $_output .= str_repeat('=', 20) . "\r\n";
}

$_output .= str_repeat('=:=', 20) . "\r\n";


/**
 * DECODE DATA
 */

/**
 * Start each test
 */
foreach ($_testSources as $_area => $_source)
{
 /**
  * Start the timer
  */
 $_data = serialize($_source);
 
 $_serializeStart = microtime(TRUE);
 
 for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
 {
  unserialize($_data);
 }
 
 $_serializeEnd = microtime(TRUE);
 
 $_output .= sprintf(
  $_templateDecode,
  'unserialize()', 
  $_area, 
  $_serializeEnd - $_serializeStart
 );
 
 /**
  * JSON
  */
 $_data = json_encode($_source);
 
 $_jsonStart = microtime(TRUE);
 
 for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
 {
  json_decode($_data, TRUE);
 }
 
 $_jsonEnd = microtime(TRUE);
 
 $_jsonOutput = json_encode($_source);
 
 $_output .= sprintf(
  $_templateDecode,
  'json_decode()', 
  $_area, 
  $_jsonEnd - $_jsonStart
 );
 
 /**
  * igbinary
  */
 $_data = igbinary_serialize($_source);
 
 $_igbinaryStart = microtime(TRUE);
 
 for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
 {
  igbinary_unserialize($_data);
 }
 
 $_igbinaryEnd = microtime(TRUE);
 
 $_igbinaryOutput = igbinary_serialize($_source);
 
 $_output .= sprintf(
  $_templateDecode,
  'igbinary_unserialize()', 
  $_area, 
  $_igbinaryEnd - $_igbinaryStart
 );
 
 $_output .= str_repeat('=', 20) . "\r\n";
}
 
echo '<pre>' . $_output . '</pre>';

Serializing results
When serializing data we are always concerned about the size of the result but also about the time it took for the data to be serialized.

As far as size is concerned, I have highlighted in bold the winner of each test. json_encode seems to be producing the smallest result in bytes for most of the tests.

Size comparison

Strings

serialize() [strings]: Size: 105 bytes, 1.8710339069366 time to encode
json_encode() [strings]: Size: 67 bytes, 1.5691390037537 time to encode
igbinary_serialize() [strings]: Size: 64 bytes, 3.2276048660278 time to encode

Integers

serialize() [integers]: Size: 121 bytes, 3.0198090076447 time to encode
json_encode() [integers]: Size: 34 bytes, 1.2248229980469 time to encode
igbinary_serialize() [integers]: Size: 58 bytes, 2.2877519130707 time to encode

Booleans

serialize() [booleans]: Size: 62 bytes, 2.0834550857544 time to encode
json_encode() [booleans]: Size: 39 bytes, 1.0889070034027 time to encode
igbinary_serialize() [booleans]: Size: 27 bytes, 1.8252439498901 time to encode

Floats

serialize() [floats]: Size: 709 bytes, 27.496570825577 time to encode
json_encode() [floats]: Size: 77 bytes, 5.0476500988007 time to encode
igbinary_serialize() [floats]: Size: 142 bytes, 2.4856028556824 time to encode

Mixed

serialize() [mixed]: Size: 178 bytes, 6.301619052887 time to encode
json_encode() [mixed]: Size: 54 bytes, 2.0463008880615 time to encode
igbinary_serialize() [mixed]: Size: 50 bytes, 2.3894169330597 time to encode

Objects

serialize() [objects]: Size: 326 bytes, 4.8698291778564 time to encode
json_encode() [objects]: Size: 148 bytes, 2.4744520187378 time to encode
igbinary_serialize() [objects]: Size: 177 bytes, 6.472992181778 time to encode

All data types

serialize() [all]: Size: 1567 bytes, 42.437592029572 time to encode
json_encode() [all]: Size: 462 bytes, 9.9569129943848 time to encode
igbinary_serialize() [all]: Size: 478 bytes, 18.053789138794 time to encode

Speed comparison

Analyzing the time it took for each test to be completed, we see again that json_encode is the clear winner (highlighted in bold the shortest time for the function).

Strings

serialize() [strings]: Size: 105 bytes, 1.8710339069366 time to encode
json_encode() [strings]: Size: 67 bytes, 1.5691390037537 time to encode
igbinary_serialize() [strings]: Size: 64 bytes, 3.2276048660278 time to encode

Integers

serialize() [integers]: Size: 121 bytes, 3.0198090076447 time to encode
json_encode() [integers]: Size: 34 bytes, 1.2248229980469 time to encode
igbinary_serialize() [integers]: Size: 58 bytes, 2.2877519130707 time to encode

Booleans

serialize() [booleans]: Size: 62 bytes, 2.0834550857544 time to encode
json_encode() [booleans]: Size: 39 bytes, 1.0889070034027 time to encode
igbinary_serialize() [booleans]: Size: 27 bytes, 1.8252439498901 time to encode

Floats

serialize() [floats]: Size: 709 bytes, 27.496570825577 time to encode
json_encode() [floats]: Size: 77 bytes, 5.0476500988007 time to encode
igbinary_serialize() [floats]: Size: 142 bytes, 2.4856028556824 time to encode

Mixed

serialize() [mixed]: Size: 178 bytes, 6.301619052887 time to encode
json_encode() [mixed]: Size: 54 bytes, 2.0463008880615 time to encode
igbinary_serialize() [mixed]: Size: 50 bytes, 2.3894169330597 time to encode

Objects

serialize() [objects]: Size: 326 bytes, 4.8698291778564 time to encode
json_encode() [objects]: Size: 148 bytes, 2.4744520187378 time to encode
igbinary_serialize() [objects]: Size: 177 bytes, 6.472992181778 time to encode

All data types

serialize() [all]: Size: 1567 bytes, 42.437592029572 time to encode
json_encode() [all]: Size: 462 bytes, 9.9569129943848 time to encode
igbinary_serialize() [all]: Size: 478 bytes, 18.053789138794 time to encode

Combination
Having the smallest result in size might not always be the best metric to base the choice of the serialization algorithm. For instance, looking at the results above in the Strings test, igbinary produces indeed the smallest result in size (64 bytes) but it takes twice as much to serialize the result in comparison to json_encode (3.22 vs. 1.56 seconds) and the size difference is a mere 3 bytes (64 vs. 67).

Similarly, for the Boolean test, igbinary produces 27 bytes and json_encode 39 bytes. It does however take igbinary nearly 80% more time to produce the result compared to json_encode.

For the Floats test the situation is reversed. json_encode produces a result that is around 50% smaller than the one of igbinary but it takes twice as much time to produce it.

As far as serializing data, in my personal opinion, json_encode is the clear winner.

Unserializing Results

Unserializing data is equally - and at times - more important than serializing. In many applications, developers sacrifice performance in writing but don't compromise when reading data.

In the tests below (shortest time highlighted in bold) once can easily see that igbinary is the clear winner. At times the unserialize function is very close (or outperforms igbinary) but overall, igbinary is the the function that unserializes data the fastest.

Speed comparison

Strings

unserialize() [strings]: 1.8259189128876 time to decode
json_decode() [strings]: 2.6482670307159 time to decode
igbinary_unserialize() [strings]: 1.8359968662262 time to decode

Integers

unserialize() [integers]: 2.3886890411377 time to decode
json_decode() [integers]: 2.8659090995789 time to decode
igbinary_unserialize() [integers]: 2.4441809654236 time to decode

Booleans

unserialize() [booleans]: 1.8097970485687 time to decode
json_decode() [booleans]: 2.4416139125824 time to decode
igbinary_unserialize() [booleans]: 1.7585029602051 time to decode

Floats

unserialize() [floats]: 18.512004137039 time to decode
json_decode() [floats]: 3.7896130084991 time to decode
igbinary_unserialize() [floats]: 2.6730649471283 time to decode

Mixed

unserialize() [mixed]: 4.6794769763947 time to decode
json_decode() [mixed]: 2.7775249481201 time to decode
igbinary_unserialize() [mixed]: 1.9598047733307 time to decode

Objects

unserialize() [objects]: 5.5468521118164 time to decode
json_decode() [objects]: 5.7660481929779 time to decode
igbinary_unserialize() [objects]: 5.2672090530396 time to decode

All data types

unserialize() [all]: 31.01339006424 time to decode
json_decode() [all]: 14.574991941452 time to decode
igbinary_unserialize() [all]: 10.734386920929 time to decode

Conclusion
If your application is mostly focused on reads rather than writes, igbinary is the clear winner, since it will unserialize your data faster than the other two functions. If however you are more focused on storing data, json_encode is the clear choice.

Git pre-commit - Another check to ensure clean code [How-To]

2011-11-13T13:45:00.001-05:00

Throughout my career I have been using various revision control systems. I started off with Visual SourceSafe which I thought at the time was great, primarily because of the small size of our team and the ease of use though the Visual Basic's IDE.

Later on I switched to SVN which is a great version control system and it fulfilled all my needs for proper version control.

When I moved to the US and started working here, I introduced SVN to the company that I was working at the time. We adopted the technique of having one branch per project, since we were working on different projects at any given time. Using the plugin through Eclipse or in Windows the integrated Explorer plugin, I was checking in code, switching through projects etc.

Recently I changed jobs and although we did use Subversion for a few months, we did switch to Git. The reasons behind it are too many to count - and I will go through them in another blog post.

One really good feature that Git has is its hooks. Although Subversion also supports hooks, I have only been exposed to the ones from Git and have used them - hence this blog post.

Git comes with some predefined hooks that one can use as a starting point. The code is checked before it is being committed and if it does pass whatever the pre-commit (for instance) hook does, it will allow you to commit; alternatively it will stop until you correct the mistakes made.

One very popular pre-commit hook is the one here by Remigijus Jarmalavičius. It checkes the files that have been modified/added and runs the php -l on it to ensure that whatever will be committed does not have PHP syntax errors.

I have downloaded that code and easily added it to my repository so everything is being checked prior to any of my commits.

In our MemberFuse™ platform, we have - like any other developer - many helper functions that are used solely for development. One of the mostly used one is the vdd($message); Effectively what this function does is a var_dump of the $message variable, echoes out the file that it was called as well as the line it is in, and dies. It is great for quick and dirty debugging and inspecting a variable as it passes through the code. Granted it is not very TDD but I am sure that all developers have little things like these to aid them with their daily programming and debugging.

From time to time, as we fix bugs and explore certain behaviors that sometimes it is difficult to assess, we use this function and we output the data on the browser. However this is just a development/debugging function and must never be used in the production environment.

I have been the culprit of using the function, forgot that I had it in a certain part of the code, and committed that code in our development branch. That caused some colleagues to experience inconsistent behavior with errors showing up on the screen and time was wasted.

To combat this scenario, I wrote a pre-commit hook that will allow you to check for the existence of certain functions in the code and ensure that those functions (or strings for that matter since I am using grep) do not exist in what is to be committed.

The pre-commit hook that I wrote is listed below and the code has been heavily based on Remigijus Jarmalavičius's pre-commit syntax checker.

#!/bin/bash
# Author: Nikolaos Dimopoulos 
# Based on code by Remigijus Jarmalavičius  
# Checks the files to be committed for the presence of print_r(), var_dump(), die()
# The array below can be extended for further checks

checks[1]="var_dump("
checks[2]="print_r("
checks[3]="die"

element_count=${#checks[@]}
let "element_count = $element_count + 1"

ROOT_DIR="$(pwd)/"
LIST=$(git status | grep -e '\#.*\(modified\|added\)')
ERRORS_BUFFER=""
for file in $LIST
do
    if [ "$file" == '#' ]; then
        continue
    fi
    if [ $(echo "$file" | grep 'modified') ]; then
        FILE_ACTION="modified"
    elif [ $(echo "$file" | grep 'added') ]; then
        FILE_ACTION="added"
    else 
        EXTENSION=$(echo "$file" | grep ".php$")
        if [ "$EXTENSION" != "" ]; then

            index=1
            while [ "$index" -lt "$element_count" ]
            do
                echo "Checking $FILE_ACTION file: $file [${checks[$index]}]" 
                ERRORS=$(grep "${checks[$index]}" $ROOT_DIR$file >&1)
                if [ "$ERRORS" != "" ]; then
                    if [ "$ERRORS_BUFFER" != "" ]; then
                        ERRORS_BUFFER="$ERRORS_BUFFER\n$ERRORS"
                    else
                        ERRORS_BUFFER="$ERRORS"
                    fi
                    echo "${checks[$index]} found in file: $file "
                fi
                let "index = $index + 1"
            done
        fi
    fi
done
if [ "$ERRORS_BUFFER" != "" ]; then
    echo 
    echo "These errors were found in try-to-commit files: "
    echo -e $ERRORS_BUFFER
    echo 
    echo "Can't commit, fix errors first."
    exit 1
else
    echo "Commited successfully."
fi

If you want to check for any kind of string, just add one extra line (or modify the existing ones) of the checks array. The code will loop through the array and 'grep' the modified/added files for that entry. If it is found the commit will not be allowed and you will have to manually go and change whatever needs to be changed.

I hope you find this hook useful. You can download the file from my github repository.

Note: If you wish to run more than one pre-commit hooks, you don't need to merge them all in the same file. You can create a pre-commit file which will have the following contents:

#!/bin/bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

$DIR/pre-commit.l
$DIR/pre-commit.output

The *.l file is the one that runs the PHP syntax check, while the *.output one is the one mentioned in this blog post. You can extend the list to your liking and usage.

New beginnings – sleep(6566400) [Personal][Update]

2011-11-11T11:11:00.000-05:00

It has been a very long time since I last blogged.

The primary reason is that I have been busy and somewhat lazy. Hopefully that trend (the laziness) will not last that much and I will be able to post in higher frequency.

The biggest update for me came this July. I was actively searching for a new employment opportunity for quite some time, but due to the current market situation, it has been a really difficult task.

In July I received (and accepted) an offer from Avectra Inc. I am now a PHP developer for MemberFuse™, a professional networking application that integrates with the client's association management software and builds an online community for that client.

My colleagues and I are programming in PHP using a frameworks such as Doctrine, Zend Framework, Smarty and jQuery on the client side. The application we produce is offered as SAAS (Software As A Service) and we have well over 1 million users.

Although adjusting to the new environment was rough and the learning curve was steep (it didn't help that one colleague decided to resign and I inherited his workload from week 2), I managed to 'survive' and I am more and more confident every day.

I can only say that working with smart people that push you to excel is a blessing!

The second update is this blog's location. With the introduction of the new dynamic templates in Blogger, I decided to move from Wordpress (and my personal installation for my blog) to Blogger.

The downside was that Blogger could not import my blog. I would always get errors when trying to upload the XML file and after a few tries I decided to go the manual way. I therefore sat down and copied and pasted all the content that I had posted in the past to the new platform.

Luckily it was not a very difficult task, but it was time consuming. Last night I managed to finish everything and write this blog post.

With this move unfortunately I lost all the comments on my posts and I am sure that there will be some broken links here and there but in the end it will all work out.

It was my intention to post this message today (11-11-11) at 11:11 but I kinda messed up the scheduling so the post is being posted a few hours later.

Using autounmask to Unmask Packages in Gentoo [How-To][Gentoo]

2011-05-18T14:49:00.000-04:00

Gentoo is one of my favorite Linux distributions. Although I am comfortable with other distributions, Gentoo has a special place in my heart and whenever I can use it I do :)

There are however some times that I would like to install a package - mostly to test something - and the package is masked. Masked packages are not "production ready" so they are not included in the portage tree i.e. available to be installed.

To allow a masked package to be installed, you will need to unmask that package by adding a corresponding entry in the etc/portage/package.keywords file.

The problem happens when the masked package (that you just unmasked) depends on other packages that are also masked. You will then need to rinse and repeat the process to ensure that everything is in place so that you can install that unmasked package.

NOTE: Playing with masked packages is like playing with fire. If you don't know what you are doing or you are not ready to potentially have an unusable system, don't follow the instructions below or unmask any packages.

app-portage/autounmask
The Gentoo developers have created a little utility that will unmask each package that needs to be unmasked. The utility is app-portage/autounmask.

I wanted to unmask www-misc/fcgiwrap so my manual method would be:

echo "=www-misc/fcgiwrap ~amd64" >> /etc/portage/packages.keywords

and would then emerge the package

emerge =www-misc/fcgiwrap

Instead I used autounmask:

emerge app-portage/autounmask

and

autounmask www-misc/fcgiwrap-1.0.3

autounmask version 0.27 (using PortageXS-0.02.09 and portage-2.1.9.42)
* Using repository: /usr/portage
* Using package.keywords file: /etc/portage/package.keywords * Using package.unmask file: /etc/portage/package.unmask * Using package.use file: /etc/portage/package.use
* Unmasking www-misc/fcgiwrap-1.0.3 and its dependencies.. this might take a while..
* Added '=www-misc/fcgiwrap-1.0.3 ~amd64' to /etc/portage/package.keywords * done!

Once that is done I can issue the emerge command and voila!

emerge www-misc/fcgiwrap

Although in my case there was only one dependency to unmask, when trying to unmask packages that have multiple dependencies such as gnome, kde etc., autounmask can be a very helpful utility.

LibreOffice Auto Update [How-To][Linux]

2011-04-21T14:45:00.000-04:00

We all knew it was coming. Ever since Oracle took a really surprising turn towards the non-community orientation of OpenOffice, a lot of users on the Internet were curious (and concerned) as to what will happen with the whole OpenOffice saga.

Thankfully the project was forked and LibreOffice was born. Within a few months the same (and better) quality product was released for the community. As the project entered its alpha stage, then the beta stage and then the RC stages, I was waiting anxiously to install it on my notebook (which is running Ubuntu 10.10 at the moment).

A couple of months ago, I was finally happy with the package and the release, so I downloaded it on my notebook and started the installation. I don't have a problem with getting my hands dirty and working with the command prompt, but I am lazy - so I want my packages to update easily (say though the package manager). At the time I could not use the package manager, so it was all terminal work :)

The installation was pretty easy using the dpkg command. I run literally two commands and the application was installed :)

There was a recent announcement about OpenOffice from Oracle. I posted a link at HackerNews for an article I read in MarketWire, regarding this "Oracle to move OpenOffice.org to a Community-Based Project". Unfortunately the link does not work any more but you might be able to read a bit about the subject here. In short:

OpenOffice is dead, long live LibreOffice.

Time for me to start getting serious about updating LibreOffice.

First of all I wanted to make sure that I did not have any cruft remaining from OpenOffice on my system.

sudo apt-get remove openoffice*.*

Surprisingly there were some packages that were still there.

The second step was to uninstall the current version of LibreOffice

sudo apt-get remove libreoffice*.*

And now comes the easy part. I am going to use a PPA, to ensure that I will get notified about pending updates and upgrade easily. To do so all I had to do was run the following commands on a terminal window:

sudo add-apt-repository ppa:libreoffice/ppa

Output:

Executing: gpg --ignore-time-conflict --no-options --no-default-keyring
--secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg
--keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg
--keyserver keyserver.ubuntu.com --recv 36E81C9267FD1383FCC4490983FBA1751378B444gpg:
requesting key 1378B444 from hkp server keyserver.ubuntu.com
gpg: key 1378B444: public key "Launchpad PPA for LibreOffice Packaging" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Then update the packages again:

sudo apt-get update

And now install LibreOffice:

sudo apt-get install libreoffice
sudo apt-get install libreoffice-gnome

Note: KUbuntu users will need to run

sudo apt-get install libreoffice-kde

For those that prefer the graphical interface, all you will have to do is add "ppa:libreoffice/ppa" to your software sources.

Enjoy your LibreOffice installation!

Building a HA Cluster on Linode [How-To][HA][Linux][Series]

2011-04-04T14:41:00.000-04:00

I have recently blogged about Slicehost vs. Linode and my decision to move my sites to the latter. Since then I can safely say that I made the right choice about the move. Linode's support is phenomenal. There has never been a ticket unanswered more than 5 minutes and all the tickets have been resolved. Even when I asked about a configuration issue regarding Heartbeat, which was clearly not in the realm of support, the support engineers did look at my configuration and did identify the error area. That alone saved me hours of troubleshooting and trying to find where the error was.

The Task
Your website must be available - all the time. Why? Because if you have something to say (a blog, a journal, a rant page, a forum etc.) you need your audience to be able to read your material all the time. Discussion forums, websites with custom applications or services, even information based websites need to have as close to 100% uptime as possible.. How can this be achieved? The solution is a High Availability (or HA) cluster.

Linux has been used for HA tasks for many years. The Linux-HA's wiki has a lot of valuable information as well as guides and resources that one can use to increase the redundancy of their websites. In addition to this, Google is your friend. There are numerous bloggers that have shared their experiences with the public on how to create HA resources. Finally, one can check Linode's Library - a set of guides to allow you to create HA clusters with your Linodes.

My task for the last few weeks has been to create a High Availability cluster of services to serve PHP and MySQL and also have the ability to grow infinitely (well close to that is). In the next few weeks I will post a series of blog posts outlining how to achieve a HA cluster for your sites.

Architecture
The cluster will be build using CentOS as the OS of choice. I have also experimented with Gentoo and Ubuntu. You can do everything I do here with Ubuntu if you wish. There are slight differences in certain commands and steps which the blog posts will not cover. As far as Gentoo is concerned, at the moment there is a block between Pacemaker and Heartbeat. Once that is resolved, I will try to redo the whole thing using Gentoo - as it is my OS of preference.

We will build two boxes to serve as load balancers. The boxes (or Linodes) will have Heartbeat, Pacemaker and nginx installed on them. A "floating" IP address will be used to move from one node to another in the case of a failure. Each Linode uses nginx as a proxy to forward all requests to a set of web servers using nginx's proxy functionality.

We will also build a web server. This again is a CentOS box running PHP and nginx. The web server will store the data locally and connect to the database cluster. Once the last part of this How-To is completed (creation of a HA NFS) then we will be able to add as many web servers as we need.

The next set of boxes form the database cluster. The setup is two servers with Heartbeat, Pacemaker and MySQL setup with a Master/Master replication. Again a "floating" IP address is used to move from one server to another in the case of failure.

The NFS is also a 2 box setup. Again a "floating" IP address is used to connect to the file system. DRBD is installed on them to cater for the replication.

Administration
All Linodes are located in the same data center. At the moment there is no way to create Linodes in different data centers and implement the above mentioned setup in an effort to achieve geographical redundancy. The whole setup is using 9 Linodes the last one being used for administrative tasks (note in my count I used 2 boxes as web servers).

All Linodes have active iptables configurations. All nodes have been configured to work with 322 as the SSH port to avoid the novice hacker. Every port is blocked from communicating with the Internet apart from the ones needed for essential services. For instance the Load Balancers allow connections on ports 80 and 443 (http and https). However, they only communicate with each other on the Heartbeat port. Equally the web servers do not allow connections to their 80/443 ports to any machine other than the Load Balancers. The Database Servers allow connections only from the Web servers etc.

The administrative node resides on a different data center. It runs Nagios and it monitors all the nodes of our HA cluster. Naturally the iptables setup of each node is adjusted to allow connections from this particular node.

Conclusion
I hope that these blog posts will serve as a learning exercise/guide to those who want to delve in High Availability websites. The primary reason I built this cluster is to offer these services to customers that have busy sites that require maximum availability. I am currently setting up the final touches of the hosting service I am going to offer so stay tuned for the details.

Disclaimer: The Linode link in this blog post is an affiliate one. If you wish to sign up for their services you are more than welcome to click it. The non affiliate link is here.

Slicehost vs. Linode [Hosting][VPS]

2011-03-21T14:10:00.000-04:00

Through the years I have hosted my sites on various hosting companies. I had the really good experiences like Vertexhost and really terrible ones - I don't remember the name of the host, but that kid, as it turned out to be later on, managed to lose 1.6GB of my data. You can safely say that I have been burned by bad hosting companies but also have enjoyed the services of good ones. In the case of Vertexhost, I part owned that company a few years back and I know that the current owner is a straight up guy and really cares for his customers.

Since I moved my emails to Google Apps I only need the hosting for my personal sites such as my blog, my wife's sites (burntoutmom.com, greekmommy.net) and a few other small sites.

I used to host those sites on one of my company's clusters. The bandwidth consumed was nothing to write home about (I think in total it was a couple of GB per month ~ 1.00 USD) so it didn't matter that I had them there. However, recent events forced me to move them out of that cluster. I was on the market for good and relatively cheap hosting. I did not want to purchase my own server or colocate with someone else. My solution was going to be a VPS since I would be in control of what I install and what I need.

Slicehost

Without much thought, I signed up for Slicehost, which is a subsidiary of Rackspace, a very well known and reputable company.

I got their 4GB package (250.00 USD per month) and installed Gentoo on it. Apart from the price which was a bit steep, everything else was fine. I was happy to be able to host my sites in a configuration that I was comfortable with, under the understanding that if the VPS failed, then all my sites would go down. That however is the risk that everyone takes while hosting their sites on a single machine. The higher the availability and redundancy the higher the cost.

I must admit that signing up was not a very happy experience. I went and paid with my credit card, as they pro-rate your month based on your package. Almost immediately after signing up, came the email informing me that my credit card has been charged for the relevant amount. I got into the box through ssh, updated the /etc/make.conf file with the USE flags that I needed, run emerge --sync and then emerge --update --deep --newuse --verbose world so as to update the system.

It must have been around 5-10 minutes into the process that I received an email from Slicehost saying that they are checking my account information and that I need to confirm my credit card details. I immediately replied to their email (gotta love the desktop notifications on GMail), with the information they needed.

After I sent the email, I noticed that the box was not responding. I tried to log back in and could not. I was also logged out (and could not log back in) to their management console on slicehost.com site. I was fuming! Effectively they severed the connection to the VPS in the middle of compilation to check my credit card information.I understand that they need to perform checks for fraud but two questions came to mind:

Why did they have to sever the connection and not just send an email, and if I did not reply, just block access to the box? That would have been a heck of a lot of an inconvenience to myself i.e. the end user.
Why did the initial email say that my credit card has been charged and it had not?

No more than 10 minutes later the whole thing had been resolved. I received an email saying that "everything is OK and your account has been restored", at which point I logged back in to redo the compilations. I also received emails from their support/billing team apologizing but stating that although the initial email states that they charge the credit card, they don't. It is something they need to correct because it pisses people (like me) off.

There was nothing wrong with my setup - everything was working perfectly but the price was really what was bothering me. I would be able to support the sites for a few months, but since literally none of them is making money (maybe a dollar here or there from my blog but that is about it), I would have to pay out of pocket for the hosting every month. I had to find a different solution that would be:

cheaper than Slicehost
flexible in terms of setup
easy to use in terms of controlling your VPS

After a lot of research I ended up with two winners: Linode and Prgmr. I opted for Linode, because although it was quite a bit more expensive than Prgmr, it had the better console in handling your VPS. I will, however, try out Prgmr's services in the near future so as to assess how good they are. They definitely cannot be beat in price.

Linode

Setting up an account with Linode was very easy. I didn't have any of the mini-saga I had with Slicehost. The account was created right there and then, my credit card charged and I was up and running in no time. Immediately I could see a difference in price. Linode's package for 4GB or RAM is 90.00 USD cheaper (159.00 USD vs. 250.00 USD for Slicehost). For the same package, the price difference is huge.

I started testing the network, creating my VPS in the Atlanta, GA datacenter (Linode offers a number of datacenters for you to create your own). The functionality that was available to me was identical and in some cases superior to that of Slicehost. There are a lot more distributions to choose from, and you can partition your VPS the way you want it to name a couple.

Shifting through the documentation, I saw a few topics regarding high availability websites. The articles described using DRBD, nginx, heartbeat and pacemaker etc. to keep your sites highly available. I was intrigued by the information and set off to create a load balancer using two VPSs and nginx. I have documented the process and this is another blog post that will come later on this week.

While experimenting with the load balancer (and it was Saturday evening) I had to add a new IP address to one of the VPS instances. At the time my account would not allow such a change and I had to contact support. I did and got a reply in less than 5 minutes. I was really impressed by this. Subsequent tickets were answered within the 5 minute time frame. Kudos to Linode support for their speed and professionalism.

Conclusion

For a lot cheaper, Linode offered the same thing that Slicehost did. Moving my sites from one VPS to another was a matter of changing my DNS records to point to the new IP address.

I have been using Linode for a week and so far so good. The support is superb and the library is full of how-to's that allows me to experiment with anything I want to - and the prices are not going to break me.

Resources

Vertexhost, Slicehost, Rackspace, Gentoo, Linode, Linode Library

Additional storage for Google Apps users [Information][Review]

2011-03-10T14:00:00.000-05:00

I have been using Google Apps for a number of years now and I have gotten so used to it that I cannot fathom any other way of operating. I am sure that some of you share that sentiment. :)

Limitations
Up until a few months ago, Google Apps had its limitations. The actual Apps was in some sort of a jailshell, isolated from the whole Google suite of applications. For that reason you could not use your Google Apps login to enjoy the service of Google Reader for instance. You had to be sneaky about it. You had to create a Google Account with the same username (and password if you liked) as your Google Apps domain and although the two did not communicate, you could have effectively "one login" for all services.

This limitation became more apparent with the increased usage of Android phones (where you need to have a Google Account on your phone) as well as Google Voice. Users have been asking about the "merge" and Google responded with significant infrastructure changes to cater for the transition. In my blog post "Google Apps and Google Accounts Merge" I present additional information about this, inclusive of a How-To on the transition for administrators of Google Apps. Unfortunately the process is not perfect and there are still some services that are not fully integrated with the new infrastructure (but will be in the future). For instance in my domain, since I use Google Voice with my domain email account, I am still on the "old" system because the account could not be transitioned. It will happen in the end, it just takes time.

Storage Needs
The biggest issue for me that was related to these two separated accounts (Google Apps Account vs. Google Account) was Picasa and Google Docs.

I have been very methodical in my filing, utilizing electronic storage as much as possible. For that reason I have been scanning documents and uploading them to Google Docs (or if they were available in PDF format I would just upload them). The documents would range from personal, utility bills, bank statements, anything that I want to store. Soon I realized that the 1GB that Google Apps offers for documents will not cut it. I therefore created a new account which I named DocsMule1 (clearly to signify its purpose). I created one folder in that account, uploaded as many documents as I could there and shared that document with my own account as well as my wife's. Soon I found more limitations since I ended up with 3 mule accounts. Since there was no option for me to upgrade the storage (even if I paid for it), I had to change my strategy. Managing documents from 3 or more different accounts is not an easy and convenient task.

I downloaded all my documents back to my computer (gotta love Google's Data Liberation) and deleted them from the Google Apps mule accounts and then deleted those accounts - just to keep everything tidy. I then launched my Gmail account and signed into Google Docs. I created one folder which I shared with my Google Apps accounts (my wife's and mine) and then paid $50.00 for a whole year - which provided me with 200GB of space. You can always check how much space you are using by visiting the Manage Storage page of your Google Account.

Once that was done, I started creating my folders (collections now) and uploaded all my documents up there. In addition to that, since my parents live in Greece, they rely on VoIP chat as well as my Picasa to stay in touch with their grandchildren. My wife and I, through the use of our mini camera as well as our Android phones, take a lot of pictures of the kids, documenting the little things that they do on a regular basis. This serves as a good archive for them when they grow up but also as a good way to stay in touch with my parents. Google's additional storage was the solution.

Problem solved. With minimal money I had everything sorted out. It did however inconvenience me quite a bit in the end, since a lot of my data was scattered now. The GMail account would keep Picasa and Docs, the Google Apps account my email, my Google account my Reader, Web history etc. Not very convenient but it works.

Storage for Google Apps
Around February, Google announced that they will be offering the option to Google Apps users to purchase additional storage. I was really happy about that since I could therefore ditch the GMail account for handling my docs and keep everything under the domain account. However something was wrong. When Google revealed their pricing (the announcement is not there any more but the prices I am quoting are real), I quickly found out that for the storage I currently have, I would need to spend $700.00 a year instead of $50.00. That did not make sense at all. Needless to say, I stayed with my existing plan.

Initial Pricing Plan

Storage Price (per year)

   5 GB         17.50 USD
  20 GB         70.00 USD
  80 GB        280.00 USD
 200 GB        700.00 USD
 400 GB      1,400.00 USD
   1 TB      3,500.00 USD
   2 TB      7,000.00 USD
   4 TB     14,000.00 USD
   8 TB     28,000.00 USD
  16 TB     56,000.00 USD

New Pricing Plan
A few days ago, Google announced changes in the pricing of additional storage for Google Apps users as well as changes to the free storage that Google offers for Picasa. Picasa Web Albums does offer 1GB of free storage but now photos of 800x800 pixels or less as well as videos of 15 minutes or less do not count against the 1GB of storage. You can read more about the Picasa Web Albums storage in the relevant help page.

 Storage   Price (per year)
  20 GB         5.00 USD
  80 GB        20.00 USD
 200 GB        50.00 USD
 400 GB       100.00 USD
   1 TB       256.00 USD
   2 TB       512.00 USD
   4 TB     1,024.00 USD
   8 TB     2,048.00 USD
  16 TB     4,096.00 USD

As far as the new pricing is concerned, Google brought everything in line with Google Accounts pricing (effectively scrapping the initial - expensive - pricing plan for extra storage). The help page "Google Storage - How it Works" offers additional information for those that want to use/upgrade their storage while using a Google Apps account. Effectively it now costs exactly the same to purchase additional space for your Google Apps account (to store documents) as it would if you were using a different Google Account. That probably means that I have to download everything to my computer and re-upload it to my Google Apps account....

To take advantage of this feature, you will have to go to the Purchase additional storage page while logged in with the Google Apps account that you wish to purchase storage for. Note that there is a warning that appears in red (see image) that warns you that you are using a Google Apps account. Google provides this information since your Google Apps account relies on the Google Apps administrator. If you have an account on Google Apps and you purchase storage, that storage will be gone if the administrator deletes or restricts access to your account.

Conclusion
In my view, Google has done it again. They now offer an extremely affordable and secure way of storing your data. There are loads of people that have concerns about where their data is stored, who has ownership of the data stored, what does Google do with the data etc. A lot of these questions can easily be answered if you google (duh) the relevant terms or search in Google's Help Center. Data Liberation allows you to retrieve your data whenever you want to. If on the other hand you are skeptical and do not wish to store your data there, don't. It is your choice.

References
Google Docs, Picasa Web Albums, Data Liberation, Google Account - Manage Storage, Photo and Video Requirements: Size Limits for Picasa Web Albums, Google Storage - How it Works, Google Account - Purchase additional storage, Google's Help Center

Keeping your Linux users in sync in a cluster [How-To][Linux][Sync]

2011-02-20T13:56:00.000-05:00

As websites become more and more popular, your application might not be able to cope with the demand that your users put on your server. To accommodate that, you will need to move out of the one server solution and create a cluster. The easiest cluster to create would be with two servers, one to handle your web requests (HTTP/PHP etc.) and one to handle your database load (MySQL). Again this setup can only get you so far. If your site is growing, you will need a cluster of servers.

Database
The scope of this How-To will not cover database replication; I will need to dedicate a separate blog post for that. However, clustering your database is relatively easy with MySQL replication. You can set a virtual hostname (something like mysql.mydomain.com) which is visible only within your network. You then set up the configuration of your application to use that as the host. The virtual hostname will map to the current master server, while the slave(s) will only replicate.

For instance, if you have two servers A and B, you can configure both of them to become master or slave in MySQL. You then set one of them as master (A) and the other as slave (B). If something happens to A, B gets promoted to master instantly. Once A comes back up, it gets demoted to a slave and the process is repeated if/when B has a problem. This can be a very good solution but you will need to have pretty evenly matched servers to keep with the demand. Alternatively B can be less powered than A and when A comes back up you keep it momentarily as a slave (until everything is replicated) and then promote it back to master.

One thing to note about replication (that I found through trial and error). MySQL keeps binary logs to handle replication. If you are not cautious in your deployment, MySQL will never recycle those logs and therefore you will soon run out of space when having a busy site. By default those logs will be under /var/lib/mysql.

By changing directives in my.cnf you can store the binary logs in a different folder and even set up 'garbage collection' or recycling. You can for instance set the logs to rotate every X days with the following directive in my.cnf:

expire_logs_days  = 5

I set mine to 5 days which is extremely generous. If your replication is broken you must have the means to know about it within minutes (see nagios for a good monitoring service). In most cases 2 days is more than enough.

Files
There are numerous ways of keeping your cluster in sync. A really good tool that I have used when playing around with a cluster is csync2. Installation is really easy and and all you will need is to run a cron task every X minutes (up to you) to synchronize the new files. Imagine it as a two way rsync. Another tool that can do this is unison but I found it to be slow and difficult to implement - that's just me though.

Assume an implementation of a website being served by two (or more) servers behind a load balancer. If your users upload files, you don't know where those files are uploaded, which server that is. As a result if user A uploads the file abc.txt to server A, user B might be served the content from server B and would not be able to access the file. csync2 would synchronize the file across the number of servers, thus providing access to the content and keeping multiple copies of the content (additional backup if you like).

NFS
An alternative to keeping everything synchronized is to use a NFS. This approach has many advantages and some disadvantages. It is up to you on whether the disadvantages are something you can live with.

Disadvantages

NFS is slow - slower than the direct access to a local hard drive.
Most likely you will use a symlink to the NFS folder, which can slow things down even more.

Advantages

The NFS does not rely on the individual web servers for content.
The web servers can be low to medium spec boxes without the need to have really fast and large hard drives
A well designed NFS with DRDB provides a raid-1 over a network. Using gigabit Network Interface Cards you can keep performance at really high levels.

I know that my friend Floren does not agree with my approach on the NFS and would definitely have gone with the csync2 approach. Your implementation depends on your needs.

Users and Groups
Using the NFS approach, we need to keep the files and permissions properly set up for our application. Assume that we have two servers and we need to create one user to access our application and upload files.

The user has been created on both servers and the files are stored on the NFS. Connecting to server A and looking at the files we can see something like this:

drwxr-xr-x 17 niden  niden  4096 Feb 18 13:41 www.niden.net
drwxr-xr-x  5 niden  niden  4096 Nov 15 22:10 www.niden.net.files
drwxr-xr-x  7 beauty beauty 4096 Nov 21 17:42 www.beautyandthegeek.it

However when connecting to server B, the same listing tells another story:

drwxr-xr-x 17 508    510    4096 Feb 18 13:41 www.niden.net
drwxr-xr-x  5 508    510    4096 Nov 15 22:10 www.niden.net.files
drwxr-xr-x  7 510    511    4096 Nov 21 17:42 www.beautyandthegeek.it

The problem here is the uid and gid of the users and groups of each user respectively. Somehow (and this is really easy to happen) server A had one or more users added to it, thus the internal counter of the user IDs has been increased by one or more and is not identical to that one of server B. So adding a new user in server A will get the uid 510 while on server B the same process will produce a user with a uid of 508.

To have all users setup on all servers the same way, we need to use two commands: groupadd and useradd (in some Linux distributions you might find them as addgroup and adduser).

groupadd
First of all you will need to add groups. You can of course keep all users in one group but my implementation was to keep one user and one group per access. To cater for that I had to first create a group for every user and then the user account itself. Like users, groups have unique ids (gid). The purpose of gid is:

The numerical value of the groups ID. This value must be unique, unless the -o option is used. The value must be non-negative. The default is to use the smallest ID value greater than 999 and greater than every other group. Values between 0 and 999 are typically reserved for system accounts.

I chose to assign each group a unique id (you can override this behavior by using the -o switch in the command below, thus allowing a gid to be used in more than one group). The arbitrary number that I chose was 2000.

As an example, I will set niden as the user/group for accessing this site and beauty as the user/group that accesses BeautyAndTheGeek.IT. Note that this is only an example.

groupadd --gid 2000 niden
groupadd --gid 2001 beauty

Repeat the process as many times as needed for your setup. Connect to the second server and repeat this process. Of course if you have more than two servers, repeat the process on each of the servers that you have (and each accesses your NFS)

useradd
The next step is to add the users. Like groups, we will need to set the uid up. The purpose of the uid is:

The numerical value of the users ID. This value must be unique, unless the -o option is used. The value must be non-negative. The default is to use the smallest ID value greater than 999 and greater than every other user. Values between 0 and 999 are typically reserved for system accounts.

Like with the groups, I chose to assign each user a unique id starting from 2000.

So to in the example above, the commands that I used were:

useradd --uid 2000 -g niden --create-home niden
useradd --uid 2000 -g beauty --create-home beauty

You can also use a different syntax, utilizing the numeric gids:

useradd --uid 2000 --gid 2000 --create-home niden
useradd --uid 2000 --gid 2001 --create-home beauty

Again, repeat the process as many times as needed for your setup and to as many servers as needed.

In the example above I issued the --create-home switch (or -m) so as a home folder to be created under /home for each user. Your setup might not need this step. Check the references at the bottom of this blog post for the manual pages for groupadd and useradd.

I would suggest that you keep a log of which user/group has which uid/gid. It helps in the long run, plus it is a good habit to keep proper documentation on projects :)

Passwords?
So how about the passwords on all servers? My approach is crude but effective. I connected to the first server, and set the password for each user, writing down what the password was:

passwd niden

Once I had all the passwords set, I opened the /etc/shadow file.

nano /etc/shadow

and that revealed a long list of users and their scrambled passwords:

niden:$$$$long_string_of_characters_goes_here$$$$:13864:0:99999:7:::
beauty:$$$$again_long_string_of_characters_goes_here$$$$:15009:0:99999:7:::

Since I know that I added niden and beauty as users, I copied these two lines. I then connected to the second server, opened /etc/shadow and located the two lines where the niden and beauty users are referenced. I deleted the existing lines, and pasted the ones that I had copied from server A. Saved the file and now my passwords are synchronized in both servers.

Conclusion
The above might not be the best way of keeping users in sync in a cluster but it gives you an idea on where to start. There are different implementations available (Google is your friend) and your mileage might vary. The above has worked for me for a number of years since I never needed to add more than a handful of users on the servers each year.

References
MySQL System Variables, Nagios, csync2, rsync, Unison File Synchronizer, DRDB, Axivo Inc., groupadd man page, useradd man page

A look in the past and the future [Update]

2011-02-06T13:47:00.000-05:00

It has been months since I last posted a blog post. A lot of things have happened since August and I have a lot of material to post - just not the discipline to sit down and proof all the drafts so that I can post them.

2010
2010 has been a rough year mostly on a personal level. I lost my brother in law in March, my daughter was born in May, there was great uncertainty at work during the summer, an upgrade went bad for Long Hair Care Forum to name a few of the highlights.

Since this is officially my first year of blogging, I was happy to see some of the statistics for that year (well 8 months to be exact since I haven't posted since August).

Visits	8,940
Pageviews	11,979
Pages/Visit	1.41
Bounce Rate	81.34%
Avg Time on Site	00:01:10
% New Visits	85.04%

The most visited posts were:

Google Apps and Google Accounts merge [Information][How-To] with 1,601 pageviews (13.37%) Subversion Backup [How-To] with 1,289 pageviews (10.76%) Android 2.2 (Froyo) [Review] with 1,124 pageviews (9.38%)

Not that bad for a first year.

2011
When I started blogging my main focus was to mostly address coding issues. However due to the plethora of solutions on the Internet, the content that I wanted to relay was not that interesting or had been covered a hundred times or more. For instance I did start posting about design patterns here and here, but at the same time Giorgio Sironi started blogging about design patterns and did an excellent job at it, so that topic was scrapped. My posts then started covering a much broader scope, mostly that of technology with a small personal flare.

For 2011 I will continue on the same scope. I intend on blogging about interesting things technologically, how-to's and topics that I want to share with the community.

As always, whatever is presented in this blog is my personal opinion. Every post covered here as well with any code are free of copyright and you are free to use them in your projects at your own risk.

A big thank you to everyone that visited this blog whether you shared your opinion with me or not. I hope to meet the expectations of producing interesting content in the future.

Create an inexpensive hourly remote backup [How-To]

2010-08-21T13:38:00.000-04:00

There are two kinds of people, those who backup regularly, and those that never had a hard drive fail

As you can tell the above is my favorite quote. It is so true and I believe everyone should evaluate how much their data (emails, documents, files) is worth to them and, based on that value, create a backup strategy that suits them. I know for sure that if I ever lost the pictures and videos of my family I would be devastated since those are irreplaceable.

So the question is how can I have an inexpensive backup solution? All my documents and emails are stored in Google, since my domain is on Google Apps. What happens to the live/development servers though that host all my work? I program on a daily basis and the code has to be backed up regularly so as to avoid any hard drive failures and thus result in loss of time and money.

So here is my solution. I have an old computer (IBM Thincentre) which I decided to beef up a bit. I bought 4Gb of RAM from eBay for less than $100 for it. Although this is was not necessary since my solution would be based on Linux (Gentoo in particular), I wanted to have faster compilation times for packages.

I bought two external drives (750Gb and 500Gb respectively) and one 750Gb internal drive. I already have a 120Gb hard drive in the computer. The two external ones are connected to the computer using USB while the internal ones are connected using SATA.

The external drives are formatted using NTFS while the whole computer is built using ReiserFS.

Here is the approach:

I have installed and have a working Gentoo installation on the machine
I have an active Internet connection
I have installed LVM on the machine and set up the core system on the 120Mb drive while the 500Mb is on LVM
I have 300Mb active on the LVM (from the available 500Mb)
I have generated a public SSH key (I will need this to exchange it with the target servers)
I have mounted the internal 500Mb drive to the /storage folder
I have mounted the external USB 750Mb drive to the /backup_hourly folder
I have mounted the external USB 500Mb drive to the /backup_daily folder

Here is how my backup works:

Every hour a script runs. The script uses rsync to syncrhonize files and folders from a remote server locally. Those files and folders are kept in relevant server named subfolders in the /storage folder (remember this is my LVM). So for instance my subfolders will be /storage/beryllium.niden.net, /storage/nitrogen.niden.net, /storage/argon.niden.net etc.

Once the rsync completes, the script continues by compressing the relevant 'server' folder and creates the compressed file with a date-time stamp on its name.

When all compressions are completed, if the time that the script has executed is midnight, the backups are moved from the /storage folder to the /backup_daily folder (which has the external USB 500Gb mounted). If it is any other time, the files are moved in the /backup_hourly folder (which has the external USB 750Gb mounted).

This way I ensure that I keep a lot of backups (daily and hourly ones). The backups are being recycled, so older ones get deleted. The amount of data that you need to archive as well as the storage space you have available dictate how far back you can go in your hourly and daily cycles.

So let's get down to business. The script itself:

#!/bin/bash
DATE=`date +%Y-%m-%d-%H-%M`
DATE2=`date +%Y-%m-%d`
DATEBACK_HOUR=`date --date='6 days ago' +%Y-%m-%d`
DATEBACK_DAY=`date --date='60 days ago' +%Y-%m-%d`
FLAGS="--archive --verbose --numeric-ids --delete --rsh='ssh'"
BACKUP_DRIVE="/storage"
DAY_USB_DRIVE="/backup_daily"
HOUR_USB_DRIVE="/backup_hourly"

These are some variables that I need for the script to work. The DATE and DATE2 are used to date/time stamp the backups, while the DATEBACK_* are used to clear previous backups. In this case it is set to 6 days ago (for my system). It can be set to whatever you want provided that you do not run out of space.

The FLAGS variable keeps the rsync command options while the BACKUP_DRIVE, DAY_USB_DRIVE and HOUR_USB_DRIVE hold the locations of the rsync folders, daily backup and hourly backup sorage areas.

The script works with arrays. I have 4 arrays to do the work and the 3 of them must have exactly the same elements.

# RSync Information
rsync_info[1]="beryllium.niden.net html rsync"
rsync_info[2]="beryllium.niden.net db rsync"
rsync_info[3]="nitrogen.niden.net html rsync"
rsync_info[4]="nitrogen.niden.net html db"
rsync_info[5]="nitrogen.niden.net html svn"
rsync_info[6]="argon.niden.net html rsync"

This is the first array which holds descriptions to what needs to be done as far as source is concerned. These descriptions get appended to the log and helps me identify what step I am in.

# RSync Source Folders
rsync_source[1]="beryllium.niden.net:/var/www/localhost/htdocs/"
rsync_source[2]="beryllium.niden.net:/niden_backup/db/"
rsync_source[3]="nitrogen.niden.net:/var/www/localhost/htdocs/"
rsync_source[4]="nitrogen.niden.net:/niden_backup/db"
rsync_source[5]="nitrogen.niden.net:/niden_backup/svn"
rsync_source[6]="argon.niden.net:/var/www/localhost/htdocs/"

This array holds the source host and folder. Remember that I have already exchanged SSH keys with each server, therefore when the script runs there is a direct connection to the source server. If you need to keep things a bit more secure for you, then you will need to alter the contents of the rsync_source array so that it reflects the user that you log in with as well as the password.

# RSync Target Folders
rsync_target[1]="beryllium.niden.net/html/"
rsync_target[2]="beryllium.niden.net/db/"
rsync_target[3]="nitrogen.niden.net/html/"
rsync_target[4]="nitrogen.niden.net/db/"
rsync_target[5]="nitrogen.niden.net/svn/"
rsync_target[6]="argon.niden.net/html/"

This array holds the target locations for the rsync. These folders exist in my case under the /storage subfolder.

# GZip target files
servers[1]="beryllium.niden.net"
servers[2]="nitrogen.niden.net"
servers[3]="argon.niden.net"

This array holds the names of the folders to be archived. These are the folders directly under the /storage folder and I am also using this array for the prefix of the compressed files. The suffix of the compressed files is a date/time stamp.

Here is how the script evolves:

echo "BACKUP START"  >> $BACKUP_DRIVE/logs/$DATE.log
date >> $BACKUP_DRIVE/logs/$DATE.log

echo "BACKUP START"  >> $BACKUP_DRIVE/logs/$DATE.log
date  >> $BACKUP_DRIVE/logs/$DATE.log

# Loop through the RSync process
element_count=${#rsync_info[@]}
let "element_count = $element_count + 1"
index=1
while [ "$index" -lt "$element_count" ]
do
    echo ${rsync_info[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    rsync $FLAGS ${rsync_source[$index]} $BACKUP_DRIVE/${rsync_target[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    let "index = $index + 1"
done

The snippet above loops through the rsync_info array and prints out the information in the log file. Right after that it uses the rsync_source and rsync_target arrays (as well as the FLAGS variable) to rsync the contents of the source server with the local folder. Remember that all three arrays have to be identical in size (rsync_info, rsync_source, rsync_target).

The next thing to do is zip the data (I loop through the servers array)

# Looping to GZip data
element_count=${#servers[@]}
let "element_count = $element_count + 1"
index=1
while [ "$index" -lt "$element_count" ]
do
    echo "GZip ${servers[$index]}" >> $BACKUP_DRIVE/logs/$DATE.log
    tar cvfz $BACKUP_DRIVE/${servers[$index]}-$DATE.tgz $BACKUP_DRIVE/${servers[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    let "index = $index + 1"
done

The compression method I use is tar/gzip. I found it to be fast with a good compression ratio. You can choose anything you like.

Now I need to delete old files from the drives and copy the files on those drives. I use the servers array again.

# Looping to copy the produced files (if applicable) to the daily drive
element_count=${#servers[@]}
let "element_count = $element_count + 1"
index=1

while [ "$index" -lt "$element_count" ]
do
    # Copy the midnight files
    echo "Removing old daily midnight files" >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $DAY_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK_DAY*.* >> $BACKUP_DRIVE/logs/$DATE.log
    echo "Copying daily midnight files" >> $BACKUP_DRIVE/logs/$DATE.log
    cp -v $BACKUP_DRIVE/${servers[$index]}-$DATE2-00-*.tgz $DAY_USB_DRIVE/${servers[$index]}  >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $BACKUP_DRIVE/${servers[$index]}-$DATE2-00-*.tgz >> $BACKUP_DRIVE/logs/$DATE.log

    # Now copy the files in the hourly
    echo "Removing old hourly files" >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $HOUR_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK_HOUR*.* >> $BACKUP_DRIVE/logs/$DATE.log
    echo "Copying daily midnight files" >> $BACKUP_DRIVE/logs/$DATE.log
    cp -v $BACKUP_DRIVE/${servers[$index]}-$DATE.tgz $HOUR_USB_DRIVE/${servers[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $HOUR_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK*.* >> $BACKUP_DRIVE/logs/$DATE.log
    let "index = $index + 1"
done

echo "BACKUP END"  >> $BACKUP_DRIVE/logs/$DATE.log

The last part of the script loops through the servers array and:

Deletes the old files (recycling of space) from the daily backup drive (/storage/backup_daily) according to the DATEBACK_DAY variable. If the files are not found a warning will appear in the log.
Copies the daily midnight file to the daily drive (if the file does not exist it will simply echo a warning in the log - I do not worry about warnings of this kind in the log file and was too lazy to use an IF EXISTS condition)
Removes the daily midnight file from the /storage drive.

The reason I am using copy and then remove instead of the move (mv) command is that I have found this method to be faster. Finally the same thing happens with the hourly files

Old files are removed (DATEBACK_HOUR variable)
Hourly file gets copied to the /backup_hourly drive
Hourly file gets deleted from the /storage drive

All I need now is to add the script in my crontab and let it run every hour. NOTE: The first time you will run the script you will need to do it manually (not in a cron job). The reason behind it is that the first time rsync will need to download all the contents of the source servers/folders in the /storage drive so as to create an exact mirror. Once that lengthy step is done, the script can be added in the crontab. Subsequent runs of the script will download only the changed/deleted files.

This method can be very effective while not using a ton of bandwidth every hour. I have used this method for the best part of a year now and it has saved me a couple of times.

The last thing I need to present you is the backup script that I have for my databases. As you notice above the source folder of beryllium.niden.net as far as databases are concerned is beryllium.niden.net/db/. What I do is I dump and zip the databases every hour on my servers. Although this is not a very efficient way of doing things and it adds to the bandwidth consumption every hour (since the dump will create a new file every hour) I have the following script running on my database servers every hour at the 45th minute:

#!/bin/bash

DBUSER=mydbuser
DBPASS='dbpassword'
DBHOST=localhost
BACKUPFOLDER="/niden_backup"
DBNAMES="`mysql --user=$DBUSER --password=$DBPASS --host=$DBHOST --batch --skip-column-names -e "show databases"| sed 's/ /%/g'`"
OPTIONS="--quote-names --opt --compress "

# Clear the backu folder
rm -fR $BACKUPFOLDER/db/*.*

for i in $DBNAMES; do
    echo Dumping Database: $i
    mysqldump --user=$DBUSER --password=$DBPASS --host=$DBHOST $OPTIONS $i > $BACKUPFOLDER/db/$i.sql
    tar cvfz $BACKUPFOLDER/db/$i.tqz $BACKUPFOLDER/db/$i.sql
    rm -f $BACKUPFOLDER/db/$i.sql
done

That's it.

The backup script can be found in my GitHub here.

Update: The metric units for the drives were GB not MB. Thanks to Jani Hartikainen for pointing it out.

Google Apps and Google Accounts merge [Information][How-To]

2010-08-18T13:17:00.000-04:00

Anyone that has a Google Apps account and wants to access other services like Google Reader, Google Voice etc. knows that the username and password of the Google Apps account does not work for these services, since those are not available for Google Apps accounts. To get around this limitation, what you could do (and what I have done in the past) is to sign up for a new Google Account with the same email address as your Google Apps account. This of course creates confusion at times, duplication of data and disassociation of services. The most obvious example of this is on an Android device. To use my phone, I need to sign in with my Google Apps account. However, to use my Google Voice number, I have to use sign in again for that service but now using my Google Account (which uses the same email address). This works but I still have to keep two sets of contacts - one for the Google Apps and one in Google Voice.

According to Google, 9 of the top 20 requests from Google Apps customers are for their accounts to work with more services from Google. To facilitate this, the Google Apps account had to be merged with the Google Account. This page in Google Apps Help that describes the transition.

Unfortunately this was not a simple flip of a switch for Google. Significant infrastructure changes had to be in place prior to the merging of the accounts. We were promised that this change would be in place by fall and we have not been disappointed. Google is rolling out the update slowly but if you want to 'speed up' the process, you can sign up for an early round of testing of the new infrastructure. I have signed up for several of my Google Apps domains a couple of weeks ago.

I am happy to announce that one of my domains has already gone through the merge process. The domain is BeautyAndTheGeek.IT which is a Google Apps Premier edition. My other domains that are hosted in Google Apps have not transitioned yet and I suspect that Google is first merging Premium accounts, then Educational, Government and it will finish up with the Standard edition of Google Apps, which kind of makes sense (paid customers first!).

Email Invitation

Invitation email to the Admin

The day before yesterday I received an email inviting me to participate in the new infrastructure change and merge the Google Apps accounts with the Google Accounts of my users.

The merge process works as follows:

- If your Google Apps account email address has never been used to access Google products such as Google Reader, Google Voice, Google Code etc. then your Google Apps account will be converted to a Google Account.

- If your Google Apps account email address has been used to access Google products such as Google Reader, Google Voice, Google Code etc. then your Google Apps account is conflicting with your Google Account. This page in Google Apps Administrator Help provides an overview of conflicting accounts. In short both of your accounts will merge and the data will be associated with your Google Apps account, which will now be the same as your Google Account.

There are certain limitations to this merge (as one would expect :)). These are as follows:

If the offline feature is enabled in GMail, the user has to synchronize before the transition. If they don't, they will lose their offline messages.
Offline Google Calendar doesn't work for Google Apps accounts that have transitioned.
The following Google products don't work with Google Apps accounts that have transitioned:
- Android Market for Developers
- Google Extra Storage (bummer)
- Health (bummer again)
- PowerMeter
- Profiles
- Web History
- YouTube
The purchase of additional Google Storage is currently unavailable for Google Apps accounts. No storage can be purchased for a Google Apps account, although it will be available later this year. If a user account has already purchased extra storage (like I have) in an existing Google Account, the storage will remain in that account and will not transfer to the respective Google Apps account after the transition.
Delegating email only works with the same account type (i.e. Google Apps)
Any Picasa Web Album, Profile, or Wave usernames cannot be moved from an existing account to your Google Apps account.

Resources: Transition readiness checklist, Conflicting accounts, Early adopters, Additional storage for Google Apps, Google Apps Administrator Help Center

The process

Google Apps - Dashboard Warning

Dashboard Warning

Once I logged in my Google Apps Administration area, the Dashboard presented a new notice at the top. In short the notice states that new services will be available to the Google Apps accounts. This will be achieved with the transition of the Google Apps Account to a Google Account. The update is free and Google will automatically roll it out on September 30, 2010.

Resource: Google Apps core suite

Google Apps - Understand Transition

Understanding the transition

Clicking the "Get Started" button at the bottom of the notice will start a wizard that helps with the transition of the domain's accounts to Google accounts of your users (some or all). The first screen contains information that aids in understanding the transition and what is involved. Moving to the new infrastructure will update your control panel and give you control over which Google services your users can access with their accounts.

Resource: Transition readiness checklist, Google Apps Administrator Help Center

Google Apps - New Services

Review New Services

Based on the services that are available for your domain user accounts, the screen above might be slightly different. It does however give you (the administrator) the ability to enable or disable services for your users. Note that turning off a service will disallow your users to sign up for that service using their Google Apps account for your domain. It will however not stop them from using a totally different Google Account (personal for instance) to access that service.

Google Apps - Notify Conflicting Accounts

Notify Conflicting Accounts

Confirming the new services, the wizard will display a new screen, that will allow you to notify those users that have conflicting accounts. This screen provides information on how many users have conflicting accounts (in my case only 1), what happens to those accounts (the merge process) and what should you do as the administrator. Google will not provide information on which of your users have conflicting accounts. What they will however do, is offer you a temporary email address and an email template, that you can use to email your users. The email template is shown in the "Google Apps - Email to User" section below.

Google Apps - Select Users

Select Users

Clicking "Continue", brings up the User Selection screen. Here the administrator can select whose account will be transitioned for now. You can choose to pilot test this transition with a small set of users or everyone. The difference is that if you choose a small set of users to pilot test this transition, you can revert their accounts back to what they were prior to this step. If you choose "Everyone" from this screen, the change will be across the organization and cannot be undone. Reminder here that the transition will happen either way by the end of September, 2010. Finally you have the option to inform the users when their account transition is complete. This generates an email (in English) to the user with relevant information. You can see the email in section "Google Apps - Email to User" below.

Resource: Early adopters

Google Apps - Confirmation

Confirmation

Clicking "Continue"again will bring up the final screen which is the confirmation page. Google is very thorough and I like the confirmation screen. The information in this screen is a summary of what the wizard collected in previous steps. You (the administrator) will have to confirm in several places that you have understood the process, read the agreement and then accept the whole process. Clicking "I accept. Start the transition." will make things happen :)

Resources: Transition readiness checklist

Google Apps - Transition in progress

Transition in Progress

Navigating back to the dashboard, you will see a message notifying you that there is a transition in place for the accounts of your domain and can take up to 24 hours to be completed. This serves as a reminder on what has just happened. The notice will disappear once the transition is completed.

Google Apps - Email to User

Email to User

If you (the administrator) chose to notify your users of this transition (see step Select Users above), they will receive an email like the one shown above. If you chose to notify your users with a different message then that message should have been sent to the temporary email address that Google has provided (see step Notifying Conflicting Accounts above)

Resources: Data ownership, Transition readiness checklist, Google Apps Administrator Help Center

Google Apps - Email to User after Transition

Email to User after transition

Finally another email will be sent to the user summarizing what has happened with their account. This email contains links to additional resources.

Resources: List of products, New sign in option, Data ownership, What is a browser?, Using multiple accounts, Help Center for Admins, Google Apps Administrator Help Center

Google Apps - User Login message for Account Merge

Message to user after login for account merge

Once the transition operation is completed, the next time the user logs in their account, they will see the screen above. In short this screen informs the user of what has happened and provides links to online resources for help. The user must click "I accept. Continue to my account." for them to have access to their account. This is a one off process.

Resources: Google Apps Administrator Help Center, Google Security and Privacy, Using multiple accounts, Terms of Service

Conclusion
Without a doubt this is a huge step forward for Google and for us as admins or users. The ability to have a single sign on to use their products not only helps them but us by simplifying everything. That also means that if a hacker guesses that my password is "password1" they have access to all my Google related services... but oh well :)

Joking apart though, I am really excited that this change has finally occurred. I am guessing that the next steps would be to give users or admins the ability to purchase additional storage for a single account is something that in my view will make quite a lot of money for Google. A lot of people will want to keep all their emails and are nearing or have reached their email quota. Administrators would be happy to pay for certain users that have depleted their email storage allocation without converting their whole Google Apps edition to the Premier one. After all, if your organization has 100 users and 5 of them are near capacity, even if you pay $20 per user it will cost $100 per year. If however the whole account is changed to a Premier account (with 25Gb per mailbox) the cost will go up to $5,000.

Some questions come to mind:

Will the user be able to purchase storage for all services or just for email?
Will the 1Gb limit from Docs be lifted and will it be combined with storage for mail?
$20 a year gives you 80Gb in Picasa. Will Google follow that model or change it? According to the Additional storage for Google Apps page it appears that it will be more expensive now to have more space.
If storage is so cheap (Picasa) will it be extended to Google Documents? It appears that 7.5Gb is enough for the vast majority of users as far as email is concerned, but 1Gb for Docs is not enough. For a society moving towards a full electronic document storage this could help a lot.

Still a long way to go until everything clears up. I am not sure that Google has all the answers yet but they are moving forward and this is the most encouraging news of all!

Subversion Backup [How-To]

2010-08-01T13:11:00.000-04:00

I will start this post once again with the words of a wise man:

There are two kinds of people, those who backup regularly, and those that never had a hard drive fail

So the moral of the story here is backup often. If something is to happen, the impact on your operations will be minimal if your backup strategy is in place and operational.

There are a lot of backup scenarios and strategies. Most of them suggest a backup once a day, usually at the early hours of the day. This however might not work very well with a fast paced environment where data changes several times per hour. This kind of environment is usually a software development one.

If you have chosen Subversion to be your software version control software then you will need a backup strategy for your repositories. Since the code changes very often, this strategy cannot rely on the daily backup schedule. The reason being is that, in software, a day's worth of work usually costs a lot more than the actual daily rate of the programmers.

Below are some of the scripts I have used over the years for my incremental backups, that I hope will help you too. You are more than welcome to copy and paste the scripts and use them or modify them to suit your needs. Please note though that the scripts are provided as is and that you must check your backup strategy with a full backup/restore cycle. I cannot assume responsibility of something that might happen in your system.

Now that the 'legal' stuff are out of the way, here are the different strategies that you can adopt. :)

svn-hot-backup
This is a script that is provided with Subversion. It copies (and compresses if requested) the whole repository to a specified location. This technique allows for a full copy of the repository to be moved to a different location. The target location can be a resource on the local machine or a network resource. You can also backup on the local drive and then as a next step transfer the target files to an offsite location with FTP, SCP, RSync or any other mechanism you prefer.

#!/bin/bash

# Grab listing of repositories and copy each
# repository accordingly

SVNFLD="/var/svn"
BACKUPFLD="/backup"

# First clean up the backup folder
rm -f $BACKUPFLD/*.*

for i in $(ls -1v $SVNFLD); do
    if [ $i != 'conf' ]; then
        /usr/bin/svn-hot-backup --archive-type=bz2 $SVNFLD/$i $BACKUPFLD
    fi
done

This script will create a copy of each of your repositories and compress it as a bz2 file in the target location. Note that I am filtering for 'conf'. The reason being is that I have a conf file with some configuration scripts in the same SVN folder. You can adapt the script to your needs to include/exclude repositories/folders as needed.

This technique gives the ability to immediately restore a repository (or more than one) by changing the configuration file of SVN to point to the backup location. If you run the script every hour or so then your downtime and loss will be minimal, should something happens.

There are some configuration options that you can tweak by editing the actual svn-hot-backup script. In Gentoo it is located under /usr/bin/. The default number of backups (num_backups) that the script will keep is 64. You can choose 0 to 'keep them all' but you can adjust it according to your storage or your backup strategy.

One last thing to note is that you can change the compression mechanism by changing the parameter of the --archive-type option. The compression types supported are gz (.tar.gz), bz2 (.tar.bz2) and zip (.zip)

Full backup using dump
This method is similar to the svn-hot-backup. It works by 'dumping' the repository in a portable file format and compressing it.

#!/bin/bash

# Grab listing of folders and dump each
# repository accordingly

SVNFLD="/var/svn"
BACKUPFLD="/backup"

# First clean up the backup folder
rm -f $BACKUPFLD/svn/*.*

for i in $(ls -1v $SVNFLD); do
    if [ $i != 'conf' ]; then
        svnadmin dump $SVNFLD/$i/ > $BACKUPFLD/$i.svn.dump
        tar cvfz $BACKUPFLD/svn/$i.tgz $BACKUPFLD/$i.svn.dump
        rm -f $BACKUPFLD/$i.svn.dump
    fi
done

As you can see, this version does the same thing as the svn-hot-backup. It does however give you a bit more control over the whole backup process and allows for a different compression mechanism - since the compression happens on a separate line in the script.

NOTE: If you use the hotcopy parameter in svnadmin (svnadmin hotcopy .....) you will effectively be duplicating the behavior of svn-hot-backup.

Incremental backup using dump based on revision
This last method is what I use at work. We have our repositories backed up externally and we rely on the backup script to have everything backed up and transferred to the external location within an hour, since our backup strategy is an hourly backup. We have discovered that sometimes the size of a repository can cause problems with the transfer, since the Internet line will not be able to transfer the files across in the allocated time. This happened once in the past with a repository that ended up being 500Mb (don't ask :)).

So in order to minimize the upload time, I have altered the script to dump each repository's revision in a separate file. Here is how it works:

We backup using rsync. This way the 'old' files are not being transferred.

Every hour the script loops through each repository name and does the following:

Checks if the *.latest file exists in the svn-latest folder. If no, then it sets the LASTDUMP variable to 0.
If the file exists, it reads it and obtains the number stored in that file. It then stores that number incremented by 1 in the LASTDUMP variable.
Checks the number of the latest revision and stores it in the LASTVERSION variable
It loops through the repository, dumps each revision (LASTDUMP to LASTVERSION) and compresses it

This method creates new files every hour so long as new code has been added in each repository via the checkin process. The rsync command will then pick only the new files and nothing else, therefore the data transferred is reduced to a bare minimum allowing easily for hourly external backups. With this method we can also restore a single revision in a repository if we need to.

The script that achieves that is as follows:

#!/bin/bash

# Grab listing of folders and dump each
# repository accordingly

SVNFLD="/var/svn"
BACKUPFLD="/backup"
CHECKFLD=$BACKUPFLD/svn-latest

for i in $(ls -1v $SVNFLD); do
    if [ $i != 'conf' ]; then
        # Find out what our 'start' will be
        if [ -f $CHECKFLD/$i.latest ]
        then
            LATEST=$(cat $CHECKFLD/$i.latest)
            LASTDUMP=$LATEST+1
        else
            LASTDUMP=0
        fi

        # This is the 'end' for the loop
        LASTREVISION=$(svnlook youngest $SVNFLD/$i/)

        for ((r=$LASTDUMP; r< =$LASTREVISION; r++ )); do
            svnadmin dump $SVNFLD/$i/ --revision $r > $BACKUPFLD/$i-$r.svn.dump
            tar cvfz $BACKUPFLD/svn/$i-$r.tgz $BACKUPFLD/$i-$r.svn.dump
            rm -f $BACKUPFLD/$i-$r.svn.dump
            echo $r > $CHECKFLD/$i.latest
        done
    fi
done

Conclusion
You must always backup your data. The frequency is dictated by the rate that your data updates and how critical your data is. I hope that the methods presented in this blog post will complement your programming and source control should you choose to adopt them.

Android 2.2 (Froyo) [Review]

2010-06-21T12:31:00.000-04:00

Android 2.2 (Froyo)

The newest version of the Android Operating System has been codenamed Froyo (as in Frozen Yogurt).

Although I have heard of the name, I was not following very closely the development of the said OS, so I did not know what to expect. The presentation on Day 2 at Google I/O was more than impressive so I had to get my hands on it :)

After the GoogleIO presentation, I was regularly checking (Settings – About Phone – System Updates) my phone to see what kind of version I am running and whether the update was waiting for me. Unfortunately I was disappointed every time, therefore I resigned to the idea that it will update when it is pushed to my phone.

A few days later though, reports on the blogosphere started appearing of users now running on Froyo. The update started rolling out so it was a matter of time for me. Most of the reports were coming from California, so if I were to look at the geography, it would take the update quite a bit of time to reach West Virginia :)

Through my regular research on the Internet, I found a few interesting blog posts that claimed that one can update the Nexus One without waiting for the update to be rolled out. Having nothing to lose, I decided to try it on my phone and see what happens.

Upgrade to Froyo

Android 2.2 (Froyo)

Below is how I upgraded my Nexus One to Froyo. I had a couple of failed attempts, loads of Googling but I finally managed to get it to work. Luckily a couple of days later an article appeared on Lifehacker which confirmed that the steps I took were the correct ones!

I connected my Nexus One to my computer using the USB cable.

The phone was detected and I mounted the SD card from the phone (bring down the notification area and select the USB connection – select mount to mount the SD card)

When I performed these steps I had to navigate to a URL on the Android website (see article on Lifehacker).

I copied the file and pasted it on my newly mounted drive which was the SD card of my Nexus One

The file is 45Mb and when the copy was completed, I unmounted the SD card from the computer initially (click the status area and select Safely remove <X> drive) and then from the phone (from the notification area select USB connection and select Unmount)

The update instructions below are the original work of SimonNWalker and can be referenced here.

With the update file uploaded on the Nexus One, all I had to do is shut down the device so that I can reboot it in recovery mode. The steps I took are as follows:

Figure 1: Skating Androids

I clicked and held the power button until the menu appeared. I selected 'Power Off', confirming that I wish the phone to power off.

Once the phone powered off completely, I pressed and held the trackball button down and then pressed the power button. A new screen appeared (which, like myself, you probably have never seen before) with three androids on skateboards at the bottom (Figure 1) and several options at the top (Figure 2).

Figure 2: Android FastBoot

I navigated using the power and the volume buttons. I selected the first option (Bootloader) using the volume buttons and pressed the power button to activate it.

A new menu appeared with the following options:

FASTBOOT

RECOVERY

CLEAR STORAGE

SIMLOCK

I navigated to RECOVERY and selected it

The familiar Nexus One X appeared on the screen. A few seconds later, a little android with a big exclamation mark in a triangle replaced the X.

The next step I took was to press simultaneously the power button and the volume up button.

This brought a new menu at the top (Figure 3).

Figure 3: Recovery

The options available were:

reboot system now
apply sdcard:update.zip
wipe data/factory reset
wipe cache partition

Using the trackball, I navigated to the second option and pressed the trackball button down.

At this point the update started and some information was flashing on the screen for a while, where some files were patched, some deleted, new ones copied and others replaced accordingly. The whole process took roughly 5 minutes from start to finish.

Features that I found

Please note that these are my own observations. Some of the features here might have been present in the previous version of Android (Eclair) and I simply did not notice them. If that is the case please let me know and I will correct the post accordingly.

Phone search (Search throughout the phone itself)

Figure 4: Phone Search

This is one of the updates that I just love. Froyo allows you to search a lot of content that is stored on your phone. The content includes your contacts, sms messages, applications, twitter feeds and many more. With all this power it is very easy to find the information you are looking for with the press of a few keys on the virtual keyboard (Figure 4).

Since "with great power comes great responsibility", Froyo allows you to select what can be searched (Figure 5). Go to Settings - Search - Searchable Items and you will find the following options:

- Web (web search, bookmarks and browser history)

- Aldiko (only if you have it installed)

- Apps (names of installed applications)

- Contacts (names of your contacts)

Figure 5: Search Areas

- Finance (stock tickers, company names) (only if you have it installed)

- Google Sky Map (only if you have it installed)

- Messaging (text in your messages)

- Music (artists, albums and tracks)

- Twitter (searchable tweets) (only if you have it installed)

I am sure that the search extends to other applications. Your results might vary based on the apps you have installed.

Tethering - Wi-Fi Hotspot

Figure 6: Wi-Fi Hotspot

When I first heard about this feature during the presentation of Froyo at Google I/O, I was really excited.

I spend every day 3 hours on the train, where I mostly work on my notebook. Having the ability to tether my phone and work online is essential. A couple of years back when I bought my first iPhone, I managed to jailbreak it and installed iPhoneModem on it to achieve the functionality that I wanted (tethering). It wasn't fancy (after all don't expect miracles with AT&T's EDGE network) but it worked. I could get my emails reply quickly and disconnect.

When I got my Nexus One, I bought PDA Net to achieve the same result. It worked too but again in a very basic mode but it was draining the battery very fast and was running the phone very hot.

After I upgraded to Froyo, I uninstalled PDA Net and have been working with the built in functionality ever since.

Figure 7: Configure Wi-Fi Hotspot

To access the Tethering options, you will need to go to Settings - Wireless & Networks - Tethering & portable hotspot. The options shown on Figure 6 appear, which allow you to switch the Wi-Fi hotspot on or off. A USB tethering option also exists for those that want to keep their phone charged with the power that comes from their notebook or attached device. Once the hotspot is enabled, a blue icon appears at the top notification bar. The configuration of the hotspot is really easy and can be seen in figures 7 and 8. For those wondering why I chose the SSID of my hotspot to be one of the most infectious (Windows) viruses ever, it is exactly for that reason. Apart from the WPA2 encryption, the name itself is a deterrent for anyone that might get ideas in stealing bandwidth.

Everything works perfectly apart from AT&T's EDGE network, which is really slow. The Nexus One I have can support T-Mobile's 3G network. I haven't switched to T-Mobile yet since there was no need. I fear though that I will not be able to escape the inevitable.

Figure 8: Wi-Fi Hotspot settings

On June 2nd AT&T issued a press release where they showed us once again that they don't give a damn about their customers since they lie, deceive and overcharge for a mediocre (at best) service. You can read the press release and draw your own conclusions but to me it seemed once again a slap in the face.

No more unlimited data plans for new customers, yet it is not clear what will happen to existing customers such as myself. I read somewhere in the blogosphere that those plans will remain as is, but this is AT&T that we are talking about - the same company that claims that calls to 1-800 numbers are free yet they charge you minutes for it.

I have signed for the unlimited data plan, yet AT&T does not allow me to tether. Why? What difference does it make? They still offer the same crappy EDGE network whether I watch a YouTube video on my phone or on my computer. The answer is in the press release. They want more money. As phones get 'smarter' they phone companies get greedier. John Gruber offers a good analysis of the new AT&T data plans.

I do not know how this will evolve but I will definitely continue using the Wi-Fi hotspot on my train ride, whether this will be with AT&T or T-Mobile. Perhaps if I change to T-Mobile I will be able to have better coverage. AT&T in West Virginia is not the greatest carrier. Notable is a recent phone conversation that I had with my wife while I was driving (I am wearing the headsets btw) where the line dropped 9 times :(

Phone/Web buttons at the bottom of every screen and desktop

Figure 9: Phone - Web buttons

This was something that was needed in my opinion (Figure 9). I was pressing way too many buttons to get to the phone, especially if I was not on the main screen. This little shortcut is very well received and thought of. I do not think that there is enough space for an additional 2 buttons (for the future release of Android) but you never know.

Perhaps a future version will allow you to customize those buttons.

Android development though has to be very cautious when releasing functionality. I am sure that Apple is checking everything that Android does with a microscope. With their enormous patent book near by, they will not hesitate to sue Android (or Google for that matter) for patent infringement (see Apple sues HTC).

More accounts to sync with

Figure 10: Sync Accounts

Another area that I noticed improvements, is the synchronization of the online accounts with the phone (Figure 10).

This area can be accessed via Settings - Accounts & sync settings.

I noticed that not only my regular GMail and Google Apps accounts appear but also YouTube as well as Twitter. I have installed the official Twitter application so I am pretty sure that this synchronization appears because of that application.

I have not tried it with Seesmic or any other twitter application, so if you have any additional information please let me know and I will modify this post accordingly.

I would also be very interested to know which other applications offer synchronization capabilities or take advantage of Android's synchronization API. If you have any other applications that synchronize on your phone, please let me know and I will include it in this post too.

Easy account switch (Email)

Figure 11: Email Account Switch

When I switched from an iPhone to an Android based phone, certain things were just "not right", not because there was something wrong with them - it was how I was used to things being done. One of those areas was the email. With the iPhone I was used to a traditional listing of accounts and then once something was selected I would go into the folders and then emails. If I wanted to change accounts I would have to go back two steps and then enter the account that I wanted.

This seems a very logical approach and it is easy to get used to. Although the Android has a better email management interface, it lacked the ability to quickly switch accounts and thus not spend time tapping away going back or forth. In that area the iPhone was better. Note the was. It was not because the user will tap less times, but because you would tap the back button twice which was located in the same area of the screen at all times. For the Android you had to press the menu button and then select Accounts. This was again the same amount of steps but the iPhone approach felt more natural.

With Froyo a new button appears at the top right of your email screen which will allow you to quickly go to the account selection screen. This effectively reduces the steps by one.

Analyzing briefly my emails, I can say that on average I receive 25 emails on my personal account and 35 on my business (I chose the two accounts that I get the most traffic). So if in theory I get two emails every time I check my email, that would mean that I am checking my phone 12 - 17 times a day (assuming again that I get the batches of emails on both accounts at the same time). It would therefore be safe to assume that I check my emails 15 times a day where I need to switch from one account to another.

So the math gives us:

	Taps	Average Email Checks per Day	Week	Month	Year
iPhone	2	15	210	900	10,950
Android	1	15	105	450	5,475

Clearly with the above I am using now half of the screen taps than I used to with Android Eclair or with the iPhone. However the this rough calculation shows how much I was tapping in the past prior to Froyo. Goodbye RSI. :)

Colored labels in your email (like GMail)

Figure 12: Email Colored Labels

If you are like me and use Google Apps or GMail, you are by now accustomed to the colored labels on your emails, that you don't know what you have been doing without them all this time.

With Froyo, this functionality is now available in my mobile device allowing me to visually identify emails of high interest.

For instance, Figure 12 shows my setup. As you can see I mark clients with a green label color and financial institutions (bills mainly) with a red color. When an email reaches my mailbox and is automatically labeled due to a relevant filter, I can easily identify its importance using this color coding.

Having this functionality on my mobile device is invaluable!

Easier navigation between emails - New < > buttons appear on the phone to get you from email to email

Another huge improvement in the navigation part as far as emails are concerned came in the screen where I read a specific email.

Figure 13: Email Navigation

There are two extra buttons at the bottom of that screen which allow navigation to the previous or next message (Figure 13).

Another great tap saving feature!

Better voice recognition

I had some problems with this feature from time to time primarily due to my accent – a blend of Greek – English British – German and English American. After a few tries on the new system, I can say that there is improvement since it recognized now queries that it had failed in the past.

The voice search will probably never be able to detect everything that everyone is saying due to the different accents and voices of people but it is getting pretty close to perfect in my case.

One thing that I love about voice search is the voice navigation. I have purchased the car dock for my Nexus One and I use the voice navigation almost everywhere I go. Understanding that I want to go to Rockville, MD instead of Rock Creek is awesome!

Marketplace – Update all, autoupdate for each application

Figure 14: Update All

This feature was missing and was probably one of the ones that were mostly requested by the users. The Update All to the installed applications.

Luckily the Android developers have heard our pleas and Froyo now features an Update All button at the bottom of the Downloads section in the Marketplace application (Figure 14).

Adding to this functionality, the user now has the ability to automatically update selected (or all) applications. When clicking on one of the applications to update, a checkbox appears which allows for automatic updates. If the checkbox is checked, the next time the selected application has an update, the phone will download it and install it.

There will always be a notification regarding the action in the notification area, but unless you know where the application is coming from (and you trust the source) you should keep this checked off. I know I might be getting a bit paranoid here but that is what I did.

Different notification for Text Messages (trackball)

I am not sure if this existed in Eclair but I just noticed it. When an email arrives, the trackball will start glowing briefly in regular intervals with a white color to visually notify me about the email(s) waiting for me.

If I receive a Google Voice message though, the trackball will still start glowing but this time it will be with a green color. This way I know that a text message is waiting for me.

I have sent a text message to my AT&T number and did see the trackball glowing but this time it was only white. It appears that the green trackball notification is a feature of Google Voice on Froyo or again it was always a Google Voice feature and I hadn't noticed, at which point I am getting excited for nothing :)

New icons for USB Connect/Disconnect and USB debugging

Figure 15: USB Debugging

These ones I loved them the first time I saw them. If USB debugging is enabled (Settings - Applications - Development - USB Debugging), the minute the phone is connected to the computer using the USB cable, a new icon will appear in the notification area (Figure 15). It appears that it is an android bug of sorts :)

Figure 16: USB Off

Figure 17: USB On

When the phone is connected to the computer via the USB cable, the user has the ability to use the SD card as a storage device. Once the relevant entry in the notification bar is tapped, the screen with the "Turn on USB storage" will appear. If I switch the USB storage on, the screen changes slightly (Figures 16 and 17).

These were two really cool (in my view) new screens that engage the user even more in exploring their device!

Camera control

Figure 18: Camera Controls

A new enhancement appeared in the Camera application (Figure 18).

The new menu that changes position based on the orientation of your phone (horizontal or vertical) allow for zooming, flash control, white balance control, geolocation and exposure.

The options available are:

Zoom: 1x, 1.2x, 1.4x, 1.7x, 2x
Flash mode: Auto, On, Off
White balance: Auto, Incandescent, Daylight, Fluorescent, Cloudy
Store location: Off, On
Focus mode: Auto, Infinity
Exposure: -2, -1, 0, +1, +2

Unfortunately these controls only appear when taking photos and not when shooting video. I am sure however that this functionality (and more) will be extended to the video capturing aspect of the camera application.

Select text

Figure 19: Select Text

This is an area where the iPhone was far better than the Eclair and unfortunately still is with Froyo. The gap though has decreased significantly.

With Froyo I can now select text from say an email and paste it somewhere else - even a different application. The Select Text option is hidden under the More menu button and once selected, it creates a small mouse pointer. That is the start of where it will start selecting (Figure 20).

I simply point to the top left area I want to copy from and drag my finger diagonally to end up at the bottom right of the area I want to select. This will select the text in a very appealing pink color and as soon as I lift my finger from the screen it will copy the text on the keyboard (Figure 21).

Figure 20: Mouse pointer

The technique on the iPhone is better but not by much - as I wrote the gap has decreased significantly. On the iPhone you have a magnifying glass where you can pinpoint exactly where you want to start copying (or inserting text - same functionality).

Again unfortunately this functionality (the one with the magnifying glass for selecting text) is patented by Apple Inc. and will not be seen on an Android based phone but I am sure that the Android developers will come up with something that will give us the same if not better user experience.

FLASH Support

Steve Jobs never kept a secret of his thoughts about Flash support on an iPhone. Although many people were hopeful that something will be worked out with Adobe in the end, Mr. Jobs shut the door on Adobe. Adobe then invested time and resources to work something out and right before they were able to release their solution Apple changed the iPhone Developer License Agreement to allow only applications written in Objective-C, C, C++ or Javascript and executed by the OS Webkit engine. Adobe had enough and issued a 'screw you apple' (not officially of course).

Figure 22: Flash on Android - PacMan

The presentation on Day 2 at Google I/O included the news that Flash will be allowed and supported on Android based phones that run Froyo (or newer versions).

I have to admit, the first version of Flash that I installed (it is still in BETA) was really slow. However three versions later, I am happy to announce that it works as well as a desktop machine. I have not tried to load a heavy flash based website but my brother in law's website (www.dnm.gr) loads just fine and you can see all the information that you need to see :)

I am sure that in the coming months we will see a lot more progress in that area.

Conclusions
The Android 2.2 (Froyo) is a huge step forward. It provides users with a lot of functionality that transforms a phone to a multifunction communication device. The only thing that we are missing now is proper coverage from the national carriers (AT&T this one is for you) and without having to sell our first born children to pay for the monthly bills (AT&T this one is for you too).

Update
There has been another blog post on AndroidAndMe explaining how to update Froyo to the latest build (this one is a patch not the real thing).

Update 2010-07-04
Happy 4th of July! It came with the update for Froyo (officially now) on my cellphone. I am now running the FFR91 build (and so does my wife).

2010 Memorial Day [USA][Celebrations][Personal]

2010-05-31T12:21:00.000-04:00

Today, the Memorial Day is celebrated across the USA. There are similar celebrations in Greece, not the same day as in the US but remembering and honoring the veterans for their sacrifices for the freedoms we and our families enjoy. I received an email this morning which I thought was too precious to be archived in my mail folders. Since I hate mass mailing and chain emails, I thought I would duplicate the email here for my archiving but also for everyone else to see. Please note that I am not the original author of the email and I do not know if it has appeared as a blog post or an article someplace. I do not own the images presented below either so if there is a copyright issue, please let me know and I will remove them immediately. If you know who the original author is, please let me know so that I can give them the mention they deserve.

MEMORIAL DAY

It is the VETERAN, not the preacher, who has given us freedom of religion.

It is the VETERAN, not the reporter, who has given us freedom of the press.

It is the VETERAN, not the poet, who has given us freedom of speech.

It is the VETERAN, not the campus organizer, who has given us freedom to assemble.

It is the VETERAN, not the lawyer, who has given us the right to a fair trial.

It is the VETERAN, not the politician, Who has given us the right to vote.

It is the VETERAN who salutes the Flag,

It is the VETERAN who serves under the Flag,

ETERNAL REST GRANT THEM O LORD,
AND LET PERPETUAL LIGHT SHINE UPON THEM.

No matter what the country of your origin, there are always some people that step up in difficult situations. Those men and women gladly give their lives for the better good, so that others can enjoy a better future. Those people are the Veterans that we must always remember and honor.

There is plenty of material regarding Memorial Day on the Internet. A site worth visiting is http://www.usmemorialday.org/ and two videos worth watching on YouTube are http://youtu.be/afd_sDNYbpY and http://youtu.be/wK0T4pVHP28

Update [Personal]

2010-05-18T11:59:00.000-04:00

It has been more than a month since I last blogged. I started a couple of blog posts but they are mostly ideas written down than structured posts, hence they are in my draft folder.

Throughout this month and change that I have been absent (I know my favorite readers have missed me - all 5 of you and no mom you are not in that count) many things have happened that made me sit back and reflect on what I have and what I have lost.

As any other family living the American Dream, we outgrew our current car capacity and therefore had to purchase a minivan!

2010 Dodge Grand Caravan

We decided to get the Dodge Grand Caravan mostly due to the fact that it looks great and also because we had a Dodge for the past 10 years and the features on it were exactly what we needed. The new vehicle would accommodate easily ourselves, our son Dimosthenis, our 2 year old Golden Retriever Hairy and our soon to arrive daughter Zoe, with plenty of room for our 'stuff' when going on short trips.

So the weekend after we purchased the minivan we decided to drive to Delaware, where my in-laws live to surprise them. They always love it when Dimosthenis visits so we thought it would be a good idea to spend the weekend there. We woke up relatively early, packed our stuff and were on our way to the beach. It was when we stopped to fill the minivan up when we heard the news.

Jim and Dimosthenis

Jim - my brother in law - had passed in his sleep. It was all a bad dream, I did not want to believe it but there was no denying it.

James Ruel Poston died peacefully in his sleep on March 20th. Cause: Heart condition.

We turned back home, let Hairy in the house (our neighbor was going to look after him) and headed straight to Baltimore where Jim's apartment was. The whole family gathered there, and inevitably we had to deal with the mundane logistics of the situation i.e. moving all of his stuff out of the apartment and cleaning it. It took a great deal of effort do do so and the psychological burden of the event did not help. A few days later, exhausted physically and psychologically we managed to complete the task.

Jim's obituary reads:

James Reuel Poston, 40, died Sunday, March 20, 2010, peacefully in his home. James was born April 11, 1969 in Washington, D.C. to Joy and Phillip Poston. He studied sports broadcast journalism in college – a passion he carried with him throughout his life. Family and friends will always remember his wonderful sense of humor and strong commitment to those he loved. James is survived by his parents, Joy and Steve Smith and Phillip and Susan Poston; grandmother Mary “YiaYia” Pantazis; his sister Tia and brother-in law Nikolaos; brothers Jonathan and Peter; sisters Lauren and Abigail; and nephew Dimosthenis. A funeral service was held Thursday, March 25, at St. George Greek Orthodox Church, 8805 Coastal Highway, Ocean City, Md. Interment was in Bethel Cemetery, Lewes. In lieu of flowers, make donations to the Cal Ripken Sr. Foundation, ripkenfoundation.org. Arrangements by Watson Funeral Home, Millsboro.

On Thursday, 25th of March 2010, Jim was buried in Lewes, DE. Family and friends gathered to say goodbye for the last time.

Jim will never be forgotten. His goofiness, smile, kind heart, unconditional love towards his nephew and infinite knowledge about sports will always be the things that defined him.

After the 40 day service following his passing, we started getting ready for a new arrival, our daughter Zoe Dimitra Dimopoulos.

Zoe Dimitra

Zoe Dimitra was born at 12:06 on the 3rd of May 2010. She weighed 7lbs 11oz and was 20 inches tall. She came out screaming, but soon was calm and happy in her mother's hands. Her name means Life in Greek, and it is also the name of my mother in law. It is a tradition to name your children with the names of your parents or in-laws.

We decided that she will have a middle name (Greeks rarely have middle names) and it is Dimitra.

The reason we chose this middle name is to honor my recently departed brother in law Jim. His Greek name was Dimitris (it translates roughly to James in English) and Dimitra is the female version of that name. James/Dimitris was also my wife's grandfather ('pappous' in Greek), so we thought that it would be nice to remember him too this way.

It has been a really rough month and a half, filled with tears, sorrow, tears, joy. Life goes on and Zoe is here with us.

I have started taking the train again to go to work which will give me roughly 3 hours uninterrupted programming and blogging. I will finally be able to continue on some of my projects like the <angular/> bug tracker, a blog based on Google App Engine, Python development and more.

Stay tuned!

Online Privacy, Spam, Buzz, Banks and Mail [Rant]

2010-03-31T09:19:00.000-04:00

Electronic Presence

Everyone more or less has some sort of electronic presence. That presence can easily be seen on Twitter, Facebook, personal blog, Google account, Buzz etc.

What most people neglect to realize that this is only the tip of the iceberg. How about the information your bank stores about you? Isn't that an electronic/online presence? Some server contains vital information about you and your bank uses that information to identify you against someone else. If your bank provides online banking, then the Internet becomes the interface that you use to access your own personal information (mainly funds) which again is stored somewhere else.

The same happens when you purchase something online. Say you want to purchase flowers for your loved one. You will be probably asked to create an account with the florists's website and you will need to provide payment for the flowers. The authenticator, in this case, is some company that will check your credit card against an online record (again stored somewhere) in order to authorize the payment.

Google Buzz
With this in mind, I have been wondering lately why all of a sudden everyone seems to be trying to stab Google, due to some mishaps with Buzz? Were the controls that Buzz had initially set correctly? You might say no here and I would have to respectfully disagree. The point is that there was the option for people to opt out, the terms of service were there for everyone to read (I am wondering who did if any) so why was something set incorrectly? Why complain if the information was available for everyone to make an intelligent and educated choice as to use the new tool or not? If you did not investigate and read all there is to it regarding Buzz then you are responsible for anything that comes with it.

I hear already people screaming at me that this was supposed to be an opt-in not an opt-out service. I agree and disagree there. Allow me to elaborate.

For starters Google offers Buzz (and GMail) for free. How come all of a sudden this free product (which mind you is of extremely high quality) became everyone's right and stopped being a privilege? Yeah sure, Google runs ads and displays them on the right side of your GMail. That is how they offset their costs. Is anything wrong with that? I don't think so. Yet you have people that try to download special extensions to block ads, they change their signature to contain certain keywords that Google has deemed inappropriate - in short endangering everyone's use of GMail because if it turns out to be totally not profitable for Google they will simply close down the service or start charging for it. If that happens the first people that will complain are going to be the ones that caused this i.e. the ones that were blocking ads.

Everyone complained about Buzz. It was not secure enough, it exposed personal information etc. If you looked closely, the choice to opt out was available. People will argue that it was not the default value but that is not the point. The option was available (and still is) and people should start reading what they sign up to. Terms and conditions exist for a reason and people should start reading that stuff and exploring the options/settings prior to signing up to something!

My Financial Institution
A few days ago I got a letter from Flagstar Bank, which is (was) my mortgage bank. It appears that they sold my mortgage to EverHome (just another mortgage company). This is a really common practice over the course of a mortgage. Thus far (6 months into my mortgage), I have changed 3 companies. I would be interested to see what will happen in the course of 30 years :).

After receiving this notice from Flagstar Bank, I got a new letter from EverHome with some interesting information. The most interesting part of this documentation was the Privacy policy of EverHome. Here is what is in store for me:

Do we disclose your information to non-affiliated third parties? We may disclose all of the information about you, as described above, to certain people, subject to legal restrictions, and subject to your right to ask us not to share your information. Your right to ask us not to disclose your information is referred to as your "Opt-Out Rights" and is more fully described below. To what non-affiliated parties do we disclose your information? Subject to your Opt-Out Rights described below, we may disclose your nonpublic personal information to the following types of third parties:

Financial service providers, such as banks, mortgage companies, mortgage brokers, consumer credit companies, investment advisors and similar companies;

Insurance providers, such as life insurance companies, credit insurance companies, hazard insurance companies, automobile insurance companies; and

Other optional service providers and organizations.

Even if you opt-out of information sharing, we will still share information about you with certain third parties as permitted by law or regulation. This may include but not limited to, disclosing information:

To parties assisting in servicing or processing of your loan;

To our lawyers, accountants, auditors, regulators and advisors;

To protect our rights relating to, and the security of, your loan, our website, or our telephone customer service center;

To verify the existence or status of your loan for a third party;

To companies that perform marketing servcices on our behalf, or jointly with us;

To other financial institutions with whom we have joint marketing agreements; and

When you ask or give us permission to do so.

Do we share your information with our affiliates? We are affiliated with several types of companies including but not limited to other financial service , such as banks, mortgage companies, and title insurance agents. Even if you exercise your Opt-Out Rights, we may share with our affiliates information about our direct transactions and experiences with you. Subject to your Opt-Out Rights, we may share information about you obtained from third parties with our affiliates, including:

Information from your application, such as your age, gender, marital status, number of dependents, assets, debts, income, employment information, address and the address of the property securing your loan;

Information from a consumer report, including information about your credit worthiness, financial circumstances, and credit history, including any bankruptcies or foreclosures;

Information to verify representations made by you, such as verification of your income, employment, and open lines of credit with others; and

Information from a person about his or her employment, credit, or other relationship with you, such as your employment history with your employer, or credit history with another financial institution.

How can you Opt-Out? If you do not want us to disclose nonpublic personal information about you to nonaffiliated third parties, or if you want to limit the nonpublic personal information we may disclose to our affiliates, you may Opt-Out <Opt-Out information goes here including mailing address and 800 phone number> Please remember that even if you choose to opt-out, we will still disclose information about you as permiited by law or regulation.

From what I have heard from friends, this is pretty much standard text for almost every mortgage company. Let's have a look a bit on a couple of interesting points and how that relates to Google and Buzz.

This is not the initial mortgage company. I did not allow them to do anything with my personal information. My initial mortgage company had a clause that basically said "we are allowed to sell your mortgage and with it your information". So I am stuck. This new company has my personal information (vital information such as my SSN, information about my previous employment, account balances and possibly account numbers, name, credit cards, in short everything). Did my old mortgage company delete my records from their systems? Most probably no. So here I am having 2 records out there without having to lift a finger. Did the introduction of Buzz create a brand new record of all of your personal information and disclosed it/forwarded it to a brand new company?
This is an Opt-Out policy. Wait a minute!!! Why don't I see a revolution, people running on the streets of major cities with signs, crying and condemning EverHome for such sacrilege, sending thousands of tweets, blogging about it.... condemning the company and associating them with the devil? EverHome has hundreds of thousands of customers. I never heard any complain about this! How many of those customers have GMail accounts? Chances are quite a few. How many complained about Buzz? Probably a significant number. Did they also complain to EverHome? Did they blog about it or tweet about it?
If you carefully read the above, you will see that my whole life, all the vital information about me, can be transmitted to various other companies so long as they are affiliated to EverHome. Do I have a saying in this? Not really. The last paragraph clearly states that even if I Opt-Out, they will release my information under certain circumstances "Please remember that even if you choose to opt-out, we will still disclose information about you as permiited by law or regulation."

Who else does Opt-Out?
This post is not to bad mouth or disgrace EverHome. Far from it. This post is to condemn hypocrisy. I have read many blogs from people condemning Google for their Opt-Out policy in Buzz - whereas it should have been Opt-In. Yeah fine, I agree to that. But then again, less than a month later Google changed their policy to an Opt-In, introduced more alerts, controls and visual effects to ensure that the user wants to Opt-In. That is Google.

Where are the blog posts complaining about the Opt-Out policies that other companies that we use their services on a daily basis? Did anyone bother to read the terms and conditions of say Verizon, AT&T for your cellphone, Satellite service (Dish/Direct TV) etc.? I dare you to check your paperwork; you will be shocked to find that almost everyone uses the Opt-Out approach and not the Opt-In.

Regular Mail
Consider also your regular mail. Whatever the postman brings. How much junk do you really get in your mailbox, just because you got a store card, ordered once from a website and now you cannot get them to stop? My wife got a gift certificate once for a motorbike accessory store for our cousin and since then we receive this really nice magazine concerning motorbikes. We returned it rejected more than 5 times but they do not get the hint. Did we ask for it? No. Do they have the right to do that? No, but they do not care. Did anyone complain about all this? Not as much as they complained about Buzz.....

Conclusion
Google is not the devil. I do believe that they adhere by the "Don't be evil" motto. I use their products and I am happy that I do because the quality of products offered help me do my job better. Do I read all the disclaimers and terms and conditions? Yes I do and if there is something I do not like I do not use it. If I click 'I accept the Terms' then I have no right to complain for something that was there and I did not read it or pay attention to it. If you don't like them, don't like their policies, do not use their products. Export your data from GMail (since Google adheres to the Data Liberation principles) to your computer or a different provider, delete your Google Account and stop complaining. There is a difference between making suggestions towards making a product better and bitching about something that is mostly your fault (i.e. reading Terms and Conditions and checking settings).

I always remind myself that the free Google services are a privilege to me and not my right. Shouldn't you do the same if using those same free products?

Rant on!

From iPhone to Nexus One [Review]

2010-02-28T08:52:00.000-05:00

References: "Leaving the iPhone" by Matt Cutts, Android Equivalency Table and "Good Things: Ubuntu and Android" by Alex Payne. I am pretty sure that there are other reviews and how-to's on the Internet, outlining that one of the above (or none) is the killer of the other. I am just posting what I did to switch effortlessly.

Homework

Of the applications currently installed on my iPhone, some are really very essential (i.e. email, web, podcast manager etc.) and some I can easily live without (Sportacular). Below is a list of the applications that I have listed in importance order and their counterparts in Android.

Severity	Application - iPhone	Application - Nexus One
H	Calendar	Calendar (built in - syncs automatically with the account you log in)
H	Google Voice	Google Voice (integrates with the Nexus One so you call using your Voice number)
H	GV Mobile	Google Voice
H	iPhoneModem	PDANet ($30) or if you root your Nexus One other applications. PDANet requires Bluetooth or the USB cable, the others can hook up a WiFi connection
H	iPod	Listen (by Google)
H	Mail	Mail (built in - allows for IMAP, Exchange, POP - it also syncs easily Google based email)
H	Maps	Web based
H	Phone	Phone (built in - can be replaced by Google Voice)
H	pTerm	Connectbot
H	Browser	Browser
H	Solebon	Solitaire (not as advanced as the paid Solebon but it will do)
H	Stanza	Apparently Panda Reader is out but I could not find it. I gave up on this since I mostly work on the notebook and not longer read books on my mobile device
H	WunderRadio	Streamfurious - not as good as WunderRadio but it has the stations I listen to
H	YouTube	YouTube
M	Buzz	Web based
M	Calculator	Calculator
M	Camera	Camera
M	CardStar	Key Ring Reward Cards
M	Clock	Clock - does not have countdown or stopwatch
M	Cycorder	Camera (built in)
M	FuelGauge	aCar
M	Google Earth	Google Earth
M	iTunes	Android Marketplace
M	Lattitude	Lattitude (a lot of privacy warnings - even an email - to ensure that you want to switch it on)
M	LinkedIn	Droidln
M	Photos	Photos
M	QuickMark	Barcode Scanner
M	RedLaser	Goggles, Barcode Scanner
M	Shazam	Shazam
M	TweetDeck	Seesmic
M	TWiT	TWiT
M	VNC	Remote VNC Lite
L	Dictionary	Dictionary
L	Facebook	Facebook
L	Flixster	Flixter
L	Messages	Messages (built in - integrates with Google Voice)
L	Notes	Evernote, like a billion other note apps some of which sync to Google Docs or other similar services, Tomdroid
L	Pregnancy	A lot of applications, Pregnancy Assistant one that comes close to Pregnancy for the iPhone - either way I won't need this after May (I think :))
L	Sears2Go	Sears2Go
L	Skype	Fring, I also use Skype Go so no problem for me there
L	SnapTell	Goggles
L	Sportacular	Scoreboard
L	Stocks	Finance
L	TrafficInfo	INRIX Traffic
L	USAA	USAA Mobile
L	Voice Memos	Google Voice
L	Weather	Weather on the top bar of one of the screens, News and Weather application
L	Wordpress	Wordpress

The Purchase
I knew I wanted to buy the Nexus One the minute it was presented to the world :) There were however a lot of factors to take into account, the applications that I had and what their replacements will be (see table above), whether it would work reliably with AT&T's network or whether I would have to switch to T-Mobile etc.

I have been following closely web posts regarding experiences with the Nexus One and I must admit, it was one of the main factors that helped me make the decision to switch. The icing on the cake was Matt Cutts's post regarding the same issue: Leaving the iPhone. I decided that I would get the phone and if it works satisfactory on AT&T I will stay with them, otherwise I will switch to T-Mobile.

I discussed the issue with the boss (my wife who else :)) and the purchase was approved. As a matter of fact she was disappointed when I told her that I am going to order it because she had plans to buy it for me for my birthday in April. My eagerness to get the new phone spoiled the surprise but we both agreed that I got my birthday present two months in advance :)

The purchase was really trouble free. I also purchased the dock and the really nice surprise is that Google offers free overnight FedEx to your location. It came in handy since I really wanted the phone there and then!

First experience
The phone arrived while I was at work. When I came back home, two really nice boxes were waiting for me. As you can see the packaging is really nice and the Google colors appear subtly on it.

Nexus One and Nexus One Dock packaging

Nexus One package - Subtle Google color detail

The packaging is simply awesome. Great ergonomics which easily match the ones of the iPhone. The Nexus One was wrapped in a plastic protective wrap, while in the package there was a protective sleeve, the USB cable, the headset and the charger. The packaging for the Nexus One Dock was equally good and it contained the dock and its charger.

Nexus One - Straight out of the box

Nexus One - Protective sleeve

Nexus One - Warranty Information

Nexus One - Charger, USB cable, Headsets

Nexus One Dock

Abiding to the instructions, I opened the phone, inserted the battery and plugged it in so that I will get a full charge prior to doing anything else.

Nexus One - Open with battery

As soon as I put the battery in, the phone booted up. Really slick graphics!

Nexus One - Booting up 1

Nexus One - Booting up 2

Nexus One - Booting up 3

While the Nexus One was charging, I inspected my iPhone once more and synced it to my computer so that I do not lose any data. Once the phone was fully charged, I removed the battery and inserted the AT&T card in it. The network was identified immediately and I was able to log in using my Google Apps account! :) After a few steps of setting up the phone was ready for use.

I plugged in the dock and set the phone up on it so that it charges prior to turning in. A nice surprise was the again subtle green digital clock that appeared on the phone while it is docked. I always wanted to have a nice clock on my bed stand - now I got it :)

The day after
I started the day with the attitude of 'Forget the iPhone and start anew'. It was difficult, I must admit, but I think I managed quite well. The more I was using the Nexus One the more I liked it. I installed all the applications that I wanted (see table above) and set up the screens with the relevant application icons, trying to resemble what I had on the iPhone so that I won't spend much time looking for application icons.

I set up Google Voice and that was one thing that really impressed me. The task was completed in a matter of minutes and now everyone will receive calls from my Google Voice number and not from my AT&T number. I no longer have to go to GV Mobile or the web based application to search for a contact and then call using Google Voice. Big plus here for Nexus One over the iPhone.

Another pleasant surprise came a bit later when my wife and I had to goto a car dealer to view some cars (minivan here we come). I had the instructions and I had added the address of the car dealer in the event in Google Calendar. Usually I would use Google Maps and calculate the mileage that I travel with what Google Maps says and before I reach the next milestone I look at the map again. This time I saw a 'Navigate' link in the menu that appeared and for the fun of it I chose it. I was in turn greeted with a female voice telling me exactly where to go. As I am really notorious for getting lost, the female voice proceeded to tell me to make a U-turn to get back on track :) Really slick and accurate! Big plus again for the Nexus One over the iPhone.

The notifications for the emails (I set up all my email accounts) are equally good for both phones. The Nexus One has a very slight advantage here in my opinion, since the notification area pulls up all the notifications that you have - from installed applications, text messages, missed phone calls, new emails etc. On the iPhone you get notifications on a per application basis so you need to be on the screen that lists the application that you received a notification for. Plus again for the Nexus One over the iPhone.

Website rendering is where the Nexus One gets another plus. I think it is hardware since I am using the same AT&T network but the sites load noticeably faster on the Nexus One over the iPhone. This is normal browsing in the house so the location is pretty much the same. Big plus again for the Nexus One over the iPhone.

Email management is where the iPhone is better. If I have say 4 emails to read and I click on one of them, I have to click the menu and then the 'Newer' button to get to the next email that I have not read. On the iPhone you have an up and a down arrow that allows you to navigate a lot easier. Also on the iPhone there is a logical hierarchy of accounts and folders, so you go back to change to a different account. Although this is a lot more work than it should, it seems a lot more logical than the approach the Nexus One has. With the Nexus One all you have to do is click the menu button and click Accounts to quickly access another account. It is a lot less clicks and I guess it would have made perfect sense should I had just come from a flip phone and never knew the iPhone. Still personal preference goes with the iPhone implementation, despite the fact that I am getting used to the new way of doing things :) Big plus for the iPhone over the Nexus One.

Multitasking - multiple application notifications. This one has been the most discussed thing on the Internet regarding the iPhone (and now the iPad). You cannot run multiple applications at the same time. I just received a notification that I have a new email and someone replied to one of my tweets. The Nexus One notification bar showed both notifications and I read both messages in no time. On the iPhone though I had to keep the TweetDeck application on so that I can get the notification OR the Mail application on so that I can get the email. Huge plus for the Nexus One over the iPhone.

There are other areas that I could discuss here but they are mostly focused on personal preferences on how things are done or should be done.

Conclusion
Overall I am really glad that I made the switch. The screen and camera are amazing, the phone is really responsive and apart from a couple of times that I didn't know how to do something and I had to Google it, I am using it as if I had it for a long time. I have no restrictions, I don't have to jailbreak the device so that I can get some applications that Apple deemed that they "ruin the user experience" (according to their reply to the FCC regarding Google Voice), my data is synchronized with the cloud and I can get it out of there at will. I can even develop my own application and install it if I wish to. This is something that the iPhone does not provide and notable is the fact that I had to retype all my store cards that I used Cardscan for in the Key Card scanner and there is no way for me to get the data that FuelGauge has stored in it and potentially transfer it to aCar.

I would say that the iPhone would suit someone that is not that tech savvy and is not impressed by the technical abilities of the phone, rather is focused on the amount of applications that exist for that phone (iPhone is superior there). The Nexus One though raised the bar way high and it will take a long time for the iPhone to even come close to it. Once the Apple introduces the ability to allow multiple applications to run at the same time, provide the liberty to get your data out of it if you wish to, stop trying to police people by restricting applications, then the gap would have been bridged but until then... in my humble opinion the Nexus One is king!

Spam - oh how I hate it [Rant]

2010-02-18T08:50:00.000-05:00

Everyone (by now) knows what spam is and by far it is one of the most hated 'things' of this world. Although it comes in many forms, people tend to be a lot more sensitive to spam when it relates to their email than their snail mail or any other form of it.

Junk is junk no matter how you want to see it. If I have not asked for something, then you are spamming me. If I politely say no the first time - don't send me any of the junk you are selling - I do not want it! Simple as that.

If materials (email form, mail form etc.) ignore the above then they are categorized as spam.

I have mentioned in a previous post that I have moved my domain to Google Apps and of course with it my email. This move allowed me to be virtually spam free, since Google's spam filter is the best one there is. The random email that I will get maybe once a month is not a problem since I can report it as spam and never see it again.

With the evolution of blogs and the need to allow people to comment on one's blog entries, spammers found new ground to spread their disease. The blog's comment box became a really good target, thus automated bots have been created to post junk in a blog's comment box. These entries make no sense, they contain links mostly to pharmaceutical products and sometimes are really funny. Their purpose though is to establish links pointing to a particular site (or sites) so that search engines (mainly Google) would be 'fooled' to bring the target site higher in the search result listing.

Features like rel='nofollow' on the href declaration of every link in the comments can help with this situation. This directive instructs the search engine spider not to follow the link in the new site, thus negating the intended spam's behavior.

The Akismet plugin for Wordpress blogs (such as this one) allows for a more automated protection against spam. Akismet captures spam comments and keeps them in a queue where the admin of the blog (moi!) can go later on and check if there are any false positives caught in the spam queue. I have to give kudos to everyone involved in creating the plugin to Akismet and the service itself. This contribution to bloggers around the world is invaluable.

The cure to spam? Delete it and/or ignore it. The spammers of this world thrive if they see a return on the investment. Note that there are two parties in the equation. The first party is the spammer, the person that actually does the deed - runs the mail server, sends the emails, creates the spam comments etc. The second guilty party is the actual employer of that spammer. The XYZ corporation that wants to sell pharmaceutical products or printer supplies etc. If the latter gets a zero return on their investment, then they will stop paying the spammers to send junk to us. Since their return on investment is minimal (I read somewhere that one needs 1/100,000 return to break even) they will keep on hiring spammers to send more and more spam.

What do you need to do? Just ignore spam. Report it, delete it, ignore it, protect against it. Don't buy anything from spam related messages in an effort to reduce and hopefully negate the return on investment of the spammer's employer. This will effectively benefit not only yourself but everyone else that uses the Internet.

I will leave you with two spam messages caught in Akismet a week ago which I find really funny:

For my post about SSL Certificates:

Normally, university teachers are willing to examine the business term paper writing technique of their students, nevertheless not all students can to write correctly because of a job or other issues. Therefore, a essay writing service should aid to compose the term paper thesis professionally.

For my post about PHP and the Factory Pattern:

I am bare impressed with the article I have just read. I wish the writer of http://www.niden.net can continue to provide so much useful information and unforgettable experience to http://www.niden.net readers. There is not much to say except the following universal truth: Everyone had an uncle who tried to steal their nose. I will be back.

Design Patterns – Factory [Series][How-To]

2010-02-02T08:42:00.000-05:00

A note about these series. It appears that Giorgio Sironi and I had the same idea regarding Design Patterns and blogging about them. He covers the Factory design pattern thoroughly in his blog post, which is recommended reading.

The Problem
I started off my IT career as a network administrator. This was back in the good old Novell 3.11 days. After that it was Novell 4.0, Microsoft Servers etc. Following that I got more and more involved with Visual Basic and when Microsoft decided to move everyone to .NET I chose not to follow and ended up coding in PHP.

Since my programming knowledge came from within (studying, reading articles, trial and error), the problems that I was facing on a daily basis are the same as almost every developer faces. One particularly challenging problem that I had in the VB days as well as the PHP days was repetition of code and how to eliminate it.

When I started programming for the Ferrari Fans Fun Forecast site I was running the site using my apartment's ADSL line. In the beginning there were only 20 users or so, therefore that setup was fine. The scripts were VBScript against a local instance of Microsoft SQL Server. Later on though, I switched to PHP while keeping Microsoft SQL Server.

Initial implementation
I knew that I would have to change my scripts later on to work against MySQL since I was to change the hosting of the site. Through laziness or poor design (you can pick either or both :)) I chose to create a class for Microsoft SQL and later on I would just change it to the MySQL. It seemed the easiest thing to do at the time.

So my class was something like:

class DbMSSQL
{
    private $_conn = null;

    public function connect()
    {
         // Connect
        // This is where we have the connection parameters
        include_once 'connection.inc.php';

        $this->_conn = mssql_connect($host, $user, $password);
        if (!$this->_conn) {
            throw new Exception('Cannot connect to the database.');
        }
    }

    public function disconnect()
    {
         // Disconnect
        mssql_close($this->_conn);
    }

    public function selectdb()
    {
         // Select the db
        $db = mssql_select_db($database, $this->_conn);
        if (!$db) {
            throw new Exception('Cannot select the database');
        }
    }

    public function query($sql)
    {
     // Query the db
        $_result = mssql_query($sql);

        if (!$_result) {
            throw new Exception('Error in query : ' . $sql);
        }

        $data = array();

        while ($row = mssql_fetch_assoc($_result)) {
            $data[$row['id']] = $row;
        }

        mssql_free_result($_result);

        return $data;
    }
}

Everything worked fine so I did not worry about a thing. A few months later though I was forced to move the database (as I expected) to MySQL. I could not move everything in one go so I had to move some of the tables initially and a week later everything else.

To tackle this requirement I created a second class to handle operations against MySQL. The class that I ended up with was:

class DbMySQL
{
    private $_conn = null;

    public function connect()
    {
         // Connect
        // This is where we have the connection parameters
        include_once 'connection.inc.php';

        $this->_conn = mysql_connect($host, $user, $password);
        if (!$this->_conn) {
            throw new Exception('Cannot connect to the database :' . mysql_error());
        }
    }

    public function disconnect()
    {
         // Disconnect
        mysql_close($this->_conn);
    }

    public function selectdb()
    {
         // Select the db
        $db = mysql_select_db($database, $this->_conn);
        if (!$db) {
            throw new Exception('Cannot select the database : ' . mysql_error());
        }
    }

    public function query($sql)
    {
     // Query the db
        $_result = mysql_query($sql);

        if (!$_result) {
            throw new Exception('Error in query : ' . mysql_error() . "\n" . $sql);
        }

        $data = array();

        while ($row = mysql_fetch_assoc($_result)) {
            $data[$row['id']] = $row;
        }

        mysql_free_result($_result);

        return $data;
    }
}

You can easily see the problem here. There is a lot of repetition in the code, not so much as the actual method properties but the methods themselves. Both classes have connect(), disconnect(), selectdb() and query() as methods. In reality the code changes only slightly since the call for an operation against Microsoft SQL Server is mssql_* while for MySQL is mysql_*. During the transition week I was in programming hell. At some point I mixed the class names, I was trying to read and update the wrong server etc. (see Jani Hartikainen's post about 6 programming project mistakes you should avoid - I did all that!).

That week though taught me that I need to pay more attention in designing rather than going full speed ahead with programming and later on paying the consequences.

After thorough research, I discovered a library that would support both platforms. The library that I found was ADOdb which is a perfect example of the Factory Pattern. I used that library later on for a different project, but just looking at the code and understanding the flow of operations as well as the implementation of the pattern itself was invaluable to me.

Interfaces
First of all I need to explain what an interface is and why we need them. According to PHP.net:

Object interfaces allow you to create code which specifies which methods a class must implement, without having to define how these methods are handled.

So imagine an interface something like a graft, a blueprint on what I need to construct. The Interface will have the common methods and properties that I need to implement.

When dealing with database connections to two different database servers (Microsoft SQL Server and MySQL), I can clearly define a few methods that will follow CRUD (Create, Read, Update, Delete). Those are:

Connect to the database server
Select database
Insert record
Delete record
Update record
Select record(s)
Close connection to the database server

My interface would therefore be:

interface iDatabase
{
    public function connect();
    public function disconnect();
    public function selectdb();
    public function query($sql);
}

Design Patterns - Factory
A class implementing the Factory Pattern is like a car manufacturing plant producing three different cars on the same assembly line. All cars have common characteristics like 4 wheels, 4 doors (well most of them), a steering wheel, a dashboard etc. and all of them perform certain operations i.e. drive, reverse etc.

In my problem earlier I could have used the Factory Pattern to create one class that would have implemented my blueprint, the interface which defines the CRUD operations that I need. So based on the above, the implementation will result in three classes.

MSSQL class - stored in the file Db_mssql.php

class Db_mssql implements iDatabase
{
    private $_conn = null;

    public function connect()
    {
         // Connect
        // This is where we have the connection parameters
        include_once 'connection.inc.php';

        $this->_conn = mssql_connect($host, $user, $password);
        if (!$this->_conn) {
            throw new Exception('Cannot connect to the database.');
        }
    }

    public function disconnect()
    {
         // Disconnect
        mssql_close($this->_conn);
    }

    public function selectdb()
    {
         // Select the db
        $db = mssql_select_db($database, $this->_conn);
        if (!$db) {
            throw new Exception('Cannot select the database');
        }
    }

    public function query($sql)
    {
     // Query the db
        $_result = mssql_query($sql);

        if (!$_result) {
            throw new Exception('Error in query : ' . $sql);
        }

        $data = array();

        while ($row = mssql_fetch_assoc($_result)) {
            $data[$row['id']] = $row;
        }

        mssql_free_result($_result);

        return $data;
    }
}

MySQL class - stored in the file Db_mysql.php

class Db_mysql implements iDatabase
{
    private $_conn = null;

    public function connect()
    {
         // Connect
        // This is where we have the connection parameters
        include_once 'connection.inc.php';

        $this->_conn = mysql_connect($host, $user, $password);
        if (!$this->_conn) {
            throw new Exception('Cannot connect to the database :' . mysql_error());
        }
    }

    public function disconnect()
    {
         // Disconnect
        mysql_close($this->_conn);
    }

    public function selectdb()
    {
         // Select the db
        $db = mysql_select_db($database, $this->_conn);
        if (!$db) {
            throw new Exception('Cannot select the database : ' . mysql_error());
        }
    }

    public function query($sql)
    {
     // Query the db
        $_result = mysql_query($sql);

        if (!$_result) {
            throw new Exception('Error in query : ' . mysql_error() . "\n" . $sql);
        }

        $data = array();

        while ($row = mysql_fetch_assoc($_result)) {
            $data[$row['id']] = $row;
        }

        mysql_free_result($_result);

        return $data;
    }
}

Notice that both these classes are almost identical to the initial implementation shown earlier in this post. The only difference is that they are both implementing the iDatabase interface.

So what is different now? The class that implements the Factory Pattern.

class Db
{
    public static function factory($type)
    {
        $fileName = 'Db_' . strtolower($type) . '.php';
        if (!file_exists($fileName)) {
            throw new Exception('File not found : ' . $fileName);
        }

        $className = 'Db_' . strtolower($type);

        return new $className;
    }
}

What this class does now is it allows me to load the relevant database connection class on the fly. If I want a Microsoft SQL connection I would call:

    $mssql = Db::factory('mssql');

while for MySQL the command becomes:

    $mysql = Db::factory('mysql');

Again since both underlying classes implement the iDatabase interface, I know exactly what to expect as far as methods and functionality is concerned from each class.

Conclusion
The Factory Design Pattern is one of the most powerful design patterns. It provides 'decoupling' i.e. breaks the inherited dependency of a class and its subclasses. It also allows for great flexibility while keeping the same interface for your clients.

Zend Framework uses the Factory Pattern in Zend_Db. Specifically the example on the site shows:

// We don't need the following statement because the
// Zend_Db_Adapter_Pdo_Mysql file will be loaded for us by
// the Zend_Db factory method.

// require_once 'Zend/Db/Adapter/Pdo/Mysql.php';

// Automatically load class Zend_Db_Adapter_Pdo_Mysql
// and create an instance of it.
$db = Zend_Db::factory('Pdo_Mysql', array(
          'host'     => '127.0.0.1',
          'username' => 'webuser',
          'password' => 'xxxxxxxx',
          'dbname'   => 'test'
      ));

The Zend_Db factory accepts the name of the adapter used for the database connection as the first parameter while the second parameter is an array with connection specific information. With the use of the Factory Pattern, Zend_Db exposes a common interface which allows programmers to connect to a number of databases using the same methods. Should in the future the application needs to access a different database, the impact to the developer is minimal - in most cases a change to the adapter name (first parameter of the factory class) is all it takes.

Design Patterns - Singleton [Series][How-To]

2010-01-22T08:36:00.000-05:00

A note about these series. It appears that Giorgio Sironi and I had the same idea regarding Design Patterns and blogging about them. He covers the Singleton design pattern thoroughly in his blog post, which is recommended reading.

The Problem
When I started programming in PHP I was faced with creating a simple database driven web page for a Ferrari Fans Fun club. The page had 5 different sections that each accessed the database to retrieve data. Each section was included in more than one page and only the menu, header and footer were common in all pages.

My first design and implementation was horrible. I still have the files and for the purposes of this blog post I went back and checked on them and I can safely say I am ashamed of that code. But then again we all start from somewhere so that was my start and more importantly I do not program like that any more. The picture below shows how the page was constructed:

Header
Menu	Content	Additional information
Footer

Each of the sections was a different php script (header.php, menu.php, content.php, footer.php, info.php) and in order to retrieve information from the database for each section I had the following snippet at the top of each script:

$dbName = 'FFFF';
$dbUser = 'ffff_user';
$dbPass = 'mypassword';
$dbHost = 'localhost';

$conn = mysql_connect($dbHost, $dbUser, $dbPass);
if (!$conn) {
    die(('Cannot connect to the database :' . mysql_error());
}

$db = mysql_select_db($dbName, $conn);
if (!$db) {
    die ('Cannot select the database : ' . mysql_error());
}

Some might comment on my error handling or the naming of the variables. That is not the problem. The problem is that the snippet of code above was used in every script file (all 5 of them). As a result every page load was hitting the database 5 times. Although the intended user base was no more than 100 people, due to this design flaw I had the equivalent of 500 users.

The first step - Primitive refactoring
You might argue that the two files (header, menu) can easily be combined into one (and the same with additional information and footer) and that will save me 3 connections. The layout does not change but now each of the shaded areas represent one script:

Header
Menu	Content	Additional information
Footer

Although this is a good start it is not the solution to the problem. I have effectively reduced the number of hits to 3 per visitor (300 vs 500 before). The goal is to have one connection per visitor.

One step further - A global variable
I need to create the database connection, store it in a global variable, and then let the rest of the scripts access that variable - and subsequently the database connection - when needed. The pseudo code is as follows:

Load script header.php
Get the database credentials
Create a database connection
Select the database
Display the data
Load script content.php
Get the database connection
Display the data
Load script footer.php
Get the database connection
Display the data

I need to ensure that my database connection is initiated at the beginning of every page. The script header.php is the most obvious place:

$dbName = 'FFFF';
$dbUser = 'ffff_user';
$dbPass = 'mypassword';
$dbHost = 'localhost';

$DBconn = mysql_connect($dbHost, $dbUser, $dbPass);
if (!$DBconn) {
    die(('Cannot connect to the database :' . mysql_error());
}

$db = mysql_select_db($dbName, $DBconn);
if (!$db) {
    die ('Cannot select the database : ' . mysql_error());
}

In every script thereafter I need to reference the global variable and I can then use it in that script:

global $DBconn;

Although this is an "acceptable" way of programming, maintaining all the global variables can easily be a nightmare for

maintenance
testing
quality control
any part of the code in any script that references this global variable can effectively change that variable
that the code will look "ugly" (hey I am proud of the code that I write :))

Design Patterns - Singleton
A better approach to solve this problem is to use a design pattern. In this case I will use the Singleton Pattern.

The Singleton pattern is applied to a class which when called will create a database connection if the connection does not exist or pass the connection back to the caller if it has already been instantiated. This way I really do not care where the database credentials will be added and when the connection will be instantiated. The first time that I am calling the class that implements the Singleton pattern will connect to the database and have the connection stored ready to be used. The pseudo code is as follows:

Load script header.php
Get the database credentials
Create a database connection
Select the database
Display the data
Load script content.php
Get the database connection
Display the data
Load script footer.php
Get the database connection
Display the data

The class that I created is as follows:

class Db
{
    private static $_db   = null;
    private static $_conn = null;

    private function __construct()
    {
        // This is where we have the connection parameters
        include_once 'connection.inc.php';

        $this->_conn = mysql_connect($host, $user, $password);
        if (!$this->_conn) {
            throw new Exception('Cannot connect to the database :' . mysql_error());
        }

        $db = mysql_select_db($database, $this->_conn);
        if (!$db) {
            throw new Exception('Cannot select the database : ' . mysql_error());
        }
    }

    private function __destruct()
    {
        mysql_close($this->_conn);
    }

    // The singleton method
    public static function getInstance()
    {
        if (null === self::$db) {
            self::$_db = new Db($options);
        }

        return self::$_db;
    }

    public function query($sql)
    {
        $_result = mysql_query($sql);

        if (!$_result) {
            throw new Exception('Error in query : ' . mysql_error() . "\n" . $sql);
        }

        $data = array();

        while ($row = mysql_fetch_assoc($_result)) {
            $data[$row['id']] = $row;
        }

        mysql_free_result($_result);

        return $data;
    }
}

With this class available I really do not care if my database connection code is at the beginning of my scripts or not. Using this class allows me to create the database connection (if it is not established) and persist/reuse it further down the script execution.

The code for my header and menu (see graphic above) becomes:

    $sql = 'SELECT menu_id, menu_name FROM tbl_menu';
    $menu = Db::getInstance()->query($sql);

while the one for the rest of the site is exactly identical sans the query to be executed. The problem is solved (I now have one connection per visitor) and the code seems a lot tidier.

Note that the connection parameters are in a separate file which is accessed during the __construct() method of the class. You can use anything you want to supply these parameters in your class.

Conclusion
The Singleton Design Pattern is a blessing in disguise. If the ground work has not been done (i.e. create tests for your code and thoroughly document it) then it is difficult for a new developer coming into a project to understand what is going on, especially when the new developer needs to make alterations and run newly created tests.

A word of caution: If you choose to use this pattern in your application, make sure that everything you do is thoroughly documented and tested. This will make your life a lot easier in the long run and will aid in maintenance.

Update: Thanks to Jani Hartikainen for pointing out an error in the code.

Create a SSL Certificate (Linux) [How-To]

2010-01-10T17:41:00.000-05:00

There are times that I want to set up a secure communication with the server I am working on. This might be because I want to run phpMyAdmin over SSL (I do not like unencrypted communications over the Internet), install a certificate for an eShop for a client or just for my personal use.

The first time I did this, I had to research on the Internet and after a bit of a trial and error I managed to get everything working. However if you do not do something on a regular basis you will forget. I am no exception to this rule hence this post to allow me to remember what I did and hopefully help you too.

Prerequisites:
This how-to assumes that you are running Gentoo, however these instructions can easily be applied to any other Linux distribution.

I need to check if openssl is installed:

vanadium ~ # emerge --pretend dev-libs/openssl

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R   ] dev-libs/openssl-0.9.8l-r2

If you do not see the [R] next to the package (and you see a N for instance) that means that you need to install the package. Issuing:

vanadium  ~ # emerge --verbose dev-libs/openssl

will do the trick.

Generate the Private Key
I like to generate keys with a lot of bits. All of my certificates have 4096 bits. This is a personal preference and it does not hurt to keep that value. Your host or Signing Authority (like GoDaddy, VeriSign, Thawte etc.) might ask you in their instructions to generate one with 2048 bits so don't be alarmed there.

Creating the RSA private key with 4096 bits using Triple-DES:

vanadium ~ # openssl genrsa -des3 -out /root/vanadium.niden.net.locked.key 4096
Generating RSA private key, 4096 bit long modulus
.............................................................++
...........++
e is 65537 (0x10001)
Enter pass phrase for /root/vanadium.niden.net.locked.key:
Verifying - Enter pass phrase for /root/vanadium.niden.net.locked.key:

Remove the passphrase from the Private Key
The key that was created earlier has a passphrase. Although this is good, it does have a side effect that any web server administrator does not like - the passphrase itself. Once the certificate is installed using the key (with the passphrase), every time that Apache is restarted, it will prompt the operator for the passphrase. This can be very inconvenient if your web server reboots in the middle of the night. Since Apache will be waiting for the passphrase, your site will be inaccessible.

To avoid this inconvenience, I am removing the passphrase from the key. If you noticed the key that I have created above has the 'locked' phrase in its name. The reason is that I know that that particular key has the passphrase on it. I first need to copy the key and then remove the passphrase:

vanadium ~ # cp -v vanadium.niden.net.locked.key vanadium.niden.net.key
`vanadium.niden.net.locked.key' -> `vanadium.niden.net.key'
vanadium ~ # openssl rsa -in vanadium.niden.net.locked.key -out vanadium.niden.net.key
Enter pass phrase for vanadium.niden.net.locked.key:
writing RSA key

Generate the Certificate Signing Request (CSR)
The purpose of the CSR is to be sent to one of the Certificate Authorities (GoDaddy, VeriSign, Thawte etc.) for verification. Alternatively I can self-sign the CSR (see below).

Upon generation of this CSR I am asked about particular pieces of information to be incorporated in the CSR. The most important piece of information that I need to ensure that is correct is the "Common Name". The answer to that question has to be the name of my web server - vanadium.niden.net in my case.

NOTE: I am using the key without the passphrase.

The command to generate the CSR is as follows:

vanadium ~ # openssl req -new -key vanadium.niden.net.key -out vanadium.niden.net.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Virginia
Locality Name (eg, city) []:Arlington
Organization Name (eg, company) [Internet Widgits Pty Ltd]:niden.net
Organizational Unit Name (eg, section) []:IT
Common Name (eg, YOUR name) []:vanadium.niden.net
Email Address []:domains@niden.net

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Once this step is completed, I open the CSR file with a text editor, copy the contents and paste them in the relevant field of the Certification Authority (in my case GoDaddy), so that they can verify the CSR and issue the certificate.

If however this is a development box or you do not want your certificate signed by a Certification Authority, you can check the section below on how to generate a self-signed certificate.

Generating a Self-Signed Certificate
Generating a certificate valid for 365 days, I need to issue the following command:

At this point you will need to generate a self-signed certificate because you either don't plan on having your certificate signed by a CA, or you wish to test your new SSL implementation while the CA is signing your certificate. This temporary certificate will generate an error in the client browser to the effect that the signing certificate authority is unknown and not trusted.

To generate a temporary certificate which is good for 365 days, issue the following command:

vanadium ~ # openssl x509 -req -days 365 -in vanadium.niden.net.csr -signkey vanadium.niden.net.key -out vanadium.niden.net.crt
Signature ok
subject=/C=US/ST=Virginia/L=Arlington/O=niden.net/OU=IT/CN=vanadium.niden.net/emailAddress=domains@niden.net
Getting Private key

Installation
For my system, the certificates are kept under /etc/apache2/ssl/ so I am going to copy them there (your need to adjust the instructions below to suit your system/installation):

vanadium ~ # cp -v vanadium.niden.net.key /etc/apache2/ssl/
vanadium ~ # cp -v vanadium.niden.net.crt /etc/apache2/ssl/

I also need to open the relevant file to enable the certificate

vanadium ~ # nano -w /etc/apache2/vhosts.d/00_default_ssl_vhost.conf

In that file I need to change the following directives:

SSLCertificateFile /etc/ssl/apache2/vanadium.niden.net.crt
SSLCertificateKeyFile /etc/ssl/apache2/vanadium.niden.net.key

If my certificate was issued by a Certificate Authority, the files that I have received have the CA certificate file. I can enable it in the following line:

SSLCACertificateFile /etc/ssl/apache2/vanadium.niden.net.ca-bundle.crt

Restarting Apache
vanadium ~ # /etc/init.d/apache2 restart

vanadium ~ # /etc/init.d/apache2 restart
 * Stopping apache2 ...                              [ ok ]
 * Starting apache2 ...                              [ ok ]

Navigating to https://vanadium.niden.net should tell me if what I did was successful or not. If your browser (Google Chrome in my case) gives you a bright red screen with all sorts of warnings, that means that

either you self signed the certificate - in which case it complains about the certificate not being signed by a Certificate Authority or
you made a mistake and the Common Name in the certificate is not the same as the host name.

Both these errors are easy to fix.

Design Patterns - Series [Series][How-To]

2010-01-06T17:35:00.000-05:00

A lot of developers – including myself – have at some point in their programming careers found themselves repeating a task that they have completed for a previous project or for a previous part of the same project.

This problem of repeating code is solved by design patterns. A design pattern is the way that the code must be structured so that the problem at hand can be solved. The characteristics of design patterns are:

Name – it lets other programmers know what this pattern does.
Problem – where the pattern can be applied
Solution – how this pattern is implemented

In the course of the next few weeks, I will publish a series of blog posts, where I will try to present you with some common design patterns and how I have understood their use. Most books and online materials concentrate on just presenting the issue at hand and do not focus on how we ended up there. In these blog post series I will try to show you what I found along the way and how I solved each problem in turn.

As usual comments are more than welcome.

The design patterns I will discuss in posts of these series are as:

Singleton Pattern
Factory Pattern
Registry Pattern
Decorator Pattern
Adapter Pattern
Iterator Pattern
Strategy Pattern
Active Record Pattern
Value Object Pattern
Model View Controller Pattern
MockObject Pattern
Observer Pattern
Proxy Pattern
Table Data Gateway Pattern
Data Mapper Pattern

You can check this blog every week, subscribe to the RSS feed, follow me on Twitter or follow my Google Reader (or all of the above).