niden.net

Building a web app with PhalconPHP and AngularJS Update [How-To][PhalconPHP][AngularJS]

2012-11-29T00:00:00.000-05:00

It's been a while since I last wrote a blog post, so I wanted to touch on the effort to upgrade the application that I wrote for Harry Hog Football using PhalconPHP and AngularJS.

If you haven't read it, the first two blog posts were here and here.

The application was written using the 0.4.5 version of PhalconPHP Since then there have been significant changes to the framework, such as the introduction of a DI container, injectable objects and lately interfaces (in 0.7.0, to be released in a couple of days).

There are a couple of things that I as a developer would like to see in PhalconPHP, which I am pretty sure will appear later on, since let's face it the framework is still very young (not even 1.0 version yet). Despite its "youth" it is a robust framework with excellent support, features and a growing community. One of these features is behaviors which I had to implement myself, and this was something new that came with this upgrade.

Recently a new repo has been created on Github called the incubator, where developers can share implementations of common tasks, that act as drop ins to the framework and extend it. These implementations are all written in PHP so everyone can just download them and use them. The more submissions come in, the more the framework will grow and eventually these submissions will become part of the framework itself.

Converting the 0.4.x application to 0.5.x

The task of converting everything from 0.4 to 0.5 was a bit challenging. The reason behind it was the DI container and how best to use it to suit the needs of the current application. Now these challenges would not even be an issue if one started writing their application from scratch, but since I had everything in place, I ventured into upgrading vs. rewriting. Note that this kind of upgrade will most likely never happen again, since the framework has been changed accordingly so that future upgrades will not require developers to rewrite their code (like I did now). From 0.5.x onward the framework design has been kind of "frozen".

I decided to create a new library that will help me with my tasks. I therefore created a custom bootstrap class, that would instantiate everything I wanted in my code. A short snippet of the class is below (the full code of course is in my Github repo which you are more than welcome to download and modify to suit your needs)

namespace NDN;

use \Phalcon\Config\Adapter\Ini as PhConfig;
use \Phalcon\Loader as PhLoader;
....
use \Phalcon\Exception as PhException;

class Bootstrap
{
    private $_di;

    /**
     * Constructor
     * 
     * @param $di
     */
    public function __construct($di)
    {
        $this->_di = $di;
    }

    /**
     * Runs the application performing all initializations
     * 
     * @param $options
     *
     * @return mixed
     */
    public function run($options)
    {
        $loaders = array(
            'config',
            'loader',
            'environment',
            'timezone',
            'debug',
            'flash',
            'url',
            'dispatcher',
            'view',
            'logger',
            'database',
            'session',
            'cache',
            'behaviors',
        );


        try {

            foreach ($loaders as $service)
            {
                $function = 'init' . ucfirst($service);

                $this->$function($options);
            }

            $application = new PhApplication();
            $application->setDI($this->_di);

            return $application->handle()->getContent();

        } catch (PhException $e) {
            echo $e->getMessage();
        } catch (\PDOException $e) {
            echo $e->getMessage();
        }
    }

    // Protected functions

    /**
     * Initializes the config. Reads it from its location and
     * stores it in the Di container for easier access
     *
     * @param array $options
     */
    protected function initConfig($options = array())
    {
        $configFile = ROOT_PATH . '/app/var/config/config.ini';

        // Create the new object
        $config = new PhConfig($configFile);

        // Store it in the Di container
        $this->_di->set('config', $config);
    }

    /**
     * Initializes the loader
     *
     * @param array $options
     */
    protected function initLoader($options = array())
    {
        $config = $this->_di->get('config');

        // Creates the autoloader
        $loader = new PhLoader();

        $loader->registerDirs(
            array(
                ROOT_PATH . $config->app->path->controllers,
                ROOT_PATH . $config->app->path->models,
                ROOT_PATH . $config->app->path->library,
            )
        );

        // Register the namespace
        $loader->registerNamespaces(
            array("NDN" => $config->app->path->library)
        );

        $loader->register();
    }
    
    ....

    /**
     * Initializes the view and Volt
     *
     * @param array $options
     */
    protected function initView($options = array())
    {
        $config = $this->_di->get('config');
        $di     = $this->_di;

        $this->_di->set(
            'volt',
            function($view, $di) use($config)
            {
                $volt = new PhVolt($view, $di);
                $volt->setOptions(
                    array(
                        'compiledPath'      => ROOT_PATH . $config->app->volt->path,
                        'compiledExtension' => $config->app->volt->extension,
                        'compiledSeparator' => $config->app->volt->separator,
                        'stat'              => (bool) $config->app->volt->stat,
                    )
                );
                return $volt;
            }
        );
    }
    ....

    /**
     * Initializes the model behaviors
     *
     * @param array $options
     */
    protected function initBehaviors($options = array())
    {
        $session = $this->_di->getShared('session');

        // Timestamp
        $this->_di->set(
            'Timestamp',
            function() use ($session)
            {
                $timestamp = new Models\Behaviors\Timestamp($session);
                return $timestamp;
            }
        );
    }
}

I chose to show a few sections of this bootstrap which I will explain shortly. What this bootstrap class does is it initializes my whole environment and keeps my index.php file small.

error_reporting(E_ALL);

try {

    if (!defined('ROOT_PATH')) {
        define('ROOT_PATH', dirname(dirname(__FILE__)));
    }

    // Using require once because I want to get the specific
    // bootloader class here. The loader will be initialized
    // in my bootstrap class
    require_once ROOT_PATH . '/app/library/NDN/Bootstrap.php';
    require_once ROOT_PATH . '/app/library/NDN/Error.php';

    // Instantiate the DI container
    $di  = new \Phalcon\DI\FactoryDefault();

    // Instantiate the boostrap class and inject the DI container 
    // in it so that services can be registered
    $app = new \NDN\Bootstrap($di);
   
    // Here we go!
    echo $app->run(array());

} catch (\Phalcon\Exception $e) {
    echo $e->getMessage();
}

As you can see the index.php is very small in terms of code.

Let's have a look at a couple of the functions that are in the bootstrap.


    /**
     * Initializes the config. Reads it from its location and
     * stores it in the Di container for easier access
     *
     * @param array $options
     */
    protected function initConfig($options = array())
    {
        $configFile = ROOT_PATH . '/app/var/config/config.ini';

        // Create the new object
        $config = new PhConfig($configFile);

        // Store it in the Di container
        $this->_di->set('config', $config);
    }

Pretty straight forward. The config INI file is read from its location and stored in the DI container. I need to do this first, since a lot of the parameters of the application are controlled from that file.


    /**
     * Initializes the loader
     *
     * @param array $options
     */
    protected function initLoader($options = array())
    {
        $config = $this->_di->get('config');

        // Creates the autoloader
        $loader = new PhLoader();

        $loader->registerDirs(
            array(
                ROOT_PATH . $config->app->path->controllers,
                ROOT_PATH . $config->app->path->models,
                ROOT_PATH . $config->app->path->library,
            )
        );

        // Register the namespace
        $loader->registerNamespaces(
            array("NDN" => $config->app->path->library)
        );

        $loader->register();
    }

The loader is what does all the discovery of classes for me. As you can see I store a lot of the paths in the config INI file, and I register my custom namespace NDN.


    /**
     * Initializes the view and Volt
     *
     * @param array $options
     */
    protected function initView($options = array())
    {
        $config = $this->_di->get('config');
        $di     = $this->_di;

        $this->_di->set(
            'volt',
            function($view, $di) use($config)
            {
                $volt = new PhVolt($view, $di);
                $volt->setOptions(
                    array(
                        'compiledPath'      => ROOT_PATH . $config->app->volt->path,
                        'compiledExtension' => $config->app->volt->extension,
                        'compiledSeparator' => $config->app->volt->separator,
                        'stat'              => (bool) $config->app->volt->stat,
                    )
                );
                return $volt;
            }
        );
    }

This is an interesting one. Registering the view and Volt. Volt is the template engine that comes with Phalcon. It is inspired by Twig and written in C, thus offering maximum performance. I set the compiled path, extension and separator for the template files and also I have a variable (set in the config of course) to allow the application to always create template files or not. In a production environment that variable (stat) will be set to false since templates do not change.

    /**
     * Initializes the model behaviors
     *
     * @param array $options
     */
    protected function initBehaviors($options = array())
    {
        $session = $this->_di->getShared('session');

        // Timestamp
        $this->_di->set(
            'Timestamp',
            function() use ($session)
            {
                $timestamp = new Models\Behaviors\Timestamp($session);
                return $timestamp;
            }
        );
    }

The above is my implementation of behaviors. Of course it is far from perfect but it works the way I want to. A better implementation of this has been written by Wojtek Gancarczyk and is available in the incubator. All I do here is go through the behaviors I have (Timestamp only for now) and register them in the DI container so that I can reuse them later on with any model that needs them.

Models

Every model I have that interacts with my database tables extends the NDN\Model.

class Model extends \Phalcon\Mvc\Model
{
    protected $behaviors = array();

    /**
     * Adds a behavior in the model
     *
     * @param $behavior
     */
    public function addBehavior($behavior)
    {
        $this->behaviors[$behavior] = true;
    }

    public function beforeSave()
    {
        $di   = Di::getDefault();

        foreach ($this->behaviors as $behavior => $active)
        {
            if ($active && $di->has($behavior))
            {
                $di->get($behavior)->beforeSave($this);
            }
        }
    }

    /**
     * @param array $parameters
     *
     * @static
     * @return Phalcon_Model_Resultset Model[]
     */
    static public function find($parameters = array())
    {
        return parent::find($parameters);
    }

    /**
     * @param array $parameters
     *
     * @static
     * @return  Phalcon_Model_Base   Models
     */
    static public function findFirst($parameters = array())
    {
        return parent::findFirst($parameters);
    }
}

The class itself is pretty simple, offering find and findFirst to the class that extends this. The interesting thing is that it also registers behaviors and calls the relevant validator function. So for instance the beforeSave validator checks the registered behaviors ($behaviors array), checks if they are active, checks if they exist in the DI container and gets them from there and then calls the beforeSave in the behavior class. The behavior class is equally simple:

class Timestamp
{
    protected $session;

    public function __construct($session)
    {
        $this->session= $session;
    }

    /**
     * beforeSave hook - called prior to any Save (insert/update)
     */
    public function beforeSave($record)
    {
        $auth     = $this->session->get('auth');
        $userId   = (isset($auth['id'])) ? (int) $auth['id'] : 0;
        $datetime = date('Y-m-d H:i:s');
        if (empty($record->created_at_user_id)) {
            $record->created_at         = $datetime;
            $record->created_at_user_id = $userId;
        }
        $record->last_update         = $datetime;
        $record->last_update_user_id = $userId;
    }
}

So effectively every time I call the save() function on a model, this piece of code will be executed, populating my fields with the date time and the user that created the record and/or updated it. In order to get this functionality to work, all I have to do in my model is to register the behavior like so:

class Episodes extends \NDN\Model
{
    /**
     * Initializes the class and sets any relationships with other models
     */
    public function initialize()
    {
        $this->addBehavior('Timestamp');
        $this->hasMany('id', 'Awards', 'episode_id');
    }
}

Controllers

Very little has changed in the controller logic, so that was the easiest part of the upgrade. Of course I tweaked a few things but the code works as is. I still extended my custom NDN\Controller class which takes care of my breadcrumbs (NDN\Breadcrumbs) as well as the construction of the top menu. The biggest difference with the previous version is that I stopped using AngularJS to populate the menu (so I am no longer sending a JSON array in the view) and used Volt instead. It was a matter of preference and nothing more.

Views

Quite a bit of work had to be done in the views to switch everything to use Volt. Of course every view extension had to be changed to .volt but that was not the only change. I split the layout to use partials so that the header, navigation and footer are different sections (organizing things a bit better) and kept the master layout index.volt. I started using the built in Volt functions to generate content as well as tags and it was a nice surprise to see that everything was easy to use and it worked!

<!DOCTYPE html>
<html ng-app='HHF'>
    {{ partial('partials/header') }}
    <body>


    <div id="spinner" style="display: none;">
        {{ image('img/ajax-loader.gif') }} Loading ...
    </div>

    {{ partial('partials/navbar') }}

    <div class='container-fluid'>
        <div class='row-fluid'>
            <ul class='breadcrumb'>
                <li>
                    {% for bc in breadcrumbs %}
                    {% if (bc['active']) %}
                    {{ bc['text'] }}
                    {% else %}
                    <a href='{{ bc['link'] }}'>{{ bc['text'] }}</a> 
                    <span class='divider'>/</span>
                    {% endif %}
                    {% endfor %}
                </li>
            </ul>
        </div>

        <?php echo $this->flash->output() ?>

        <div class="row-fluid">
            <?php echo $this->getContent() ?>
        </div> <!-- row -->

        {{ partial('partials/footer') }}
    </div>

    {{ javascript_include(config.app.js.jquery, config.app.js.local) }}
    {{ javascript_include(config.app.js.jquery_ui, config.app.js.local) }}
    {{ javascript_include(config.app.js.bootstrap, config.app.js.local) }}
    {{ javascript_include(config.app.js.angular, config.app.js.local) }}
    {{ javascript_include(config.app.js.angular_resource, config.app.js.local) }}
    {{ javascript_include(config.app.js.angular_ui, config.app.js.local) }}
    {{ javascript_include('js/utils.js') }}

    </body>
</html>

The above is the index.volt. As you can see I call on the partials/header.volt, then the partials/navbar.volt (where the menu is generated) and then I construct the breadcrumbs (note the {% for bc in breadcrumbs %} block). After that the flash messenger comes into play, the main content displayed, the footer and finally the javascript includes that I need. I am still using AngularJS to make the necessary AJAX calls so that the relevant controller to retrieve the data but also to display this data on screen (which is cached to avoid unnecessary database hits).

The Episodes view became

{{ content() }}

<div>
    <ul class='nav nav-tabs'>
        <li class='pull-right'>
            {{ addButton }}
        </li>
    </ul>
</div>

<div ng-controller='MainCtrl'>
<table class='table table-bordered table-striped ng-cloak' ng-cloak>
    <thead>
    <tr>
        <th><a href='' ng-click="predicate='number'; reverse=!reverse">#</a></th>
        <th><a href='' ng-click="predicate='air_date'; reverse=!reverse">Date</a></th>
        <th><a href='' ng-click="predicate='outcome'; reverse=!reverse">W/L</a></th>
        <th><a href='' ng-click="predicate='summary'; reverse=!reverse">Summary</a></th>
    </tr>
    </thead>
    <tbody>
        <tr ng-repeat="episode in data.results | orderBy:predicate:reverse">
            <td>[[episode.number]]</td>
            <td width='7%'>[[episode.air_date]]</td>
            <td>[[episode.outcome]]</td>
            <td>[[episode.summary]]</td>
            {% if (addButton) %}
            <td width='1%'><a href='/episodes/edit/[[episode.id]]'><i class='icon-pencil'></i></a></td>
            <td width='1%'><a href='/episodes/delete/[[episode.id]]'><i class='icon-remove'></i></a></td>
            {% endif %}
        </tr>
    </tbody>
</table>
</div>

The beauty of AngularJS! I only have to pass a JSON array with my results. ng-repeat with the orderBy filter allows me to present the data to the user and offer sorting capabilities per column. This is all done at the browser level without any database hits! Pretty awesome feature.

For those that have used AngularJS in the past, you will note that I had to change the interpolate provider (i.e. the characters that wrap a string or a piece of code that AngularJS understands). Usually these characters are the curly brackets {{ }} but I changed them to [[ ]] to avoid collisions with Volt. This was done with a couple of lines of code in my definition of my AngularJS model:


var ngModule = angular.module('HHF', ['ngResource', 'ui'])
    .config(function($interpolateProvider) {
        $interpolateProvider.startSymbol('[[');
        $interpolateProvider.endSymbol(']]');
    })

Conclusion

I spent at most a day working on this mostly because I wanted to try various things and see how it works. The actual time to convert the application (because let's face it, it is a small application) was a couple of hours inclusive of the time it took me to rename certain fields, restructure the folder structure, compile the new extension on my server and upload the data upstream.

I am very satisfied with both AngularJS which helps tremendously in my presentation layer as well as PhalconPHP Phalcon's new design makes implementation a breeze, while Volt offers a lot of flexibility on the view layer.

As written before, you are more than welcome to download the source code of this application here and use it for your own needs. Some resources are:

AngularJS main site: http://angularjs.org
AngularJS documentation: http://docs.angularjs.org/api
AngularJS group: https://groups.google.com/forum/#!forum/angular
AngularJS Github: https://github.com/angular

Phalcon PHP main site: http://phalconphp.com
Phalcon PHP documentation: http://phalconphp.com/documentation
Phalcon PHP group: https://groups.google.com/forum/#!forum/phalcon
Phalcon PHP Github: https://github.com/phalcon

AngularJS - Simplicity in your browser [AngularJS]

2012-09-02T00:00:00.000-04:00

Recently I was contacted by an acquaintance through my Google+ circles, who needed some help with a project of hers.

Her task was to redesign a church website. Pretty simple stuff, CSS, HTML and content.

Scope

The particular church videotapes all the sermons and posts them on their channel in LiveStream for their followers to watch. One of the requirements was to redo the video archives page and to offer a link where followers can download the audio of each sermon for listening.

Design (kinda)

After the initial contact, I decided to get rid of all the bloated jQuery code that was there to control the video player and use AngularJS to control the generation of content. There were two key facts that influenced my decision:

the use of ng-repeat to generate the table that will list all the available sermons and
the variable binding that AngularJS offers to play the video in the available player.

I also decided to switch the player to a new updated one that LiveStream offered, which features a slider to jump through the video, volume control and more.

Previous code

The previous code for that page was around 300 lines. The file had some CSS in it, quite a few lines of HTML but was heavy on javascript. There were a lot of jQuery functions which controlled the retrieval of the available videos per playlist. Each playlist would be effectively a collection of videos for a particular year. jQuery will observe clicks on specific links and make an AJAX call to the LiveStream API to retrieve the list of available data in JSON format, and output the formatted results (as a table) on screen. It was something like this:

head
title
link - css
style (inline)
....
end inline style
script jQuery
script jQueryUI
script jquery.timer
link jquery-ui CSS
link slider CSS
style (inline)
....
end inline style
script jQuery
$(document).ready ... // Document ready
.....
$("#div_col dd").click // Bind click to a year
.....
getPlaylists(year) // Not sure why this was ever here, not used
.....
getPlaylistClips(playlistID) // Gets the clips of the playlist
.....
playClip(clipID) // Plays the clip in the player
.....
end jQuery script
script Video Player
.....
end head
body
navigation
main container
list of years
instructions
container to show video player
container to show video list
end body
end html

Enter AngularJS

I checked the latest video player from LiveStream. The code was much cleaner and all I had to do is bind one variable, the GUID of the video, in the relevant call so that the video can be played. I also bound another variable (the video title) above the video so as to offer more information to the user.

With a bit of Bootstrap CSS, I created two tabs and listed the two years 2012, 2011. A function was created in my AngularJS module to accept the year and make the relevant call to the LiveStream API to receive the data a a JSON object.

ng-repeat (with ng-cloak) was used to "print" the data on screen and the application was ready.

I removed all the cruft and created one small file that is loaded and offers the functionality that we need. It is 50 lines of code (just the javascript part. The code is below with added comments for the reader to follow:

// Create the module and inject the Resource object
var ngModule = angular.module("CHF", ['ngResource']);

// The main controller that needs the scope and resource
ngModule.controller("MainCtrl", function ($scope, $resource) {

    // Calculates the current year
    //  ensures we always get the last year on first load
    $scope.currentYear = function () {
        var currentDate = new Date();
        return currentDate.getFullYear();
    };
    
    // This is the playlist array. This is obtained by 
    //  LiveStream and it changes once every year. 
    //  Hardly an effort by the administrator
    $scope.playlists = [
        {"year":"2012", "guid":"63426-xxx-xxx-xxx"},
        {"year":"2011", "guid":"84f84-xxx-xxx-xxx"}
    ];

    // This couldn't be simpler. It merely sets some variables 
    //  in the scope. By doing so, the binding in the relevant
    //  variables will allow the video to play and the title 
    //  to update.
    $scope.playVideo = function (element) {
        $scope.currentVideo  = element.guid;
        $scope.currentTitle  = element.title;
    };

    // This is the core. It makes the AJAX request to the 
    // LiveStream API so that it can get the JSON data back
    $scope.makeRequest = function (year) {
        
        // Calculating the current year and the year selected.
        // Their difference offers an offset which effectively 
        // is the offset of the array stored in $scope.playlists
        var thisYear    = $scope.currentYear();
        var diff        = thisYear - year;

        var objData = $scope.playlists[diff];

        // Just in case something was passed that is not valid
        if (objData.guid)
        {
            var reqData = $resource(
                "http://livestream_url/2.0/:action",
                {
                    action:'listclips.json', 
                    id:objData.guid,
                    query: {isArray: true},
                    maxresults:"500",
                    callback:'JSON_CALLBACK'
                },
                {get:{method:'JSONP'}}
            );

            // Set the year and get the data
            $scope.year    = year;
            $scope.listData = reqData.get();
       }
    };

    // This is the first load - load the current year
    $scope.makeRequest($scope.currentYear());

});

Now moving into the HTML side of things:

<div id="playerContainer" style='text-align:center;'>
    <p ng-cloak>
        {{currentTitle}}
    </p>
    <iframe 
        width="560" 
        height="340" 
        src="http://ls_url?clip={{currentVideo}}&amp;params" 
        style="border:0;outline:0" 
        frameborder="0" 
        scrolling="no">
    </iframe>
</div>

<br />
<div>
    <ul class="nav nav-tabs">
        <li ng-repeat="playlist in playlists" 
               ng-class="{true:'active',false:''}[year == playlist.year]">
            <a ng-click="makeRequest(playlist.year)">{{playlist.year}}</a>
        </li>
    </ul>
    <table class="table table-bordered" style="width: 100%;">
        <thead>
            <tr>
                <th>Date/Title</th>
                <th>Audio</th>
            </tr>
        </thead>
        <tbody>
            <tr ng-cloak ng-repeat="video in listData.channel.item">
                <td ng-click="playVideo(video)">{{video.title}}</td>
                <td>
                    <span ng-show="video.description">
                        <a href="{{video.description}}" title="Download Audio">
                            <i class="icon-download-alt"></i>
                        </a>
                    </span>
                </td>
            </tr>
        </tbody>
    </table>
</div>

That is all the HTML I had to change. The full HTML file is 100 lines and 50 for the AngularJS related javascript, I can safely say that I had a 50% reduction in code offering the same functionality - and if I might say so, it is much much cleaner.

The final page looks something like this:

A couple of pointers:

<div id="playerContainer" style='text-align:center;'>
    <p ng-cloak>
    {{currentTitle}}
    </p>
    <iframe 
        width="560" 
        height="340" 
        src="http://ls_url?clip={{currentVideo}}&amp;params" 
        style="border:0;outline:0" 
        frameborder="0" 
        scrolling="no">
    </iframe>
</div>

This block displays the video player and due to the variable binding that AngularJS offers, the minute those variables change, the video is ready to be played.

    <ul class="nav nav-tabs">
        <li ng-repeat="playlist in playlists" 
               ng-class="{true:'active',false:''}[year == playlist.year]">
            <a ng-click="makeRequest(playlist.year)">{{playlist.year}}</a>
        </li>
    </ul>

This block shows the tabs depicting each playlist. In our case these are years. ng-repeat does all the hard work, printing the data that is defined in our JS file. The ng-class is there to change the class of the tab to "active" when the tab is clicked/selected. The ng-click initiates a request through makeRequest, a function defined in our javascript file (see above).

        <tbody>
            <tr ng-cloak ng-repeat="video in listData.channel.item">
                <td ng-click="playVideo(video)">{{video.title}}</td>
                <td>
                    <span ng-show="video.description">
                        <a href="{{video.description}}" 
                           title="Download Audio">
                            <i class="icon-download-alt"></i>
                        </a>
                    </span>
                </td>
            </tr>
        </tbody>

Finally the data is displayed on screen. ng-cloak makes sure that the content is displayed only when the data is there (otherwise browsers might show something like {{video.description}} which is not nice from a UI perspective). ng-repeat loops through the data and "prints" the table.

The description of the video is used as a storage for the URL that will point to the MP3 audio file so as the users can download it. Therefore I use ng-show to show the link, if it exists.

Conclusion

This whole project was no more than 30 minutes, which included the time I had to research and experiment a bit with the LiveStream API. This is effectively a flexible design, with much much cleaner code (and a lot less of it). When the admin needs to add a new playlist (aka year), all they have to do is open the JS file and type a new element in the $scope.playlists array. The application will take care of the rest automatically.

I cannot think of doing this with less lines of code than this.

If you haven't heard of AngularJS or used it, I would highly encourage you to give it a shot. Great project, awesome support and a very very responsive, helpful and polite community.

Main Site: http://www.angularjs.org
Google Group: https://groups.google.com/forum/?fromgroups=#!forum/angular

Building a web app with PhalconPHP and AngularJS Part II [How-To][PhalconPHP][AngularJS]

2012-07-12T00:00:00.000-04:00

This is Part II of a series of posts on building an application using PhalconPHP and AngularJS. Part I is located here.

Preface

I have recently discovered PhalconPHP and I was impressed with its speed and ease of use. At the time of this writing, PhalconPHP is at version 0.4.2, with some serious redesign coming down the line on 0.5.x.

PhalconPHP takes a different approach than any other PHP framework (see Zend, Symfony, CakePHP etc.). It is written in C and compiled as a module which is then loaded on your web server. Effectively the whole framework is in memory for you to use, without needing to access the file system so that you can include a file here or a file there.

Advantages

The core advantage of this approach is speed. The framework is in memory, ready to deliver its functionality, so your application is now only concerned about its files and not the framework itself. Once a framework is mature enough for usage, its files don't change that much. Yet for any of the traditional frameworks, PHP needs to scan the files, load them and the interpret them. This in effect has a serious impact on performance, especially for large projects.

Another advantage is that since the framework is a module on your web server, you don't need to upload library files to each and every application you install on your host.

Finally, you can mix and match whatever you need, using any of the components as 'glue' components rather than the whole framework. Most of the major frameworks also use this methodology for most of their components, however performance still is an issue. Additionally, in the case of any other framework, one might need to upload a very complicated and deep file structure on their web server so as to take advantage of one component to be used in an application.

Disadvantages

Support and bug tracing are the two weaknesses of PhalconPHP. By support I do not mean support from the developers. On the contrary, the developers are doing a great job listening to the relatively young community, and issuing fixes. However, as with any framework, if you find a bug, you will try to trace the code back to each component in an effort to find a solution to your problem. When developing an application and have access to the source files (the library PHP files like Zend Framework has), not only you can learn from those implementations, but you can quickly fix something that might be broken and continue working. With PhalconPHP you will need to wait until the next version is released, unless you are fluent in C and play around with the source code. For most PHP programmers (like myself), the process will be report the bug and wait for the fix.

Since the framework is a module on your web server, you will need to be careful on upgrades. If your applications do not take advantage of the latest functionality the framework offers, you might fix something in one application, while breaking something in another. You cannot mix and match versions of PhalconPHP per application.

Consideration

PhalconPHP is very young as a framework. It does have a lot of power, but there are a lot of things still missing (for instance relationships between models and a query builder). In time these pieces will be implemented and the framework will grow stronger :)

Implementation

I downloaded the INVO sample application and set it up on my web browser. Using that as a starting point, I started modifying it to fit my needs. I also set up the PhalconPHP developer tools and PHPStorm support.

For this application, I needed a table to store information about every podcast episode, a table to store all players and a table to store the users (namely Aaron, Josh and John). The Awards table would be the one that would store all the information regarding the game balls and kick in the balls awards.

Models

Once those were in place I started building my models and relevant controllers/views. Setting a model up was really easy. I would create the table in my database and then run

phalcon create-model --table-name episodes

and my model would be ready for me to use (example below for Episodes).

class Episodes extends Phalcon_Model_Base {
    public $id;
    public $number;
    public $summary;
    public $airDate;
    public $outcome;
    public $createdAt;
    public $createdAtUserId;
    public $lastUpdate;
    public $lastUpdateUserId;
}

After a while I decided I wanted to keep a track on who created a record and when, and who last updated a record and when for certain tables. After some refactoring I created my own model class that would give me the functionality I needed, and extended that class in relevant models.

My custom class (that would take care of the createdAt, createdAtUserId, lastUpdated, lastUpdatedUserId fields) also took advantage of the beforeSave hook to ensure that these fields were transparently updated. The find and findFirst static functions are used throughout the models and there is no reason to repeat them in each model, so they end up in this custom class. (Comments removed to preserve space)

use NDN_Session as Session;

class NDN_Model extends Phalcon_Model_Base
{
    public $createdAt;
    public $createdAtUserId;
    public $lastUpdate;
    public $lastUpdateUserId;

    public function beforeSave()
    {
        if (empty($this->createdAtUserId)) {
            $auth     = Session::get('auth');
            $datetime = date('Y-m-d H:i:s');

            $this->createdAt        = $datetime;
            $this->createdAtUserId  = (int) $auth['id'];
        }
    }

    static public function find($parameters = array())
    {
        return parent::find($parameters);
    }

    static public function findFirst($parameters = array())
    {
        return parent::findFirst($parameters);
    }
}

Session

Although PhalconPHP provides a flash messenger utility, I had an issue with using the _forward function on a controller, after an action (say Add or Edit) was completed. Effectively the data would not refresh on screen. To combat that I used _redirect. However, all the messages that I had in the flash messenger (Phalcon_Flash) would disappear. An easy solution was to extend the Phalcon_Session and create two new functions setFlash and getFlash. The setFlash is called whenever I want to set a message for the user to see. The function stores the message in a session variable. Before the controller is dispatched, the getFlash is called to return any messages waiting to be displayed, and after that the messages are cleared from the session and displayed on screen.

class NDN_Session extends Phalcon_Session
{
    public static function setFlash($class, $message, $css)
    {
        $data = array(
            'class'   => $class,
            'message' => $message,
            'css'     => $css,
        );
        self::set('flash', $data);
    }

    public static function getFlash()
    {
        $data = self::get('flash');
        if (is_array($data)) {
            self::remove('flash');
            return $data;
        } else {
            return null;
        }
    }
}

Breadcrumbs

I wanted to show breadcrumbs to the user, as a way to easiliy navigate throughout the application. To do so, I created my own Breadcrumbs class which holds an array of areas that the user is in. Effectively it is the 'depth' the user is in throughout the application. The class has a generate function, which returns back a JSON string. This is to be parsed by AngularJS so as to display the breadcrumbs.

Controllers

I created my controllers using the PhalconPHP developer tools. Whether you use the webtools or the command line makes no difference. The skeleton of the controller is generated for you to use.

Based on the flash messenger and _redirect that I mentioned in the previous section, I had to extend the base controller, so as to add functionality that would allow me to show messages on screen after a redirect. Other reasons for this new class were to allow for a prefix on each page title, generate breadcrumbs and menus.

use Phalcon_Tag as Tag;

use Phalcon_Flash as Flash;
use NDN_Session as Session;

class NDN_Controller extends Phalcon_Controller 
{
    protected $_bc = null;
    public function initialize()
    {
        Tag::prependTitle('HHF G&KB Awards | ');
        $this->_bc = new NDN_Breadcrumbs();
    }

    public function beforeDispatch()
    {
        $message = Session::getFlash();
        if (is_array($message)) {
            Flash::$message['class'](
                $message['message'], $message['css']
            );
        }
        $this->view->setVar('breadcrumbs', $this->_bc->generate());
    }

    protected function _constructMenu($controller)
    {
        $commonMenu = array(
            'index'      => 'Home', 
            'awards'     => 'Awards', 
            'players'    => 'Players', 
            'episodes'   => 'Episodes', 
            'about'      => 'About', 
            'contact'    => 'Contact Us', 
        ); 
        $auth = Session::get('auth'); 

        $class  = get_class($controller); 
        $class  = str_replace('Controller', '', $class); 
        $active         = strtolower($class); 
        $sessionCaption = ($auth) ? 'Log Out'         : 'Log In'; 
        $sessionAction  = ($auth) ? '/session/logout' : '/session/index'; 

        $leftMenu = array(); 
        foreach ($commonMenu as $link => $text) { 
            $isActive   = (bool) ($active == $link); 
            $newLink  = ('index' == $link) ? '/' : '/' . $link; 
            $leftMenu[] = array( 
                'active' => $isActive, 
                'link'   => $newLink, 
                'text'   => $text, 
            ); 
        } 

        $menu = new StdClass(); 
        $menu->current = $active; 
        $menu->left    = $leftMenu; 

        if ($auth != false) { 
            $sessionCaption .= ' ' . $auth['name']; 
        } 

        $menu->rightLink = $sessionAction; 
        $menu->rightText = $sessionCaption; 

        return json_encode($menu); 
    } 
}

Each controller would extend my base controller. In the initialize function:

the page title is set,
the breadcrumbs are added (and generated later on in the beforeDispatch of the base controller),
the menu is generated and passed to the view for AngularJS to process,
additional variables would be generated for displaying elements based on whether a user is logged in or not.

Views

Creating the views was really easy. I already had the structure ready from the sample application (INVO) and with the help of Bootstrap CSS, I was done in no time. The views inherit from a base view (index.phtml) located at the root of the views folder. That view holds the skeleton of the web page and content is injected accordingly based on each controller (and its view).

In that file I added the relevant variables that will be used by AngularJS as well as variables that hold conditional elements (i.e. elements that appear when a user is logged in).

More on the views in the next installment of these series.

Conclusion

With all that the application was ready as far as the main structure was concerned. Tying everything with AngularJS was the next step, which will be covered in part III of this How-To.

The whole application, from start to finish, took less than 4 hours to develop. This included breaks, reading the manual and making design decisions based on my ever changing requirements.

Building a web app with PhalconPHP and AngularJS Part I [How-To][PhalconPHP][AngularJS]

2012-07-10T00:00:00.000-04:00

There are ample frameworks on the Internet, most free, that a programmer can use to build a web application. Two of these frameworks are PhalconPHP and AngularJS.

I decided to use those two frameworks and build a simple application which will keep track of the Game Balls and Kick in the Balls awards of Harry Hog Football.

Harry Hog Football is a podcast that has been going strong since 2005, created by Redskins fans for Redskins Fans. (for those that do not know, Washington Redskins is a team on the National Football League in the USA).

Every week during the regular season, Aaron, Josh and John create a podcast, where they discuss the recent game, the injuries, the cuts, the new signings and they offer their Game Balls to the best players of the week as well as the Kick in the Balls awards for the ones that (according to the podcasters) 'suck'.

I therefore created an application to record all those game balls and kick in the balls awards, so that we can all see, who is the most valuable player and who is the least valuable player for the Redskins throughout the years (the term valuable is used loosely here).

As a starting point I used the INVO application that PhalconPHP showcases as an easy application to get you started. I modified it significantly to address my needs, refactoring classes as much as possible to get the least amount of code with maximum usability.

After building the application, I listened to all the episodes I could find, and entered the game balls and kick in the balls in the database. The models use the Phalcon_Model_Base class to handle data, while the rest of the application is handled by the Phalcon_Controller (and view of course).

The data transfer between the application and the relevant sections is primarily handled by AngularJS, which is dominant in the view layer. AngularJS controllers handle menu creation, breadcrumbx as well as displaying results on screen.

Twitter's Bootstrap CSS is used to put the final touches for the application.

In subsequent posts I will explain each layer in turn, starting with PhalconPHP and continuing with AngularJS.

This of course is by no means the perfect implementation. It has been a fun project for me, working on it on my own free time. You are more than welcome to fork the project and make any modifications you need. For those that are interested in getting straight to the code, it is available on Github here.

NOTE: The Github repository contains code that works with nginx. If you are having problems with Apache, check the public/index.php - there is a note there for nginx (probably will need to remove it)

2012 4th of July Independence Day [Celebrations]

2012-07-04T14:52:00.001-04:00

Today we celebrate the Declaration of Independence in the United States, also known widely as the 4th of July.

During this day in 1776, delegates of 13 states signed a document which serves as the foundation of the Constitutional Republic, the United States of America.

Happy 4th of July!

The transcript is below

The Declaration of Independence

IN CONGRESS, July 4, 1776.

The unanimous Declaration of the thirteen united States of America,

When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.--That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed, --That whenever any Form of Government becomes destructive of these ends, it is the Right of the People to alter or to abolish it, and to institute new Government, laying its foundation on such principles and organizing its powers in such form, as to them shall seem most likely to effect their Safety and Happiness. Prudence, indeed, will dictate that Governments long established should not be changed for light and transient causes; and accordingly all experience hath shewn, that mankind are more disposed to suffer, while evils are sufferable, than to right themselves by abolishing the forms to which they are accustomed. But when a long train of abuses and usurpations, pursuing invariably the same Object evinces a design to reduce them under absolute Despotism, it is their right, it is their duty, to throw off such Government, and to provide new Guards for their future security.--Such has been the patient sufferance of these Colonies; and such is now the necessity which constrains them to alter their former Systems of Government. The history of the present King of Great Britain is a history of repeated injuries and usurpations, all having in direct object the establishment of an absolute Tyranny over these States. To prove this, let Facts be submitted to a candid world.

He has refused his Assent to Laws, the most wholesome and necessary for the public good.
He has forbidden his Governors to pass Laws of immediate and pressing importance, unless suspended in their operation till his Assent should be obtained; and when so suspended, he has utterly neglected to attend to them.
He has refused to pass other Laws for the accommodation of large districts of people, unless those people would relinquish the right of Representation in the Legislature, a right inestimable to them and formidable to tyrants only.
He has called together legislative bodies at places unusual, uncomfortable, and distant from the depository of their public Records, for the sole purpose of fatiguing them into compliance with his measures.
He has dissolved Representative Houses repeatedly, for opposing with manly firmness his invasions on the rights of the people.
He has refused for a long time, after such dissolutions, to cause others to be elected; whereby the Legislative powers, incapable of Annihilation, have returned to the People at large for their exercise; the State remaining in the mean time exposed to all the dangers of invasion from without, and convulsions within.
He has endeavoured to prevent the population of these States; for that purpose obstructing the Laws for Naturalization of Foreigners; refusing to pass others to encourage their migrations hither, and raising the conditions of new Appropriations of Lands.
He has obstructed the Administration of Justice, by refusing his Assent to Laws for establishing Judiciary powers.
He has made Judges dependent on his Will alone, for the tenure of their offices, and the amount and payment of their salaries.
He has erected a multitude of New Offices, and sent hither swarms of Officers to harrass our people, and eat out their substance.
He has kept among us, in times of peace, Standing Armies without the Consent of our legislatures.
He has affected to render the Military independent of and superior to the Civil power.
He has combined with others to subject us to a jurisdiction foreign to our constitution, and unacknowledged by our laws; giving his Assent to their Acts of pretended Legislation:
For Quartering large bodies of armed troops among us:
For protecting them, by a mock Trial, from punishment for any Murders which they should commit on the Inhabitants of these States:
For cutting off our Trade with all parts of the world:
For imposing Taxes on us without our Consent:
For depriving us in many cases, of the benefits of Trial by Jury:
For transporting us beyond Seas to be tried for pretended offences
For abolishing the free System of English Laws in a neighbouring Province, establishing therein an Arbitrary government, and enlarging its Boundaries so as to render it at once an example and fit instrument for introducing the same absolute rule into these Colonies:
For taking away our Charters, abolishing our most valuable Laws, and altering fundamentally the Forms of our Governments:
For suspending our own Legislatures, and declaring themselves invested with power to legislate for us in all cases whatsoever.
He has abdicated Government here, by declaring us out of his Protection and waging War against us.
He has plundered our seas, ravaged our Coasts, burnt our towns, and destroyed the lives of our people.
He is at this time transporting large Armies of foreign Mercenaries to compleat the works of death, desolation and tyranny, already begun with circumstances of Cruelty & perfidy scarcely paralleled in the most barbarous ages, and totally unworthy the Head of a civilized nation.
He has constrained our fellow Citizens taken Captive on the high Seas to bear Arms against their Country, to become the executioners of their friends and Brethren, or to fall themselves by their Hands.
He has excited domestic insurrections amongst us, and has endeavoured to bring on the inhabitants of our frontiers, the merciless Indian Savages, whose known rule of warfare, is an undistinguished destruction of all ages, sexes and conditions.

In every stage of these Oppressions We have Petitioned for Redress in the most humble terms: Our repeated Petitions have been answered only by repeated injury. A Prince whose character is thus marked by every act which may define a Tyrant, is unfit to be the ruler of a free people.

Nor have We been wanting in attentions to our Brittish brethren. We have warned them from time to time of attempts by their legislature to extend an unwarrantable jurisdiction over us. We have reminded them of the circumstances of our emigration and settlement here. We have appealed to their native justice and magnanimity, and we have conjured them by the ties of our common kindred to disavow these usurpations, which, would inevitably interrupt our connections and correspondence. They too have been deaf to the voice of justice and of consanguinity. We must, therefore, acquiesce in the necessity, which denounces our Separation, and hold them, as we hold the rest of mankind, Enemies in War, in Peace Friends.

We, therefore, the Representatives of the united States of America, in General Congress, Assembled, appealing to the Supreme Judge of the world for the rectitude of our intentions, do, in the Name, and by Authority of the good People of these Colonies, solemnly publish and declare, That these United Colonies are, and of Right ought to be Free and Independent States; that they are Absolved from all Allegiance to the British Crown, and that all political connection between them and the State of Great Britain, is and ought to be totally dissolved; and that as Free and Independent States, they have full Power to levy War, conclude Peace, contract Alliances, establish Commerce, and to do all other Acts and Things which Independent States may of right do. And for the support of this Declaration, with a firm reliance on the protection of divine Providence, we mutually pledge to each other our Lives, our Fortunes and our sacred Honor.

The 56 signatures on the Declaration appear in the positions indicated:

Georgia
Button Gwinnett
Lyman Hall George Walton

North Carolina
William Hooper
Joseph Hewes
John Penn

South Carolina
Edward Rutledge
Thomas Heyward, Jr.
Thomas Lynch, Jr.
Arthur Middleton

Massachusetts
John Hancock

Maryland
Samuel Chase
William Paca
Thomas Stone
Charles Carroll of Carrollton

Virginia
George Wythe
Richard Henry Lee
Thomas Jefferson
Benjamin Harrison
Thomas Nelson, Jr.
Francis Lightfoot Lee
Carter Braxton

Pennsylvania
Robert Morris
Benjamin Rush
Benjamin Franklin
John Morton
George Clymer
James Smith
George Taylor
James Wilson
George Ross

Delaware
Caesar Rodney
George Read
Thomas McKean

New York
William Floyd
Philip Livingston
Francis Lewis
Lewis Morris

New Jersey
Richard Stockton
John Witherspoon
Francis Hopkinson
John Hart
Abraham Clark

New Hampshire
Josiah Bartlett
William Whipple
Matthew Thornton

Massachusetts
Samuel Adams
John Adams
Robert Treat Paine
Elbridge Gerry

Rhode Island
Stephen Hopkins
William Ellery

Connecticut
Roger Sherman
Samuel Huntington
William Williams
Oliver Wolcott

How to build an inexpensive RAID for storage and streaming [How-To][Storage][NAS]

2012-06-29T00:00:00.000-04:00

Overview

I have written about this before and it is still my favorite mantra

There are people that take backups and there are people that never had a hard drive fail

This mantra is basically what should drive everyone to take regular backups of their electronic files, so as to ensure that nothing is lost in case of hardware failures.

Let's face it. Hard drives or any magnetic media are man made and will fail at some point. When is what we don't know, so we have to be prepared for it.

Services like Google Drive, iCloud, Dropbox, BoundlessCloud and others offer good backup services, ensuring that there is at least one 'safe' copy of your data. But that is not enough. You should ensure that whatever happens, the memories stored in your pictures or videos, the important email communications, the important documents are all kept in a safe place and there are multiple backups of it. Once they are gone, they are gone for good, so backups are the only way to ensure that this does not happen.

Background

My current setup at home consists of a few notebooks, a mac-mini and a Shuttle computer with a 1TB hard drive, where I store all my pictures, some movies and my songs. I use Google Music Manager for my songs so that they are available at any time on my android phone, Picasa to be able to share my photos with my family and friends and Google Drive so as to keep every folder I have in sync. I also use RocksBox to stream some of that content (especially the movies) upstairs on either of our TVs through the Roku boxes we have.

Recently I went downstairs and noticed that the Shuttle computer (which run Windows XP at the time) was stuck in the POST screen. I rebooted the machine but it refused to load Windows, getting stuck either in the computer's POST screen or in the "Starting Windows".

Something was wrong and my best guess was the hard drive was failing. I booted up then in Safe mode (and that worked), set the hard drive to be checked for defects and rebooted again to let the whole process finish. After an hour or so, the system was still checking the free space and was stuck at 1%. The hard drive was making some weird noises so I shut the system down.

Although I had backups of all my files on the cloud through Picasa, Google Music Manager and Google Drive, I still wanted to salvage whatever I had in there just in case. I therefore booted up the system with a Linux live CD, mounted the hard drive and used FileZilla to transfer all the data from the Shuttle's hard drive to another computer. There was of course a bit of juggling going on since I had to transfer data in multiple hard drives due to space restrictions.

Replacing the storage

I had to find something very cheap and practical. I therefore went to Staples and found a very nice (for the money) computer by Lenovo. It was only $300, offering 1TB 7200 SATA hard drive, i5 processor and 6GB of RAM.

As soon as I got the system I booted it up, started setting everything up and before the end of the day everything was back in place, synched to the cloud.

However the question remained: what happens if the hard drive fails again? Let's face it, I did lose a lot of time trying to set everything up again so I wasn't prepared to go through that again.

My solution was simple:

Purchase a cheap RAID (hardware) controller, an identical 1TB hard drive to the one I have, and a second hard drive to have the OS on. This way, the two 1TB hard drives can be connected to the RAID controller on a mirror setup (or RAID 1), while the additional hard drive can keep the operating system.

I opted for a solid state drive from Crucial for the OS. Although it was not necessary to have that kind of speed, I thought it wouldn't hurt. It did hurt my pocket a bit but c'est la vie. For your own setup you can choose whatever you like.

Hardware

HighPoint RocketRAID 620 2 SATA Port PCI-Express 2.0 x1 SATA 6Gb/s RAID Controller
($38.15) The RAID controller
Crucial 128 GB m4 2.5-Inch Solid State Drive SATA 6Gb/s CT128M4SSD2
($122.49) The OS drive
HITACHI 0F10383 1TB SATA 3.0Gb/s 3.0 7200 RPM 32MB Buffer Hard Drive
($106.99) RAID array drive (the only reason I bought this particular drive is because it is identical to the one that came with the Lenovo system)

NOTE: For those that are not interested in having a solid state drive for the OS, one can always go with other, much cheaper drives such as this one.

Setup

Picture 1: The two screws I had to create

After all the components arrived, I opened the computer and had a look at what I am facing with. One thing I did not realize was the lack of space for the third hard drive (the one that will hold the OS). I was under the impression that it would fit under the DVD ROM drive, but I did not account for the SD card reader that was installed in that space, so I had to be a bit creative (Picture 1).

A couple of good measurements and two holes with the power drill created a perfect mounting point for the solid state hard drive. It is sitting now secure in front of the card reader connections, without interfering in any way.

The HighPoint RocketRaid 620 in place

The second hard drive and the raid card were really easy to install, just a couple of screws and everything was set in place (Picture 2).

The second hard drive ended up in the only expansion 'bay' available for this system. This is below the existing drive, mounted on the left side of the case. The actual housing has guides that allow you to slide the drive until the screw holes are aligned and from there it is a two minute job to secure the drive in place.

I also had a generic nVidia 460 1GRAM card, which I also installed in the system. This was not included in the purchases for building this system, but it is around $45 if not less now. I have had it for over a year now and it was installed in the old Shuttle computer, so I wasn't going to let it go to waste.

With everything in place, all I had to do is boot the system up and enter the BIOS screen so as to ensure that the SSD drive had a higher priority than any other drive.

Hard drive #2 installed

Once that was done, I put the installation disks in the DVD-ROM and restored the system on the SSD drive. 4 DVDs later (around 30 minutes) the system was installed and booted up. It took however another couple of hours until I had everything set up. The myriad of Windows Updates, (plus my slow Internet connection) contributed to this delay. However I have to admit, that the SSD drive was a very good purchase, since I have never seen Windows boot in less than 10 seconds (from power up to the login screen).

The Windows updates included the nVidia driver so everything was set up (well almost that is). The only driver not installed was for the HighPoint RaidRocket RAID controller.

The installation disk provided that driver, alongside with a web based configuration tool. After installing the driver and a quick reboot, the RAID configuration tool was not easy to understand but I figured it out, even without reading the manual.

Entering the Disk Manager, I initialized and formatted the drive and from then on, I started copying all my filesin there.

As a last minute change, I decided not to install RocksBox and instead go with Plex Media Server. After playing around with Plex, I found out that it was a lot easier to setup than RocksBox (RocksBox requires a web server to be installed on the server machine, whereas Plex automatically discovers servers). Installing the relevant channel on my Roku boxes was really easy and everything was ready to work right out of the box so to speak.

Problems

The only problem that I encountered had to do with Lenovo itself. I wanted basically to install the system on the SSD drive. Since the main drive is 1TB and the SSD drive 128GB I could not use Clonezilla or Image for Windows to 'move' the system from one drive to another. I tried almost everything. I shrank the 1TB system partition so as to make it fit in the 128GB drive. I had to shut hibernation off, reboot a couple of times in Safe Mode to remove unmovable files, in short it was a huge time waster.

Since Lenovo did not include the installation disks (only an applications DVD), I called their support line and inquired about those disks. I was sent from the hardware department to the software department, where a gentleman informed me that I have to pay $65 to purchase the OS disks. You can imagine my frustration to the fact that I had already paid for the OS by purchasing the computer. We went back and forth with the technician and in the end got routed to a manager who told me I can create the disks myself using Lenovo's proprietary software.

The create rescue process required 10 DVDs, so I started the process. On DVD 7 the process halted. I repeated the process, only to see the same error on DVD 4. The following day I called Lenovo hardware support and managed to talk to a lady who was more than willing to send me the installation disks free of charge. Unfortunately right after I gave her my address, the line dropped, so I had to call again.

The second phone call did not go very well. I was transferred to the software department again, where I was told that I have to pay $65 for the disks. The odd thing is that the technician tried to convince me that Lenovo actually doesn't pay money to Microsoft since they get an OEM license. Knowing that this is not correct, and after the fact that the technician was getting really rude, I asked to speak to a supervisor. The supervisor was even worse and having already spent 45 minutes on the phone, I asked to be transferred to the hardware department again. Once there, I spoke to another lady, explained the situation and how long I have been trying to get this resolved (we are at 55 minutes now) and she happily took my information and sent me the installation disks free of charge.

Conclusion

The setup discussed in this post is an inexpensive and relatively secure way of storing data in your own home/home network. The RAID 1 configuration offers redundancy, while the price of the system does not break the bank.

I am very disappointed with Lenovo, trying to charge me for something I already paid for (the Operating System that is). Luckily the ladies at the hardware department were a lot more accommodating and I got what I wanted in the end.

I hope this guide helped you.

2012 Memorial Day [USA][Celebrations]

2012-05-28T08:40:00.001-04:00

Today is Memorial Day in the USA.

It is a special day, dedicated to celebrate, remember and honor the veterans that have gone and fought for all the liberties that we all enjoy today (and in most cases take for granted).

I would like to personally say a thank you - which is not enough for sure - to all those that ensured and continue to do so, that my family, friends and I are safe and free.

You might also enjoy material from the following resources:
This post that I wrote in 2010, the site http://www.usmemorialday.org or the following video which is one of my favorites:

Localization and Country Regions [Localization]

2012-03-05T14:24:00.000-05:00

I was recently assigned a task in my current job, to try and standardize address related data.

My approach was to use the ISO codes for countries and ISO codes for regions. A region is defined as the geographical split of areas in a country. For instance in the US regions are called states, in Canada provinces etc.

I have used several sources on the internet but my main source was Wikipedia.

In order to complete this task I used Zend_Locale to retrieve the available countries and using the ISO codes (keys of the returned data).

My method is:

/**
 * Gets the countries based on a passed (or default) locale
 * 
 * @param string  $locale
 * @param boolean  $sort
 */
public static function getCountries($locale = NULL, $sort = TRUE)
{
    $results = parent::getTranslationList('territory', $locale, 2);
    
    if ($sort) 
    {
        asort($results);
    }

    return $results;
}

Having the list of countries and their ISO codes, allowed me to have a proper storage of country related data in my addresses table. The use of a select box in the view layer ensures that I keep this standardization in place.

Extending that, I wanted to have an easy way to access the region related data, without having to hit the database. I therefore used JSON encoded files that contain an array with the type of the region as the key and Region ISO Code => Region Name as the array elements. Again I used a select box for that in the view layer, which is automatically refreshed when a new country is selected.

So for instance for the US the JSON file is as follows:

{
    "state":{
        "US-AL":"Alabama",
        "US-AK":"Alaska",
        ...
        "US-WV":"West Virginia",
        "US-WI":"Wisconsin",
        "US-WY":"Wyoming"
    }
}

where the key defines the region - in this case a State, and the array elements are the relevant data. In some instances I wanted to give a bit more selection and visual aids to the end user. I therefore have several sub arrays denoted by optiongroup tags for the relevant select box showing the regions.

In the case of Great Britain:

{
    "county":{
        "England":{
            "GB-BKM":"Buckinghamshire",
            "GB-CAM":"Cambridgeshire",
            ...
            "GB-WOK":"Wokingham",
            "GB-YOR":"York"
        },
        "Northern Ireland":{
            "GB-ANT":"Antrim",
            "GB-ARD":"Ards",
            ...
            "GB-OMH":"Omagh",
            "GB-STB":"Strabane"
        },
        "Scotland":{
            "GB-ABE":"Aberdeen City",
            "GB-ABD":"Aberdeenshire",
            ...
            "GB-WDU":"West Dunbartonshire",
            "GB-WLN":"West Lothian"
        },
        "Wales":{
            "GB-BGW":"Blaenau Gwent",
            "GB-BGE":"Bridgend (Pen-y-bont ar Ogwr)",
            ...
            "GB-VGL":"Vale of Glamorgan (Bro Morgannwg)",
            "GB-WRX":"Wrexham (Wrecsam)"
        }
    }
}

So with simple commands as file_get_contents and json_decode (or using Zend_Json::decode) I was able to have a flexible system that would load standardized address related data (regions and countries).

You can download the list of JSON files (one per country) from my GitHub.

The list I have there is by no means complete, but it does have a lot of data that one can use. Additions are always welcome :)

Enjoy!

Downgrading PHPUnit from 3.6 to 3.5 [PHPUnit][Linux][HowTo]

2012-01-15T15:44:00.000-05:00

Recently I had to rebuild my computer, and decided to install Linux Mint 12 (Lisa), which is a very lean installation - for my taste that is.

Going through the whole process of reinstalling all the packages that I need or had, PHPUnit was one of them. Easy enough a couple commands did the trick

sudo apt-get install php-pear
sudo pear upgrade PEAR
sudo pear config-set auto_discover 1
sudo pear install pear.phpunit.de/PHPUnit

I wanted to run my tests after that, only to find an error in the execution:

PHP Fatal error:  Call to undefined method PHPUnit_Util_Filter::addFileToFilter()
 in /home/www/project/library/PHPUnit/Framework.php on line 48

At first I thought that it was a path error, so I included the /usr/share/php/PHPUnit and others in the php.ini file but with no luck. With a little bit of Googling I found out that there have been some changes in the 3.6 version of PHPUnit and things don't work as they did before.
Effectively, 3.6 had some refactoring done and thus the line:

PHPUnit_Util_Filter::addDirectoryToFilter("$dir/tests");

changed to

PHP_CodeCoverage_Filter::getInstance()
        ->addDirectoryToBlacklist("$dir/tests");

Since I didn't want to change my whole test suite, I had to find a solution i.e. downgrade PHPUnit to 3.5.

Unfortunately specifying the version directly did not wok

sudo pear install phpunit/PHPUnit-3.5.15

since it would pull the latest version again and I would end up with the 3.6 files.

So I went one step further and installed specific versions of the relevant dependencies to satisfy the 3.5.15 version.

Uninstallation of 3.6

pear uninstall phpunit/PHPUnit_Selenium
pear uninstall phpunit/DbUnit
pear uninstall phpunit/PHPUnit
pear uninstall phpunit/PHP_CodeCoverage
pear uninstall phpunit/PHP_Iterator
pear uninstall phpunit/PHPUnit_MockObject
pear uninstall phpunit/Text_Template
pear uninstall phpunit/PHP_Invoker
pear uninstall phpunit/PHP_Timer
pear uninstall phpunit/File_Iterator
pear uninstall pear.symfony-project.com/YAML

Installation of 3.5.15

pear install pear.symfony-project.com/YAML-1.0.2
pear install phpunit/PHPUnit_Selenium-1.0.1
pear install phpunit/PHP_Timer-1.0.0
pear install phpunit/Text_Template-1.0.0
pear install phpunit/PHPUnit_MockObject-1.0.3
pear install phpunit/File_Iterator-1.2.3
pear install phpunit/PHP_CodeCoverage-1.0.2
pear install phpunit/DbUnit-1.0.0
pear install phpunit/PHPUnit-3.5.15

I hope you find the above useful :)

Change the encoding of a MySQL database to UTF8 [How-To][PHP][MySQL]

2011-12-19T22:00:00.000-05:00

Overview
As applications grow, so do their audiences. In this day and age, one cannot assume that all the consumers of a web based application will live in a particular region and use only one language. Even if the developer assumes that one country will be served by the particular web application, there are instances that the latin1 character set will not suffice in storing data.

Therefore, developers and database designers need to implement an encoding on their database that will safely store and retrieve any kind of data, not only latin1 based (i.e. the English alphabet).

For MySQL this encoding is utf8_general_ci.

The problem
MySQL usually comes with the latin1_swedish_ci encoding as a default. This encoding will allow the developer to store data of course but when non latin1 characters need to be stored, there will be a problem. Effectively latin1 encoding will store data in 8 bits but some languages like Japanese, Thai, Arabic, even French or German have special characters that need more space in the storage engine. Trying to store a 16 bit character in a 8 bit space will fail all the time.

Latin1 based database:
Input: abcdef...ABCD...#$
Output: abcdef...ABCD...#$

Input: 日本語 ภาษาไทย Ελληνικά
Output: ??? ??????? ????????

To combat this, all you have to do is change the encoding of your database to utf8_general_ci and the character set to utf8.

The solution
I wrote a script in PHP to analyze a database server and produce ALTER statements to be executed against your database(s). The script needs to run from a web server that supports PHP.

First of all, the encoding of the database will change with the relevant SQL statement. Following that each table's encoding will change, again with the relevant SQL statement. Finally, each TEXT/VARCHAR/CHAR etc. field's encoding will change towards the target encoding you specify in the configuration section (see below).

The safest way to transform data this way is to first change the field to a BINARY field and then change the field to the target encoding and collation.

Configuration
There are a few configuration variables that need to be set prior to running the script.

$db_user       = 'username';
$db_password   = 'password';
$db_host       = 'hostname';
$output_folder = '/home/ndimopoulos'; // Do not include trailing slash
$db_name       = 'mysql';             // Leave this one as is

set_time_limit(0);

/**
 * The old collation (what needs to be changed)
 */
$from_encoding = 'latin1_swedish_ci';

/**
 * The new collation (what we will change it to)
 */
$to_encoding = 'utf8_general_ci';

/**
 * The new character set
 */
$new_collation = 'utf8';

/**
 * Add USE <database>; before each statement?
 */
$use_database = TRUE;

The $output_folder is a folder that is writeable from your web server and it is where the .sql files will be created filled with the ALTER statements. The script will output one file <hostname>.sql which will contain all the ALTER statements for all databases. It will also create files for individual databases <hostname>.<database>.sql. You can use either the big file or the individual database files. The choice is yours.

The $from_encoding is what the script will check. In my script I was checking 'latin1_swedish_ci'.
The $to_encoding is what we need the encoding to be while the $new_collation is the new character set.

The $use_database is a flag that will allow you to generate statements such as:

USE <database>; ALTER TABLE <table>.....

if it is on, and if off, the statement will be:

ALTER TABLE <table>.....

Databases loop
The script opens a connection to the server and runs the 'SHOW DATABASES' command. Based on the result returned, it populates an array with the database names.

The script ignores two databases 'information_schema' and 'mysql', but editing the $exclude_databases array will allow you to ignore more databases if you need to.

mysql_connect($db_host, $db_user, $db_password);
mysql_select_db($db_name);

$dbs = array();

exclude_databases = array('mysql', 'information_schema',);

/**
 * Get the databases available (ignore information_schema and mysql)
 */
$result = mysql_query("SHOW DATABASES");

while ($row = mysql_fetch_row($result))
{
    if (!in_array($row[0], $exclude_databases))
    {
        $dbs[] = $row[0];
    }
}

The database names are stored in an array, so as not to keep the database resource active all the time. Had I not done that, I would have had to use three different resources (one for the database, one for the table and one for the field being checked - three nested loops).

Tables loop
The script then loops through the $dbs array and selects each database in turn. Once the database is selected, the 'SHOW TABLES' query is run and a $tables array is populated with the names of the tables in that database. At the same time the ALTER DATABASE statements are being generated.

mysql_select_db($db);

$db_output = '';

$statement  = "\r\n#-------------------------------------------------\r\n\r\n";
$statement .= "USE $db;\r\n";
$statement .= "\r\n#-------------------------------------------------\r\n\r\n";
$statement .= "ALTER DATABASE $db "
           . "CHARACTER SET $new_collation COLLATE $to_encoding;\r\n";
$statement .= "\r\n#-------------------------------------------------\r\n\r\n";

$db_output .= $statement;
$output    .= $statement;

$tables     = array();

$result = mysql_query("SHOW TABLES");

while ($row = mysql_fetch_row($result))
{
    if (!in_array($row[0], $exclude_tables))
    {
        $tables[] = mysql_real_escape_string($row[0]);
    }
}

Fields loop
The script then loops through the $tables array and runs the 'SHOW FIELDS' query so as to analyze each field.

$fields_modify = array();
$fields_change = array();

$result = mysql_query("SHOW FULL FIELDS FROM `$table`");
while ($row = mysql_fetch_assoc($result)) 
{
    if ($row['Collation'] != $from_encoding)
    {
        continue;
    }
   
    // Is the field allowed to be null?
    $nullable = ($row['Null'] == 'YES') ? ' NULL ' : ' NOT NULL';

    if ($row['Default'] == 'NULL') 
    {
       $default = " DEFAULT NULL";
    } 
    else if ($row['Default']!='') 
    {
       $default = " DEFAULT '" . mysql_real_escape_string($row['Default']) . "'";
    } 
    else 
    {
       $default = '';
    }

    // Alter field collation:
    $field_name = mysql_real_escape_string($row['Field']);

    $fields_modify[] = "MODIFY `$field_name` $row[Type] CHARACTER SET BINARY";
    $fields_change[] = "CHANGE `$field_name` `$field_name` $row[Type] "
                     . "CHARACTER SET $new_collation "
                     . "COLLATE $to_encoding $nullable $default";
}

The two arrays generated ($fields_modify and $fields_change contain the MODIFY and CHANGE statements of each field. Using implode, we can easily construct the ALTER statement.

$statement .= "ALTER TABLE `$table` " 
            . implode(' , ', $fields_modify) . "; \r\n";
$statement .= "ALTER TABLE `$table` " 
            . implode(' , ', $fields_change) . "; \r\n";

Notes
You can use as mentioned earlier the $exclude_databases array to not allow certain databases to be processed. You can also use the $exclude_tables array to not allow certain tables to be processed.

The $exclude_tables_fields array allows you to exclude a field from being processed. However this is not tied to a database so any database/table that has a field with that particular name will not be processed. With a bit of refactoring you can make the script best work for your needs.

If you set the $use_database variable to TRUE then each line in your .sql statements will be prefixed with a 'USE <database>;' statement. This is to help the accompanying bash script to execute each statement in the respective database. If you intend on not running this process one statement at a time, you can set this to FALSE. You can then run each database .sql file (or the one that contains all of the statements from all databases) as one single command.

Server processing
Now that the relevant .sql files have been created, all you have to do is upload them on your web server. There are three ways of actually running the statements against the database.

Please make sure you backup your data first!

Single file processing

mysql -h<host> -u<username> -p<password> < /path/to/scripts/<host>.sql

The command above will run all the commands generated in the <host>.sql file for all databases. It is going to be taxing for your database server and there is no error handling or reporting. You can always pipe the results to an output file (just append "> /path/to/output/output.txt" at the end of the command). If this method fails for some reason (MySQL has gone away), it will be difficult to resume; you will need to edit the <host>.sql file to remove the statements that have already been processed.

Per database

mysql -h<host> -u<username> -p<password> <database_name> \
       </path/to/scripts/<host>.<database>.sql

The command above will run all the commands generated in the <host>.<database>.sql file for that particular database. This method is similar to the one above. Again you can always pipe the results to an output file (just append "> /path/to/output/output.txt" at the end of the command).

Single file processing (per statement)

/path/to/scripts/process.sh

All you need to do is edit the process.sh script and change the relevant parameters to match your environment and upload it to your server. The source file that the process.sh script will read has to be generated with $use_database set to TRUE.

The process.sh script is:

#!/bin/bash

DBUSER=root
DBPASS=1234
DBHOST=localhost
SOURCE="/home/ndimopoulos/host.sql"
LOG="/home/ndimopoulos/conversion.log"

while read line
do
    TIMENOW=`date +%Y-%m-%d-%H-%M`
    echo START $TIMENOW $line
    echo START $TIMENOW $line >> $LOG
    /usr/bin/time -f "%E real,%U user,%S sys" -v -o $LOG -a \
        mysql -h$DBHOST -u$DBUSER -p$DBPASS -e "$line"
    
    TIMENOW=`date +%Y-%m-%d-%H-%M`
    echo END $TIMENOW 
    echo END $TIMENOW >> $LOG

done < $SOURCE

exit 0

The script above will start reading the source file (<host>.sql) and execute each statement in turn, using time to measure the time taken to execute that command. The output ends up in a log file which can easily be tailed to view the progress and used later on for analysis. The results of the processing are also sent to the screen. You can change the parameters for the time command to match your needs.

The output will look something like the block below:

START 2011-12-08-23-46 USE mydatabase; \
    ALTER TABLE `tablename` DEFAULT CHARACTER SET utf8;
 Command being timed: "mysql -uroot -p1234 -e USE mydatabase; \
     ALTER TABLE `tablename` DEFAULT CHARACTER SET utf8;"
 User time (seconds): 0.01
 System time (seconds): 0.00
 Percent of CPU this job got: 0%
 Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.16
 Average shared text size (kbytes): 0
 Average unshared data size (kbytes): 0
 Average stack size (kbytes): 0
 Average total size (kbytes): 0
 Maximum resident set size (kbytes): 8192
 Average resident set size (kbytes): 0
 Major (requiring I/O) page faults: 0
 Minor (reclaiming a frame) page faults: 610
 Voluntary context switches: 11
 Involuntary context switches: 5
 Swaps: 0
 File system inputs: 0
 File system outputs: 0
 Socket messages sent: 0
 Socket messages received: 0
 Signals delivered: 0
 Page size (bytes): 4096
 Exit status: 0
END 2011-12-08-23-46

Conclusion
In order for a database to be best prepared to support localization, you need to make sure that the storage will accept any possible character. You can start by creating all your tables and fields with utf8_general_ci encoding, but for existing databases and data, you will need to run expensive processing queries on your RDBMS. Ensuring that the data does not get corrupted when performing the transformation process is essential so make sure you backup your databases before trying or running the output statements produced by the db_alter.php script.

PHP script (db_alter.php)

$db_user       = 'username';
$db_password   = 'password';
$db_host       = 'hostname';
$output_folder = '/home/ndimopoulos'; // Do not include trailing slash
$db_name       = 'mysql'; // Leave this one as is

set_time_limit(0);

/**
 * The old collation (what needs to be changed)
 */
$from_encoding = 'latin1_swedish_ci';

/**
 * The new collation (what we will change it to)
 */
$to_encoding = 'utf8_general_ci';

/**
 * The new character set
 */
$new_collation = 'utf8';

/**
 * Add USE <database> before each statement?
 */
$use_database = TRUE;

mysql_connect($db_host, $db_user, $db_password);
mysql_select_db($db_name);

$dbs = array();

$exclude_databases     = array('mysql', 'information_schema',);
$exclude_tables        = array('logs', 'logs_archived',);
$exclude_tables_fields = array('activities');

/**
 * Get the databases available (ignore information_schema and mysql)
 */
$result = mysql_query("SHOW DATABASES");

while ($row = mysql_fetch_row($result)) 
{
    if (!in_array($row[0], $exclude_databases))
    {
        $dbs[] = $row[0];
    }
}

$output = '';
/**
 * Now select each db and start parsing the tables
 */
foreach ($dbs as $db)
{
    mysql_select_db($db);
    $db_output = '';
    
    $statement  = "\r\n#----------------------------------------\r\n\r\n";
    $statement .= "USE $db;\r\n";
    $statement .= "\r\n#----------------------------------------\r\n\r\n";
    $statement .= "ALTER DATABASE $db "
               . "CHARACTER SET $new_collation COLLATE $to_encoding;\r\n";
    $statement .= "\r\n#----------------------------------------\r\n\r\n";
    
    $db_output .= $statement;
    $output    .= $statement;
    $tables     = array();
    
    $result = mysql_query("SHOW TABLES");
    
    while ($row = mysql_fetch_row($result))
    {
        if (!in_array($row[0], $exclude_tables))
        {
            $tables[] = mysql_real_escape_string($row[0]);
        }
    }
    
    /**
     * Alter statements for the tables
     */
    foreach ($tables as $table)
    {
        $statement = '';
        if ($use_database)
        {
            $statement  = "USE $db; ";
        }
        $statement .= "ALTER TABLE `$table` "
                   . "DEFAULT CHARACTER SET $new_collation;\r\n";
        $db_output .= $statement;
        $output    .= $db_output;
    }
    $statement .= "\r\n#----------------------------------------\r\n\r\n";

    $db_output .= $statement;
    $output    .= $statement;
    
    /**
     * Get the fields for each table
     */
    foreach ($tables as $table)
    {
        if (in_array($table, $exclude_tables_fields))
        {
            continue;
        } 
        
        $fields_modify = array();
        $fields_change = array();
        
        $result = mysql_query("SHOW FULL FIELDS FROM `$table`");
        while ($row = mysql_fetch_assoc($result)) 
        {
            if ($row['Collation'] != $from_encoding)
            {
                continue;
            }
            
            // Is the field allowed to be null?
            $nullable = ($row['Null'] == 'YES') ? ' NULL ' : ' NOT NULL';
            
            if ($row['Default'] == 'NULL') 
            {
                $default = " DEFAULT NULL";
            } 
            else if ($row['Default']!='') 
            {
                $default = " DEFAULT '"
                         . mysql_real_escape_string($row['Default']) . "'";
            }
            else 
            {
                $default = '';
            }
            
            // Alter field collation:
            $field_name = mysql_real_escape_string($row['Field']);
            
            $fields_modify[] = "MODIFY `$field_name` $row['Type'] "
                             . "CHARACTER SET BINARY";
            $fields_change[] = "CHANGE `$field_name` `$field_name` $row['Type'] "
                             . "CHARACTER SET $new_collation "
                             . "COLLATE $to_encoding $nullable $default";
        }
        
        if (count($fields_modify) > 0)
        {
            $statement = '';
            if ($use_database)
            {
                $statement = "USE $db; ";
            }
            $statement .= "ALTER TABLE `$table` "
                        . implode(' , ', $fields_modify) . "; \r\n";
            if ($use_database)
            {
                $statement = "USE $db; ";
            }
            $statement .= "ALTER TABLE `$table` "
                        . implode(' , ', $fields_change) . "; \r\n";
            
            $db_output .= $statement;
            $output    .= $statement;
        }
    }
    
    $bytes = file_put_contents(
        $output_folder . '/' . $db_host . '.' . $db . '.sql', $db_output
    );
}
    
$bytes = file_put_contents($output_folder . '/' . $db_host . '.sql', $output);

echo "<pre>$db_host $bytes \r\n$output</pre>";

Downloads
You can use these scripts at your own risk. Also feel free to distribute them freely - a mention would be nice. Both scripts can be found in my GitHub.

Fast serialization of data in PHP [How-To][Performance]

2011-11-22T17:34:00.001-05:00

Serializing/Unserializing data

Serialization is the process of converting a data structure or object state into a format that can be stored and "resurrected" later in the same or another computer environment. source

There are a lot of areas where one can use serialization. A couple are:

in a database (storing an array of options specific to the user),
in an AJAX enabled application (call to get a status update and display to the user without refreshing the whole page), etc.

Based on the the application, serializing and unserializing data can be a very intensive process and can prove to have a big performance hit on the overall system.

Options
The most obvious option for serializing and unserializing data are the serialize and unserialize PHP functions. A bit less popular are json_encode and json_decode. There is also a third option, using a third party module that one can easily install on their server. This module is called igbinary.

In this blog post I am comparing the three options, in the hope that it will aid you with your selection of the best option for you so as to increase the performance of your application.

I created a test script that used several arrays of data (strings, integers, floats, booleans, objects, mixed data, all of the data types) to test the speed and size of the serialization and speed of unserialization of each of the three candidate function pairs. I run the same function to serialize or unserialize the data respectively for 1,000,000 times so as to produce the results below.

The script that I have used is listed below:

$_testStrings = array(
    'AK' => 'Alaska', 'AZ' => 'Arizona', 'VT' => 'Vermont',
    'VA' => 'Virginia', 'AZ' => 'West Virginia',
);
 
$_testIntegers = array(0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 84, 144,);

$_testBooleans = array(TRUE, TRUE, FALSE, FALSE, FALSE, TRUE, TRUE,);
 
$_testFloats = array(
    0, 1.1, 1.1, 2.22, 3.33, 5.55, 8.88, 13.13, 21.2121, 34.3434, 
    55.5555, 84.8484, 144.144,
);
 
$_testMixed = array(
    'one', 13 => 'two', 0 => 25.46, 'four' => 0.007, 
    'five' => TRUE, TRUE => 42,
);
 
$_objectOne            = new stdClass();
$_objectOne->firstname  = 'Leroy';
$_objectOne->lastname   = 'Jenkins';
$_objectOne->profession = 'Gamer';
$_objectOne->status     = 'Legend';
 
$_objectTwo        = new stdClass();
$_objectTwo->series = 'Fibonacci';
$_objectTwo->data   = $_testIntegers;
 
$_testObjects = array($_objectOne, $_objectTwo,);

$_maxLoop = 1000000;

$_templateEncode = "%s [%s]: Size: %s bytes, %s time to encode\r\n";
$_templateDecode = "%s [%s]: %s time to decode\r\n";

set_time_limit(0);

$_output = '';

/**
 * Set the source arrays
 */
$_allTestData = array(
    'str' => $_testStrings,
    'int' => $_testIntegers,
    'bln' => $_testBooleans,
    'flt' => $_testFloats,
    'mix' => $_testMixed,
    'obj' => $_testObjects,
);

$_testSources = array(
    'strings'  => $_testStrings,
    'integers' => $_testIntegers,
    'booleans' => $_testBooleans,
    'floats'   => $_testFloats,
    'mixed'    => $_testMixed,
    'objects'  => $_testObjects,
    'all'      => $_allTestData,
);
 
/**
 * ENCODE DATA
 */

/**
 * Start each test
 */
foreach ($_testSources as $_area => $_source)
{
    /**
     * Start the timer
     */
    $_serializeStart = microtime(TRUE);

    for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
    {
        serialize($_source);
    }

    $_serializeEnd = microtime(TRUE);

    $_serializeOutput = serialize($_source);

    $_output .= sprintf(
        $_templateEncode,
        'serialize()', 
        $_area, 
        strlen($_serializeOutput), 
        $_serializeEnd - $_serializeStart
    );
 
    /**
     * JSON
     */
    $_jsonStart = microtime(TRUE);

    for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
    {
        json_encode($_source);
    }

    $_jsonEnd = microtime(TRUE);

    $_jsonOutput = json_encode($_source);

    $_output .= sprintf(
        $_templateEncode,
        'json_encode()', 
        $_area, 
        strlen($_jsonOutput), 
        $_jsonEnd - $_jsonStart
    );
 
    /**
     * igbinary
     */
    $_igbinaryStart = microtime(TRUE);

    for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
    {
        igbinary_serialize($_source);
    }

    $_igbinaryEnd = microtime(TRUE);

    $_igbinaryOutput = igbinary_serialize($_source);

    $_output .= sprintf(
        $_templateEncode,
        'igbinary_serialize()', 
        $_area, 
        strlen($_igbinaryOutput), 
        $_igbinaryEnd - $_igbinaryStart
    );

    $_output .= str_repeat('=', 20) . "\r\n";
}

$_output .= str_repeat('=:=', 20) . "\r\n";


/**
 * DECODE DATA
 */

/**
 * Start each test
 */
foreach ($_testSources as $_area => $_source)
{
    /**
     * Start the timer
     */
    $_data = serialize($_source);

    $_serializeStart = microtime(TRUE);

    for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
    {
        unserialize($_data);
    }

    $_serializeEnd = microtime(TRUE);

    $_output .= sprintf(
        $_templateDecode,
        'unserialize()', 
        $_area, 
        $_serializeEnd - $_serializeStart
    );

    /**
     * JSON
     */
    $_data = json_encode($_source);

    $_jsonStart = microtime(TRUE);

    for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
    {
        json_decode($_data, TRUE);
    }

    $_jsonEnd = microtime(TRUE);

    $_jsonOutput = json_encode($_source);

    $_output .= sprintf(
        $_templateDecode,
        'json_decode()', 
        $_area, 
        $_jsonEnd - $_jsonStart
    );

    /**
     * igbinary
     */
    $_data = igbinary_serialize($_source);

    $_igbinaryStart = microtime(TRUE);

    for ($_counter = 0; $_counter < $_maxLoop; $_counter++)
    {
        igbinary_unserialize($_data);
    }

    $_igbinaryEnd = microtime(TRUE);

    $_igbinaryOutput = igbinary_serialize($_source);

    $_output .= sprintf(
        $_templateDecode,
        'igbinary_unserialize()', 
        $_area, 
        $_igbinaryEnd - $_igbinaryStart
    );

    $_output .= str_repeat('=', 20) . "\r\n";
}
 
echo '<pre>' . $_output . '</pre>';

Serializing results
When serializing data we are always concerned about the size of the result but also about the time it took for the data to be serialized.

As far as size is concerned, I have highlighted in bold the winner of each test. json_encode seems to be producing the smallest result in bytes for most of the tests.

Size comparison

Strings

serialize() [strings]: Size: 105 bytes, 1.8710339069366 time to encode
json_encode() [strings]: Size: 67 bytes, 1.5691390037537 time to encode
igbinary_serialize() [strings]: Size: 64 bytes, 3.2276048660278 time to encode

Integers

serialize() [integers]: Size: 121 bytes, 3.0198090076447 time to encode
json_encode() [integers]: Size: 34 bytes, 1.2248229980469 time to encode
igbinary_serialize() [integers]: Size: 58 bytes, 2.2877519130707 time to encode

Booleans

serialize() [booleans]: Size: 62 bytes, 2.0834550857544 time to encode
json_encode() [booleans]: Size: 39 bytes, 1.0889070034027 time to encode
igbinary_serialize() [booleans]: Size: 27 bytes, 1.8252439498901 time to encode

Floats

serialize() [floats]: Size: 709 bytes, 27.496570825577 time to encode
json_encode() [floats]: Size: 77 bytes, 5.0476500988007 time to encode
igbinary_serialize() [floats]: Size: 142 bytes, 2.4856028556824 time to encode

Mixed

serialize() [mixed]: Size: 178 bytes, 6.301619052887 time to encode
json_encode() [mixed]: Size: 54 bytes, 2.0463008880615 time to encode
igbinary_serialize() [mixed]: Size: 50 bytes, 2.3894169330597 time to encode

Objects

serialize() [objects]: Size: 326 bytes, 4.8698291778564 time to encode
json_encode() [objects]: Size: 148 bytes, 2.4744520187378 time to encode
igbinary_serialize() [objects]: Size: 177 bytes, 6.472992181778 time to encode

All data types

serialize() [all]: Size: 1567 bytes, 42.437592029572 time to encode
json_encode() [all]: Size: 462 bytes, 9.9569129943848 time to encode
igbinary_serialize() [all]: Size: 478 bytes, 18.053789138794 time to encode

Speed comparison

Analyzing the time it took for each test to be completed, we see again that json_encode is the clear winner (highlighted in bold the shortest time for the function).

Strings

serialize() [strings]: Size: 105 bytes, 1.8710339069366 time to encode
json_encode() [strings]: Size: 67 bytes, 1.5691390037537 time to encode
igbinary_serialize() [strings]: Size: 64 bytes, 3.2276048660278 time to encode

Integers

serialize() [integers]: Size: 121 bytes, 3.0198090076447 time to encode
json_encode() [integers]: Size: 34 bytes, 1.2248229980469 time to encode
igbinary_serialize() [integers]: Size: 58 bytes, 2.2877519130707 time to encode

Booleans

serialize() [booleans]: Size: 62 bytes, 2.0834550857544 time to encode
json_encode() [booleans]: Size: 39 bytes, 1.0889070034027 time to encode
igbinary_serialize() [booleans]: Size: 27 bytes, 1.8252439498901 time to encode

Floats

serialize() [floats]: Size: 709 bytes, 27.496570825577 time to encode
json_encode() [floats]: Size: 77 bytes, 5.0476500988007 time to encode
igbinary_serialize() [floats]: Size: 142 bytes, 2.4856028556824 time to encode

Mixed

serialize() [mixed]: Size: 178 bytes, 6.301619052887 time to encode
json_encode() [mixed]: Size: 54 bytes, 2.0463008880615 time to encode
igbinary_serialize() [mixed]: Size: 50 bytes, 2.3894169330597 time to encode

Objects

serialize() [objects]: Size: 326 bytes, 4.8698291778564 time to encode
json_encode() [objects]: Size: 148 bytes, 2.4744520187378 time to encode
igbinary_serialize() [objects]: Size: 177 bytes, 6.472992181778 time to encode

All data types

serialize() [all]: Size: 1567 bytes, 42.437592029572 time to encode
json_encode() [all]: Size: 462 bytes, 9.9569129943848 time to encode
igbinary_serialize() [all]: Size: 478 bytes, 18.053789138794 time to encode

Combination
Having the smallest result in size might not always be the best metric to base the choice of the serialization algorithm. For instance, looking at the results above in the Strings test, igbinary produces indeed the smallest result in size (64 bytes) but it takes twice as much to serialize the result in comparison to json_encode (3.22 vs. 1.56 seconds) and the size difference is a mere 3 bytes (64 vs. 67).

Similarly, for the Boolean test, igbinary produces 27 bytes and json_encode 39 bytes. It does however take igbinary nearly 80% more time to produce the result compared to json_encode.

For the Floats test the situation is reversed. json_encode produces a result that is around 50% smaller than the one of igbinary but it takes twice as much time to produce it.

As far as serializing data, in my personal opinion, json_encode is the clear winner.

Unserializing Results

Unserializing data is equally - and at times - more important than serializing. In many applications, developers sacrifice performance in writing but don't compromise when reading data.

In the tests below (shortest time highlighted in bold) once can easily see that igbinary is the clear winner. At times the unserialize function is very close (or outperforms igbinary) but overall, igbinary is the the function that unserializes data the fastest.

Speed comparison

Strings

unserialize() [strings]: 1.8259189128876 time to decode
json_decode() [strings]: 2.6482670307159 time to decode
igbinary_unserialize() [strings]: 1.8359968662262 time to decode

Integers

unserialize() [integers]: 2.3886890411377 time to decode
json_decode() [integers]: 2.8659090995789 time to decode
igbinary_unserialize() [integers]: 2.4441809654236 time to decode

Booleans

unserialize() [booleans]: 1.8097970485687 time to decode
json_decode() [booleans]: 2.4416139125824 time to decode
igbinary_unserialize() [booleans]: 1.7585029602051 time to decode

Floats

unserialize() [floats]: 18.512004137039 time to decode
json_decode() [floats]: 3.7896130084991 time to decode
igbinary_unserialize() [floats]: 2.6730649471283 time to decode

Mixed

unserialize() [mixed]: 4.6794769763947 time to decode
json_decode() [mixed]: 2.7775249481201 time to decode
igbinary_unserialize() [mixed]: 1.9598047733307 time to decode

Objects

unserialize() [objects]: 5.5468521118164 time to decode
json_decode() [objects]: 5.7660481929779 time to decode
igbinary_unserialize() [objects]: 5.2672090530396 time to decode

All data types

unserialize() [all]: 31.01339006424 time to decode
json_decode() [all]: 14.574991941452 time to decode
igbinary_unserialize() [all]: 10.734386920929 time to decode

Conclusion
If your application is mostly focused on reads rather than writes, igbinary is the clear winner, since it will unserialize your data faster than the other two functions. If however you are more focused on storing data, json_encode is the clear choice.

2013-03-07: memcached was not used with igbinary. PHP version for tests was 5.3.1 on a Linux Mint machine with 6GB RAM

Git pre-commit - Another check to ensure clean code [How-To]

2011-11-13T13:45:00.001-05:00

Throughout my career I have been using various revision control systems. I started off with Visual SourceSafe which I thought at the time was great, primarily because of the small size of our team and the ease of use though the Visual Basic's IDE.

Later on I switched to SVN which is a great version control system and it fulfilled all my needs for proper version control.

When I moved to the US and started working here, I introduced SVN to the company that I was working at the time. We adopted the technique of having one branch per project, since we were working on different projects at any given time. Using the plugin through Eclipse or in Windows the integrated Explorer plugin, I was checking in code, switching through projects etc.

Recently I changed jobs and although we did use Subversion for a few months, we did switch to Git. The reasons behind it are too many to count - and I will go through them in another blog post.

One really good feature that Git has is its hooks. Although Subversion also supports hooks, I have only been exposed to the ones from Git and have used them - hence this blog post.

Git comes with some predefined hooks that one can use as a starting point. The code is checked before it is being committed and if it does pass whatever the pre-commit (for instance) hook does, it will allow you to commit; alternatively it will stop until you correct the mistakes made.

One very popular pre-commit hook is the one here by Remigijus Jarmalavičius. It checkes the files that have been modified/added and runs the php -l on it to ensure that whatever will be committed does not have PHP syntax errors.

I have downloaded that code and easily added it to my repository so everything is being checked prior to any of my commits.

In our MemberFuse™ platform, we have - like any other developer - many helper functions that are used solely for development. One of the mostly used one is the vdd($message); Effectively what this function does is a var_dump of the $message variable, echoes out the file that it was called as well as the line it is in, and dies. It is great for quick and dirty debugging and inspecting a variable as it passes through the code. Granted it is not very TDD but I am sure that all developers have little things like these to aid them with their daily programming and debugging.

From time to time, as we fix bugs and explore certain behaviors that sometimes it is difficult to assess, we use this function and we output the data on the browser. However this is just a development/debugging function and must never be used in the production environment.

I have been the culprit of using the function, forgot that I had it in a certain part of the code, and committed that code in our development branch. That caused some colleagues to experience inconsistent behavior with errors showing up on the screen and time was wasted.

To combat this scenario, I wrote a pre-commit hook that will allow you to check for the existence of certain functions in the code and ensure that those functions (or strings for that matter since I am using grep) do not exist in what is to be committed.

The pre-commit hook that I wrote is listed below and the code has been heavily based on Remigijus Jarmalavičius's pre-commit syntax checker.

#!/bin/bash
# Author: Nikolaos Dimopoulos 
# Based on code by Remigijus Jarmalavičius  
# Checks the files to be committed for the presence of print_r(), 
# var_dump(), die()
# The array below can be extended for further checks

checks[1]="var_dump("
checks[2]="print_r("
checks[3]="die"

element_count=${#checks[@]}
let "element_count = $element_count + 1"

ROOT_DIR="$(pwd)/"
LIST=$(git status | grep -e '\#.*\(modified\|added\)')
ERRORS_BUFFER=""
for file in $LIST
do
    if [ "$file" == '#' ]; then
        continue
    fi
    if [ $(echo "$file" | grep 'modified') ]; then
        FILE_ACTION="modified"
    elif [ $(echo "$file" | grep 'added') ]; then
        FILE_ACTION="added"
    else 
        EXTENSION=$(echo "$file" | grep ".php$")
        if [ "$EXTENSION" != "" ]; then

            index=1
            while [ "$index" -lt "$element_count" ]
            do
                echo "Checking $FILE_ACTION file: $file [${checks[$index]}]" 
                ERRORS=$(grep "${checks[$index]}" $ROOT_DIR$file >&1)
                if [ "$ERRORS" != "" ]; then
                    if [ "$ERRORS_BUFFER" != "" ]; then
                        ERRORS_BUFFER="$ERRORS_BUFFER\n$ERRORS"
                    else
                        ERRORS_BUFFER="$ERRORS"
                    fi
                    echo "${checks[$index]} found in file: $file "
                fi
                let "index = $index + 1"
            done
        fi
    fi
done
if [ "$ERRORS_BUFFER" != "" ]; then
    echo 
    echo "These errors were found in try-to-commit files: "
    echo -e $ERRORS_BUFFER
    echo 
    echo "Can't commit, fix errors first."
    exit 1
else
    echo "Commited successfully."
fi

If you want to check for any kind of string, just add one extra line (or modify the existing ones) of the checks array. The code will loop through the array and 'grep' the modified/added files for that entry. If it is found the commit will not be allowed and you will have to manually go and change whatever needs to be changed.

I hope you find this hook useful. You can download the file from my github repository.

Note: If you wish to run more than one pre-commit hooks, you don't need to merge them all in the same file. You can create a pre-commit file which will have the following contents:

#!/bin/bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

$DIR/pre-commit.l
$DIR/pre-commit.output

The *.l file is the one that runs the PHP syntax check, while the *.output one is the one mentioned in this blog post. You can extend the list to your liking and usage.

New beginnings – sleep(6566400) [Personal][Update]

2011-11-11T11:11:00.000-05:00

It has been a very long time since I last blogged.

The primary reason is that I have been busy and somewhat lazy. Hopefully that trend (the laziness) will not last that much and I will be able to post in higher frequency.

The biggest update for me came this July. I was actively searching for a new employment opportunity for quite some time, but due to the current market situation, it has been a really difficult task.

In July I received (and accepted) an offer from Avectra Inc. I am now a PHP developer for MemberFuse™, a professional networking application that integrates with the client's association management software and builds an online community for that client.

My colleagues and I are programming in PHP using a frameworks such as Doctrine, Zend Framework, Smarty and jQuery on the client side. The application we produce is offered as SAAS (Software As A Service) and we have well over 1 million users.

Although adjusting to the new environment was rough and the learning curve was steep (it didn't help that one colleague decided to resign and I inherited his workload from week 2), I managed to 'survive' and I am more and more confident every day.

I can only say that working with smart people that push you to excel is a blessing!

The second update is this blog's location. With the introduction of the new dynamic templates in Blogger, I decided to move from Wordpress (and my personal installation for my blog) to Blogger.

The downside was that Blogger could not import my blog. I would always get errors when trying to upload the XML file and after a few tries I decided to go the manual way. I therefore sat down and copied and pasted all the content that I had posted in the past to the new platform.

Luckily it was not a very difficult task, but it was time consuming. Last night I managed to finish everything and write this blog post.

With this move unfortunately I lost all the comments on my posts and I am sure that there will be some broken links here and there but in the end it will all work out.

It was my intention to post this message today (11-11-11) at 11:11 but I kinda messed up the scheduling so the post is being posted a few hours later.

Using autounmask to Unmask Packages in Gentoo [How-To][Gentoo]

2011-05-18T14:49:00.000-04:00

Gentoo is one of my favorite Linux distributions. Although I am comfortable with other distributions, Gentoo has a special place in my heart and whenever I can use it I do :)

There are however some times that I would like to install a package - mostly to test something - and the package is masked. Masked packages are not "production ready" so they are not included in the portage tree i.e. available to be installed.

To allow a masked package to be installed, you will need to unmask that package by adding a corresponding entry in the etc/portage/package.keywords file.

The problem happens when the masked package (that you just unmasked) depends on other packages that are also masked. You will then need to rinse and repeat the process to ensure that everything is in place so that you can install that unmasked package.

NOTE: Playing with masked packages is like playing with fire. If you don't know what you are doing or you are not ready to potentially have an unusable system, don't follow the instructions below or unmask any packages.

app-portage/autounmask
The Gentoo developers have created a little utility that will unmask each package that needs to be unmasked. The utility is app-portage/autounmask.

I wanted to unmask www-misc/fcgiwrap so my manual method would be:

echo "=www-misc/fcgiwrap ~amd64" >> /etc/portage/packages.keywords

and would then emerge the package

emerge =www-misc/fcgiwrap

Instead I used autounmask:

emerge app-portage/autounmask

and

autounmask www-misc/fcgiwrap-1.0.3

autounmask version 0.27 (using PortageXS-0.02.09 and portage-2.1.9.42)
* Using repository: /usr/portage
* Using package.keywords file: /etc/portage/package.keywords * Using package.unmask file: /etc/portage/package.unmask * Using package.use file: /etc/portage/package.use
* Unmasking www-misc/fcgiwrap-1.0.3 and its dependencies.. this might take a while..
* Added '=www-misc/fcgiwrap-1.0.3 ~amd64' to /etc/portage/package.keywords * done!

Once that is done I can issue the emerge command and voila!

emerge www-misc/fcgiwrap

Although in my case there was only one dependency to unmask, when trying to unmask packages that have multiple dependencies such as gnome, kde etc., autounmask can be a very helpful utility.

LibreOffice Auto Update [How-To][Linux]

2011-04-21T14:45:00.000-04:00

We all knew it was coming. Ever since Oracle took a really surprising turn towards the non-community orientation of OpenOffice, a lot of users on the Internet were curious (and concerned) as to what will happen with the whole OpenOffice saga.

Thankfully the project was forked and LibreOffice was born. Within a few months the same (and better) quality product was released for the community. As the project entered its alpha stage, then the beta stage and then the RC stages, I was waiting anxiously to install it on my notebook (which is running Ubuntu 10.10 at the moment).

A couple of months ago, I was finally happy with the package and the release, so I downloaded it on my notebook and started the installation. I don't have a problem with getting my hands dirty and working with the command prompt, but I am lazy - so I want my packages to update easily (say though the package manager). At the time I could not use the package manager, so it was all terminal work :)

The installation was pretty easy using the dpkg command. I run literally two commands and the application was installed :)

There was a recent announcement about OpenOffice from Oracle. I posted a link at HackerNews for an article I read in MarketWire, regarding this "Oracle to move OpenOffice.org to a Community-Based Project". Unfortunately the link does not work any more but you might be able to read a bit about the subject here. In short:

OpenOffice is dead, long live LibreOffice.

Time for me to start getting serious about updating LibreOffice.

First of all I wanted to make sure that I did not have any cruft remaining from OpenOffice on my system.

sudo apt-get remove openoffice*.*

Surprisingly there were some packages that were still there.

The second step was to uninstall the current version of LibreOffice

sudo apt-get remove libreoffice*.*

And now comes the easy part. I am going to use a PPA, to ensure that I will get notified about pending updates and upgrade easily. To do so all I had to do was run the following commands on a terminal window:

sudo add-apt-repository ppa:libreoffice/ppa

Output:

Executing: gpg --ignore-time-conflict --no-options --no-default-keyring
--secret-keyring /etc/apt/secring.gpg --trustdb-name /etc/apt/trustdb.gpg
--keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg
--keyserver keyserver.ubuntu.com --recv 36E81C9267FD1383FCC4490983FBA1751378B444gpg:
requesting key 1378B444 from hkp server keyserver.ubuntu.com
gpg: key 1378B444: public key "Launchpad PPA for LibreOffice Packaging" imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Then update the packages again:

sudo apt-get update

And now install LibreOffice:

sudo apt-get install libreoffice
sudo apt-get install libreoffice-gnome

Note: KUbuntu users will need to run

sudo apt-get install libreoffice-kde

For those that prefer the graphical interface, all you will have to do is add "ppa:libreoffice/ppa" to your software sources.

Enjoy your LibreOffice installation!

Building a HA Cluster on Linode [How-To][HA][Linux][Series]

2011-04-04T14:41:00.000-04:00

I have recently blogged about Slicehost vs. Linode and my decision to move my sites to the latter. Since then I can safely say that I made the right choice about the move. Linode's support is phenomenal. There has never been a ticket unanswered more than 5 minutes and all the tickets have been resolved. Even when I asked about a configuration issue regarding Heartbeat, which was clearly not in the realm of support, the support engineers did look at my configuration and did identify the error area. That alone saved me hours of troubleshooting and trying to find where the error was.

The Task
Your website must be available - all the time. Why? Because if you have something to say (a blog, a journal, a rant page, a forum etc.) you need your audience to be able to read your material all the time. Discussion forums, websites with custom applications or services, even information based websites need to have as close to 100% uptime as possible.. How can this be achieved? The solution is a High Availability (or HA) cluster.

Linux has been used for HA tasks for many years. The Linux-HA's wiki has a lot of valuable information as well as guides and resources that one can use to increase the redundancy of their websites. In addition to this, Google is your friend. There are numerous bloggers that have shared their experiences with the public on how to create HA resources. Finally, one can check Linode's Library - a set of guides to allow you to create HA clusters with your Linodes.

My task for the last few weeks has been to create a High Availability cluster of services to serve PHP and MySQL and also have the ability to grow infinitely (well close to that is). In the next few weeks I will post a series of blog posts outlining how to achieve a HA cluster for your sites.

Architecture
The cluster will be build using CentOS as the OS of choice. I have also experimented with Gentoo and Ubuntu. You can do everything I do here with Ubuntu if you wish. There are slight differences in certain commands and steps which the blog posts will not cover. As far as Gentoo is concerned, at the moment there is a block between Pacemaker and Heartbeat. Once that is resolved, I will try to redo the whole thing using Gentoo - as it is my OS of preference.

We will build two boxes to serve as load balancers. The boxes (or Linodes) will have Heartbeat, Pacemaker and nginx installed on them. A "floating" IP address will be used to move from one node to another in the case of a failure. Each Linode uses nginx as a proxy to forward all requests to a set of web servers using nginx's proxy functionality.

We will also build a web server. This again is a CentOS box running PHP and nginx. The web server will store the data locally and connect to the database cluster. Once the last part of this How-To is completed (creation of a HA NFS) then we will be able to add as many web servers as we need.

The next set of boxes form the database cluster. The setup is two servers with Heartbeat, Pacemaker and MySQL setup with a Master/Master replication. Again a "floating" IP address is used to move from one server to another in the case of failure.

The NFS is also a 2 box setup. Again a "floating" IP address is used to connect to the file system. DRBD is installed on them to cater for the replication.

Administration
All Linodes are located in the same data center. At the moment there is no way to create Linodes in different data centers and implement the above mentioned setup in an effort to achieve geographical redundancy. The whole setup is using 9 Linodes the last one being used for administrative tasks (note in my count I used 2 boxes as web servers).

All Linodes have active iptables configurations. All nodes have been configured to work with 322 as the SSH port to avoid the novice hacker. Every port is blocked from communicating with the Internet apart from the ones needed for essential services. For instance the Load Balancers allow connections on ports 80 and 443 (http and https). However, they only communicate with each other on the Heartbeat port. Equally the web servers do not allow connections to their 80/443 ports to any machine other than the Load Balancers. The Database Servers allow connections only from the Web servers etc.

The administrative node resides on a different data center. It runs Nagios and it monitors all the nodes of our HA cluster. Naturally the iptables setup of each node is adjusted to allow connections from this particular node.

Conclusion
I hope that these blog posts will serve as a learning exercise/guide to those who want to delve in High Availability websites. The primary reason I built this cluster is to offer these services to customers that have busy sites that require maximum availability. I am currently setting up the final touches of the hosting service I am going to offer so stay tuned for the details.

Disclaimer: The Linode link in this blog post is an affiliate one. If you wish to sign up for their services you are more than welcome to click it. The non affiliate link is here.

Slicehost vs. Linode [Hosting][VPS]

2011-03-21T14:10:00.000-04:00

Through the years I have hosted my sites on various hosting companies. I had the really good experiences like Vertexhost and really terrible ones - I don't remember the name of the host, but that kid, as it turned out to be later on, managed to lose 1.6GB of my data. You can safely say that I have been burned by bad hosting companies but also have enjoyed the services of good ones. In the case of Vertexhost, I part owned that company a few years back and I know that the current owner is a straight up guy and really cares for his customers.

Since I moved my emails to Google Apps I only need the hosting for my personal sites such as my blog, my wife's sites (burntoutmom.com, greekmommy.net) and a few other small sites.

I used to host those sites on one of my company's clusters. The bandwidth consumed was nothing to write home about (I think in total it was a couple of GB per month ~ 1.00 USD) so it didn't matter that I had them there. However, recent events forced me to move them out of that cluster. I was on the market for good and relatively cheap hosting. I did not want to purchase my own server or colocate with someone else. My solution was going to be a VPS since I would be in control of what I install and what I need.

Slicehost

Without much thought, I signed up for Slicehost, which is a subsidiary of Rackspace, a very well known and reputable company.

I got their 4GB package (250.00 USD per month) and installed Gentoo on it. Apart from the price which was a bit steep, everything else was fine. I was happy to be able to host my sites in a configuration that I was comfortable with, under the understanding that if the VPS failed, then all my sites would go down. That however is the risk that everyone takes while hosting their sites on a single machine. The higher the availability and redundancy the higher the cost.

I must admit that signing up was not a very happy experience. I went and paid with my credit card, as they pro-rate your month based on your package. Almost immediately after signing up, came the email informing me that my credit card has been charged for the relevant amount. I got into the box through ssh, updated the /etc/make.conf file with the USE flags that I needed, run emerge --sync and then emerge --update --deep --newuse --verbose world so as to update the system.

It must have been around 5-10 minutes into the process that I received an email from Slicehost saying that they are checking my account information and that I need to confirm my credit card details. I immediately replied to their email (gotta love the desktop notifications on GMail), with the information they needed.

After I sent the email, I noticed that the box was not responding. I tried to log back in and could not. I was also logged out (and could not log back in) to their management console on slicehost.com site. I was fuming! Effectively they severed the connection to the VPS in the middle of compilation to check my credit card information.I understand that they need to perform checks for fraud but two questions came to mind:

Why did they have to sever the connection and not just send an email, and if I did not reply, just block access to the box? That would have been a heck of a lot of an inconvenience to myself i.e. the end user.
Why did the initial email say that my credit card has been charged and it had not?

No more than 10 minutes later the whole thing had been resolved. I received an email saying that "everything is OK and your account has been restored", at which point I logged back in to redo the compilations. I also received emails from their support/billing team apologizing but stating that although the initial email states that they charge the credit card, they don't. It is something they need to correct because it pisses people (like me) off.

There was nothing wrong with my setup - everything was working perfectly but the price was really what was bothering me. I would be able to support the sites for a few months, but since literally none of them is making money (maybe a dollar here or there from my blog but that is about it), I would have to pay out of pocket for the hosting every month. I had to find a different solution that would be:

cheaper than Slicehost
flexible in terms of setup
easy to use in terms of controlling your VPS

After a lot of research I ended up with two winners: Linode and Prgmr. I opted for Linode, because although it was quite a bit more expensive than Prgmr, it had the better console in handling your VPS. I will, however, try out Prgmr's services in the near future so as to assess how good they are. They definitely cannot be beat in price.

Linode

Setting up an account with Linode was very easy. I didn't have any of the mini-saga I had with Slicehost. The account was created right there and then, my credit card charged and I was up and running in no time. Immediately I could see a difference in price. Linode's package for 4GB or RAM is 90.00 USD cheaper (159.00 USD vs. 250.00 USD for Slicehost). For the same package, the price difference is huge.

I started testing the network, creating my VPS in the Atlanta, GA datacenter (Linode offers a number of datacenters for you to create your own). The functionality that was available to me was identical and in some cases superior to that of Slicehost. There are a lot more distributions to choose from, and you can partition your VPS the way you want it to name a couple.

Shifting through the documentation, I saw a few topics regarding high availability websites. The articles described using DRBD, nginx, heartbeat and pacemaker etc. to keep your sites highly available. I was intrigued by the information and set off to create a load balancer using two VPSs and nginx. I have documented the process and this is another blog post that will come later on this week.

While experimenting with the load balancer (and it was Saturday evening) I had to add a new IP address to one of the VPS instances. At the time my account would not allow such a change and I had to contact support. I did and got a reply in less than 5 minutes. I was really impressed by this. Subsequent tickets were answered within the 5 minute time frame. Kudos to Linode support for their speed and professionalism.

Conclusion

For a lot cheaper, Linode offered the same thing that Slicehost did. Moving my sites from one VPS to another was a matter of changing my DNS records to point to the new IP address.

I have been using Linode for a week and so far so good. The support is superb and the library is full of how-to's that allows me to experiment with anything I want to - and the prices are not going to break me.

Resources

Vertexhost, Slicehost, Rackspace, Gentoo, Linode, Linode Library

Additional storage for Google Apps users [Information][Review]

2011-03-10T14:00:00.000-05:00

I have been using Google Apps for a number of years now and I have gotten so used to it that I cannot fathom any other way of operating. I am sure that some of you share that sentiment. :)

Limitations
Up until a few months ago, Google Apps had its limitations. The actual Apps was in some sort of a jailshell, isolated from the whole Google suite of applications. For that reason you could not use your Google Apps login to enjoy the service of Google Reader for instance. You had to be sneaky about it. You had to create a Google Account with the same username (and password if you liked) as your Google Apps domain and although the two did not communicate, you could have effectively "one login" for all services.

This limitation became more apparent with the increased usage of Android phones (where you need to have a Google Account on your phone) as well as Google Voice. Users have been asking about the "merge" and Google responded with significant infrastructure changes to cater for the transition. In my blog post "Google Apps and Google Accounts Merge" I present additional information about this, inclusive of a How-To on the transition for administrators of Google Apps. Unfortunately the process is not perfect and there are still some services that are not fully integrated with the new infrastructure (but will be in the future). For instance in my domain, since I use Google Voice with my domain email account, I am still on the "old" system because the account could not be transitioned. It will happen in the end, it just takes time.

Storage Needs
The biggest issue for me that was related to these two separated accounts (Google Apps Account vs. Google Account) was Picasa and Google Docs.

I have been very methodical in my filing, utilizing electronic storage as much as possible. For that reason I have been scanning documents and uploading them to Google Docs (or if they were available in PDF format I would just upload them). The documents would range from personal, utility bills, bank statements, anything that I want to store. Soon I realized that the 1GB that Google Apps offers for documents will not cut it. I therefore created a new account which I named DocsMule1 (clearly to signify its purpose). I created one folder in that account, uploaded as many documents as I could there and shared that document with my own account as well as my wife's. Soon I found more limitations since I ended up with 3 mule accounts. Since there was no option for me to upgrade the storage (even if I paid for it), I had to change my strategy. Managing documents from 3 or more different accounts is not an easy and convenient task.

I downloaded all my documents back to my computer (gotta love Google's Data Liberation) and deleted them from the Google Apps mule accounts and then deleted those accounts - just to keep everything tidy. I then launched my Gmail account and signed into Google Docs. I created one folder which I shared with my Google Apps accounts (my wife's and mine) and then paid $50.00 for a whole year - which provided me with 200GB of space. You can always check how much space you are using by visiting the Manage Storage page of your Google Account.

Once that was done, I started creating my folders (collections now) and uploaded all my documents up there. In addition to that, since my parents live in Greece, they rely on VoIP chat as well as my Picasa to stay in touch with their grandchildren. My wife and I, through the use of our mini camera as well as our Android phones, take a lot of pictures of the kids, documenting the little things that they do on a regular basis. This serves as a good archive for them when they grow up but also as a good way to stay in touch with my parents. Google's additional storage was the solution.

Problem solved. With minimal money I had everything sorted out. It did however inconvenience me quite a bit in the end, since a lot of my data was scattered now. The GMail account would keep Picasa and Docs, the Google Apps account my email, my Google account my Reader, Web history etc. Not very convenient but it works.

Storage for Google Apps
Around February, Google announced that they will be offering the option to Google Apps users to purchase additional storage. I was really happy about that since I could therefore ditch the GMail account for handling my docs and keep everything under the domain account. However something was wrong. When Google revealed their pricing (the announcement is not there any more but the prices I am quoting are real), I quickly found out that for the storage I currently have, I would need to spend $700.00 a year instead of $50.00. That did not make sense at all. Needless to say, I stayed with my existing plan.

Initial Pricing Plan

Storage Price (per year)

   5 GB         17.50 USD
  20 GB         70.00 USD
  80 GB        280.00 USD
 200 GB        700.00 USD
 400 GB      1,400.00 USD
   1 TB      3,500.00 USD
   2 TB      7,000.00 USD
   4 TB     14,000.00 USD
   8 TB     28,000.00 USD
  16 TB     56,000.00 USD

New Pricing Plan
A few days ago, Google announced changes in the pricing of additional storage for Google Apps users as well as changes to the free storage that Google offers for Picasa. Picasa Web Albums does offer 1GB of free storage but now photos of 800x800 pixels or less as well as videos of 15 minutes or less do not count against the 1GB of storage. You can read more about the Picasa Web Albums storage in the relevant help page.

 Storage   Price (per year)
  20 GB         5.00 USD
  80 GB        20.00 USD
 200 GB        50.00 USD
 400 GB       100.00 USD
   1 TB       256.00 USD
   2 TB       512.00 USD
   4 TB     1,024.00 USD
   8 TB     2,048.00 USD
  16 TB     4,096.00 USD

As far as the new pricing is concerned, Google brought everything in line with Google Accounts pricing (effectively scrapping the initial - expensive - pricing plan for extra storage). The help page "Google Storage - How it Works" offers additional information for those that want to use/upgrade their storage while using a Google Apps account. Effectively it now costs exactly the same to purchase additional space for your Google Apps account (to store documents) as it would if you were using a different Google Account. That probably means that I have to download everything to my computer and re-upload it to my Google Apps account....

To take advantage of this feature, you will have to go to the Purchase additional storage page while logged in with the Google Apps account that you wish to purchase storage for. Note that there is a warning that appears in red (see image) that warns you that you are using a Google Apps account. Google provides this information since your Google Apps account relies on the Google Apps administrator. If you have an account on Google Apps and you purchase storage, that storage will be gone if the administrator deletes or restricts access to your account.

Conclusion
In my view, Google has done it again. They now offer an extremely affordable and secure way of storing your data. There are loads of people that have concerns about where their data is stored, who has ownership of the data stored, what does Google do with the data etc. A lot of these questions can easily be answered if you google (duh) the relevant terms or search in Google's Help Center. Data Liberation allows you to retrieve your data whenever you want to. If on the other hand you are skeptical and do not wish to store your data there, don't. It is your choice.

References
Google Docs, Picasa Web Albums, Data Liberation, Google Account - Manage Storage, Photo and Video Requirements: Size Limits for Picasa Web Albums, Google Storage - How it Works, Google Account - Purchase additional storage, Google's Help Center

Keeping your Linux users in sync in a cluster [How-To][Linux][Sync]

2011-02-20T13:56:00.000-05:00

As websites become more and more popular, your application might not be able to cope with the demand that your users put on your server. To accommodate that, you will need to move out of the one server solution and create a cluster. The easiest cluster to create would be with two servers, one to handle your web requests (HTTP/PHP etc.) and one to handle your database load (MySQL). Again this setup can only get you so far. If your site is growing, you will need a cluster of servers.

Database
The scope of this How-To will not cover database replication; I will need to dedicate a separate blog post for that. However, clustering your database is relatively easy with MySQL replication. You can set a virtual hostname (something like mysql.mydomain.com) which is visible only within your network. You then set up the configuration of your application to use that as the host. The virtual hostname will map to the current master server, while the slave(s) will only replicate.

For instance, if you have two servers A and B, you can configure both of them to become master or slave in MySQL. You then set one of them as master (A) and the other as slave (B). If something happens to A, B gets promoted to master instantly. Once A comes back up, it gets demoted to a slave and the process is repeated if/when B has a problem. This can be a very good solution but you will need to have pretty evenly matched servers to keep with the demand. Alternatively B can be less powered than A and when A comes back up you keep it momentarily as a slave (until everything is replicated) and then promote it back to master.

One thing to note about replication (that I found through trial and error). MySQL keeps binary logs to handle replication. If you are not cautious in your deployment, MySQL will never recycle those logs and therefore you will soon run out of space when having a busy site. By default those logs will be under /var/lib/mysql.

By changing directives in my.cnf you can store the binary logs in a different folder and even set up 'garbage collection' or recycling. You can for instance set the logs to rotate every X days with the following directive in my.cnf:

expire_logs_days  = 5

I set mine to 5 days which is extremely generous. If your replication is broken you must have the means to know about it within minutes (see nagios for a good monitoring service). In most cases 2 days is more than enough.

Files
There are numerous ways of keeping your cluster in sync. A really good tool that I have used when playing around with a cluster is csync2. Installation is really easy and and all you will need is to run a cron task every X minutes (up to you) to synchronize the new files. Imagine it as a two way rsync. Another tool that can do this is unison but I found it to be slow and difficult to implement - that's just me though.

Assume an implementation of a website being served by two (or more) servers behind a load balancer. If your users upload files, you don't know where those files are uploaded, which server that is. As a result if user A uploads the file abc.txt to server A, user B might be served the content from server B and would not be able to access the file. csync2 would synchronize the file across the number of servers, thus providing access to the content and keeping multiple copies of the content (additional backup if you like).

NFS
An alternative to keeping everything synchronized is to use a NFS. This approach has many advantages and some disadvantages. It is up to you on whether the disadvantages are something you can live with.

Disadvantages

NFS is slow - slower than the direct access to a local hard drive.
Most likely you will use a symlink to the NFS folder, which can slow things down even more.

Advantages

The NFS does not rely on the individual web servers for content.
The web servers can be low to medium spec boxes without the need to have really fast and large hard drives
A well designed NFS with DRDB provides a raid-1 over a network. Using gigabit Network Interface Cards you can keep performance at really high levels.

I know that my friend Floren does not agree with my approach on the NFS and would definitely have gone with the csync2 approach. Your implementation depends on your needs.

Users and Groups
Using the NFS approach, we need to keep the files and permissions properly set up for our application. Assume that we have two servers and we need to create one user to access our application and upload files.

The user has been created on both servers and the files are stored on the NFS. Connecting to server A and looking at the files we can see something like this:

drwxr-xr-x 17 niden  niden  4096 Feb 18 13:41 www.niden.net
drwxr-xr-x  5 niden  niden  4096 Nov 15 22:10 www.niden.net.files
drwxr-xr-x  7 beauty beauty 4096 Nov 21 17:42 www.beautyandthegeek.it

However when connecting to server B, the same listing tells another story:

drwxr-xr-x 17 508    510    4096 Feb 18 13:41 www.niden.net
drwxr-xr-x  5 508    510    4096 Nov 15 22:10 www.niden.net.files
drwxr-xr-x  7 510    511    4096 Nov 21 17:42 www.beautyandthegeek.it

The problem here is the uid and gid of the users and groups of each user respectively. Somehow (and this is really easy to happen) server A had one or more users added to it, thus the internal counter of the user IDs has been increased by one or more and is not identical to that one of server B. So adding a new user in server A will get the uid 510 while on server B the same process will produce a user with a uid of 508.

To have all users setup on all servers the same way, we need to use two commands: groupadd and useradd (in some Linux distributions you might find them as addgroup and adduser).

groupadd
First of all you will need to add groups. You can of course keep all users in one group but my implementation was to keep one user and one group per access. To cater for that I had to first create a group for every user and then the user account itself. Like users, groups have unique ids (gid). The purpose of gid is:

The numerical value of the groups ID. This value must be unique, unless the -o option is used. The value must be non-negative. The default is to use the smallest ID value greater than 999 and greater than every other group. Values between 0 and 999 are typically reserved for system accounts.

I chose to assign each group a unique id (you can override this behavior by using the -o switch in the command below, thus allowing a gid to be used in more than one group). The arbitrary number that I chose was 2000.

As an example, I will set niden as the user/group for accessing this site and beauty as the user/group that accesses BeautyAndTheGeek.IT. Note that this is only an example.

groupadd --gid 2000 niden
groupadd --gid 2001 beauty

Repeat the process as many times as needed for your setup. Connect to the second server and repeat this process. Of course if you have more than two servers, repeat the process on each of the servers that you have (and each accesses your NFS)

useradd
The next step is to add the users. Like groups, we will need to set the uid up. The purpose of the uid is:

The numerical value of the users ID. This value must be unique, unless the -o option is used. The value must be non-negative. The default is to use the smallest ID value greater than 999 and greater than every other user. Values between 0 and 999 are typically reserved for system accounts.

Like with the groups, I chose to assign each user a unique id starting from 2000.

So to in the example above, the commands that I used were:

useradd --uid 2000 -g niden --create-home niden
useradd --uid 2000 -g beauty --create-home beauty

You can also use a different syntax, utilizing the numeric gids:

useradd --uid 2000 --gid 2000 --create-home niden
useradd --uid 2000 --gid 2001 --create-home beauty

Again, repeat the process as many times as needed for your setup and to as many servers as needed.

In the example above I issued the --create-home switch (or -m) so as a home folder to be created under /home for each user. Your setup might not need this step. Check the references at the bottom of this blog post for the manual pages for groupadd and useradd.

I would suggest that you keep a log of which user/group has which uid/gid. It helps in the long run, plus it is a good habit to keep proper documentation on projects :)

Passwords?
So how about the passwords on all servers? My approach is crude but effective. I connected to the first server, and set the password for each user, writing down what the password was:

passwd niden

Once I had all the passwords set, I opened the /etc/shadow file.

nano /etc/shadow

and that revealed a long list of users and their scrambled passwords:

niden:$$$$long_string_of_characters_goes_here$$$$:13864:0:99999:7:::
beauty:$$$$again_long_string_of_characters_goes_here$$$$:15009:0:99999:7:::

Since I know that I added niden and beauty as users, I copied these two lines. I then connected to the second server, opened /etc/shadow and located the two lines where the niden and beauty users are referenced. I deleted the existing lines, and pasted the ones that I had copied from server A. Saved the file and now my passwords are synchronized in both servers.

Conclusion
The above might not be the best way of keeping users in sync in a cluster but it gives you an idea on where to start. There are different implementations available (Google is your friend) and your mileage might vary. The above has worked for me for a number of years since I never needed to add more than a handful of users on the servers each year.

References
MySQL System Variables, Nagios, csync2, rsync, Unison File Synchronizer, DRDB, Axivo Inc., groupadd man page, useradd man page

A look in the past and the future [Update]

2011-02-06T13:47:00.000-05:00

It has been months since I last posted a blog post. A lot of things have happened since August and I have a lot of material to post - just not the discipline to sit down and proof all the drafts so that I can post them.

2010
2010 has been a rough year mostly on a personal level. I lost my brother in law in March, my daughter was born in May, there was great uncertainty at work during the summer, an upgrade went bad for Long Hair Care Forum to name a few of the highlights.

Since this is officially my first year of blogging, I was happy to see some of the statistics for that year (well 8 months to be exact since I haven't posted since August).

Visits	8,940
Pageviews	11,979
Pages/Visit	1.41
Bounce Rate	81.34%
Avg Time on Site	00:01:10
% New Visits	85.04%

The most visited posts were:

Google Apps and Google Accounts merge [Information][How-To] with 1,601 pageviews (13.37%) Subversion Backup [How-To] with 1,289 pageviews (10.76%) Android 2.2 (Froyo) [Review] with 1,124 pageviews (9.38%)

Not that bad for a first year.

2011
When I started blogging my main focus was to mostly address coding issues. However due to the plethora of solutions on the Internet, the content that I wanted to relay was not that interesting or had been covered a hundred times or more. For instance I did start posting about design patterns here and here, but at the same time Giorgio Sironi started blogging about design patterns and did an excellent job at it, so that topic was scrapped. My posts then started covering a much broader scope, mostly that of technology with a small personal flare.

For 2011 I will continue on the same scope. I intend on blogging about interesting things technologically, how-to's and topics that I want to share with the community.

As always, whatever is presented in this blog is my personal opinion. Every post covered here as well with any code are free of copyright and you are free to use them in your projects at your own risk.

A big thank you to everyone that visited this blog whether you shared your opinion with me or not. I hope to meet the expectations of producing interesting content in the future.

Create an inexpensive hourly remote backup [How-To]

2010-08-21T13:38:00.000-04:00

There are two kinds of people, those who backup regularly, and those that never had a hard drive fail

As you can tell the above is my favorite quote. It is so true and I believe everyone should evaluate how much their data (emails, documents, files) is worth to them and, based on that value, create a backup strategy that suits them. I know for sure that if I ever lost the pictures and videos of my family I would be devastated since those are irreplaceable.

So the question is how can I have an inexpensive backup solution? All my documents and emails are stored in Google, since my domain is on Google Apps. What happens to the live/development servers though that host all my work? I program on a daily basis and the code has to be backed up regularly so as to avoid any hard drive failures and thus result in loss of time and money.

So here is my solution. I have an old computer (IBM Thincentre) which I decided to beef up a bit. I bought 4Gb of RAM from eBay for less than $100 for it. Although this is was not necessary since my solution would be based on Linux (Gentoo in particular), I wanted to have faster compilation times for packages.

I bought two external drives (750Gb and 500Gb respectively) and one 750Gb internal drive. I already have a 120Gb hard drive in the computer. The two external ones are connected to the computer using USB while the internal ones are connected using SATA.

The external drives are formatted using NTFS while the whole computer is built using ReiserFS.

Here is the approach:

I have installed and have a working Gentoo installation on the machine
I have an active Internet connection
I have installed LVM on the machine and set up the core system on the 120Mb drive while the 500Mb is on LVM
I have 300Mb active on the LVM (from the available 500Mb)
I have generated a public SSH key (I will need this to exchange it with the target servers)
I have mounted the internal 500Mb drive to the /storage folder
I have mounted the external USB 750Mb drive to the /backup_hourly folder
I have mounted the external USB 500Mb drive to the /backup_daily folder

Here is how my backup works:

Every hour a script runs. The script uses rsync to syncrhonize files and folders from a remote server locally. Those files and folders are kept in relevant server named subfolders in the /storage folder (remember this is my LVM). So for instance my subfolders will be /storage/beryllium.niden.net, /storage/nitrogen.niden.net, /storage/argon.niden.net etc.

Once the rsync completes, the script continues by compressing the relevant 'server' folder and creates the compressed file with a date-time stamp on its name.

When all compressions are completed, if the time that the script has executed is midnight, the backups are moved from the /storage folder to the /backup_daily folder (which has the external USB 500Gb mounted). If it is any other time, the files are moved in the /backup_hourly folder (which has the external USB 750Gb mounted).

This way I ensure that I keep a lot of backups (daily and hourly ones). The backups are being recycled, so older ones get deleted. The amount of data that you need to archive as well as the storage space you have available dictate how far back you can go in your hourly and daily cycles.

So let's get down to business. The script itself:

#!/bin/bash
DATE=`date +%Y-%m-%d-%H-%M`
DATE2=`date +%Y-%m-%d`
DATEBACK_HOUR=`date --date='6 days ago' +%Y-%m-%d`
DATEBACK_DAY=`date --date='60 days ago' +%Y-%m-%d`
FLAGS="--archive --verbose --numeric-ids --delete --rsh='ssh'"
BACKUP_DRIVE="/storage"
DAY_USB_DRIVE="/backup_daily"
HOUR_USB_DRIVE="/backup_hourly"

These are some variables that I need for the script to work. The DATE and DATE2 are used to date/time stamp the backups, while the DATEBACK_* are used to clear previous backups. In this case it is set to 6 days ago (for my system). It can be set to whatever you want provided that you do not run out of space.

The FLAGS variable keeps the rsync command options while the BACKUP_DRIVE, DAY_USB_DRIVE and HOUR_USB_DRIVE hold the locations of the rsync folders, daily backup and hourly backup sorage areas.

The script works with arrays. I have 4 arrays to do the work and the 3 of them must have exactly the same elements.

# RSync Information
rsync_info[1]="beryllium.niden.net html rsync"
rsync_info[2]="beryllium.niden.net db rsync"
rsync_info[3]="nitrogen.niden.net html rsync"
rsync_info[4]="nitrogen.niden.net html db"
rsync_info[5]="nitrogen.niden.net html svn"
rsync_info[6]="argon.niden.net html rsync"

This is the first array which holds descriptions to what needs to be done as far as source is concerned. These descriptions get appended to the log and helps me identify what step I am in.

# RSync Source Folders
rsync_source[1]="beryllium.niden.net:/var/www/localhost/htdocs/"
rsync_source[2]="beryllium.niden.net:/niden_backup/db/"
rsync_source[3]="nitrogen.niden.net:/var/www/localhost/htdocs/"
rsync_source[4]="nitrogen.niden.net:/niden_backup/db"
rsync_source[5]="nitrogen.niden.net:/niden_backup/svn"
rsync_source[6]="argon.niden.net:/var/www/localhost/htdocs/"

This array holds the source host and folder. Remember that I have already exchanged SSH keys with each server, therefore when the script runs there is a direct connection to the source server. If you need to keep things a bit more secure for you, then you will need to alter the contents of the rsync_source array so that it reflects the user that you log in with as well as the password.

# RSync Target Folders
rsync_target[1]="beryllium.niden.net/html/"
rsync_target[2]="beryllium.niden.net/db/"
rsync_target[3]="nitrogen.niden.net/html/"
rsync_target[4]="nitrogen.niden.net/db/"
rsync_target[5]="nitrogen.niden.net/svn/"
rsync_target[6]="argon.niden.net/html/"

This array holds the target locations for the rsync. These folders exist in my case under the /storage subfolder.

# GZip target files
servers[1]="beryllium.niden.net"
servers[2]="nitrogen.niden.net"
servers[3]="argon.niden.net"

This array holds the names of the folders to be archived. These are the folders directly under the /storage folder and I am also using this array for the prefix of the compressed files. The suffix of the compressed files is a date/time stamp.

Here is how the script evolves:

echo "BACKUP START"  >> $BACKUP_DRIVE/logs/$DATE.log
date >> $BACKUP_DRIVE/logs/$DATE.log

echo "BACKUP START"  >> $BACKUP_DRIVE/logs/$DATE.log
date  >> $BACKUP_DRIVE/logs/$DATE.log

# Loop through the RSync process
element_count=${#rsync_info[@]}
let "element_count = $element_count + 1"
index=1
while [ "$index" -lt "$element_count" ]
do
    echo ${rsync_info[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    rsync $FLAGS ${rsync_source[$index]} $BACKUP_DRIVE/${rsync_target[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    let "index = $index + 1"
done

The snippet above loops through the rsync_info array and prints out the information in the log file. Right after that it uses the rsync_source and rsync_target arrays (as well as the FLAGS variable) to rsync the contents of the source server with the local folder. Remember that all three arrays have to be identical in size (rsync_info, rsync_source, rsync_target).

The next thing to do is zip the data (I loop through the servers array)

# Looping to GZip data
element_count=${#servers[@]}
let "element_count = $element_count + 1"
index=1
while [ "$index" -lt "$element_count" ]
do
    echo "GZip ${servers[$index]}" >> $BACKUP_DRIVE/logs/$DATE.log
    tar cvfz $BACKUP_DRIVE/${servers[$index]}-$DATE.tgz $BACKUP_DRIVE/${servers[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    let "index = $index + 1"
done

The compression method I use is tar/gzip. I found it to be fast with a good compression ratio. You can choose anything you like.

Now I need to delete old files from the drives and copy the files on those drives. I use the servers array again.

# Looping to copy the produced files (if applicable) to the daily drive
element_count=${#servers[@]}
let "element_count = $element_count + 1"
index=1

while [ "$index" -lt "$element_count" ]
do
    # Copy the midnight files
    echo "Removing old daily midnight files" >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $DAY_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK_DAY*.* >> $BACKUP_DRIVE/logs/$DATE.log
    echo "Copying daily midnight files" >> $BACKUP_DRIVE/logs/$DATE.log
    cp -v $BACKUP_DRIVE/${servers[$index]}-$DATE2-00-*.tgz $DAY_USB_DRIVE/${servers[$index]}  >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $BACKUP_DRIVE/${servers[$index]}-$DATE2-00-*.tgz >> $BACKUP_DRIVE/logs/$DATE.log

    # Now copy the files in the hourly
    echo "Removing old hourly files" >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $HOUR_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK_HOUR*.* >> $BACKUP_DRIVE/logs/$DATE.log
    echo "Copying daily midnight files" >> $BACKUP_DRIVE/logs/$DATE.log
    cp -v $BACKUP_DRIVE/${servers[$index]}-$DATE.tgz $HOUR_USB_DRIVE/${servers[$index]} >> $BACKUP_DRIVE/logs/$DATE.log
    rm -f $HOUR_USB_DRIVE/${servers[$index]}/${servers[$index]}-$DATEBACK*.* >> $BACKUP_DRIVE/logs/$DATE.log
    let "index = $index + 1"
done

echo "BACKUP END"  >> $BACKUP_DRIVE/logs/$DATE.log

The last part of the script loops through the servers array and:

Deletes the old files (recycling of space) from the daily backup drive (/storage/backup_daily) according to the DATEBACK_DAY variable. If the files are not found a warning will appear in the log.
Copies the daily midnight file to the daily drive (if the file does not exist it will simply echo a warning in the log - I do not worry about warnings of this kind in the log file and was too lazy to use an IF EXISTS condition)
Removes the daily midnight file from the /storage drive.

The reason I am using copy and then remove instead of the move (mv) command is that I have found this method to be faster. Finally the same thing happens with the hourly files

Old files are removed (DATEBACK_HOUR variable)
Hourly file gets copied to the /backup_hourly drive
Hourly file gets deleted from the /storage drive

All I need now is to add the script in my crontab and let it run every hour. NOTE: The first time you will run the script you will need to do it manually (not in a cron job). The reason behind it is that the first time rsync will need to download all the contents of the source servers/folders in the /storage drive so as to create an exact mirror. Once that lengthy step is done, the script can be added in the crontab. Subsequent runs of the script will download only the changed/deleted files.

This method can be very effective while not using a ton of bandwidth every hour. I have used this method for the best part of a year now and it has saved me a couple of times.

The last thing I need to present you is the backup script that I have for my databases. As you notice above the source folder of beryllium.niden.net as far as databases are concerned is beryllium.niden.net/db/. What I do is I dump and zip the databases every hour on my servers. Although this is not a very efficient way of doing things and it adds to the bandwidth consumption every hour (since the dump will create a new file every hour) I have the following script running on my database servers every hour at the 45th minute:

#!/bin/bash

DBUSER=mydbuser
DBPASS='dbpassword'
DBHOST=localhost
BACKUPFOLDER="/niden_backup"
DBNAMES="`mysql --user=$DBUSER --password=$DBPASS --host=$DBHOST --batch --skip-column-names -e "show databases"| sed 's/ /%/g'`"
OPTIONS="--quote-names --opt --compress "

# Clear the backu folder
rm -fR $BACKUPFOLDER/db/*.*

for i in $DBNAMES; do
    echo Dumping Database: $i
    mysqldump --user=$DBUSER --password=$DBPASS --host=$DBHOST $OPTIONS $i > $BACKUPFOLDER/db/$i.sql
    tar cvfz $BACKUPFOLDER/db/$i.tqz $BACKUPFOLDER/db/$i.sql
    rm -f $BACKUPFOLDER/db/$i.sql
done

That's it.

The backup script can be found in my GitHub here.

Update: The metric units for the drives were GB not MB. Thanks to Jani Hartikainen for pointing it out.

Google Apps and Google Accounts merge [Information][How-To]

2010-08-18T13:17:00.000-04:00

Anyone that has a Google Apps account and wants to access other services like Google Reader, Google Voice etc. knows that the username and password of the Google Apps account does not work for these services, since those are not available for Google Apps accounts. To get around this limitation, what you could do (and what I have done in the past) is to sign up for a new Google Account with the same email address as your Google Apps account. This of course creates confusion at times, duplication of data and disassociation of services. The most obvious example of this is on an Android device. To use my phone, I need to sign in with my Google Apps account. However, to use my Google Voice number, I have to use sign in again for that service but now using my Google Account (which uses the same email address). This works but I still have to keep two sets of contacts - one for the Google Apps and one in Google Voice.

According to Google, 9 of the top 20 requests from Google Apps customers are for their accounts to work with more services from Google. To facilitate this, the Google Apps account had to be merged with the Google Account. This page in Google Apps Help that describes the transition.

Unfortunately this was not a simple flip of a switch for Google. Significant infrastructure changes had to be in place prior to the merging of the accounts. We were promised that this change would be in place by fall and we have not been disappointed. Google is rolling out the update slowly but if you want to 'speed up' the process, you can sign up for an early round of testing of the new infrastructure. I have signed up for several of my Google Apps domains a couple of weeks ago.

I am happy to announce that one of my domains has already gone through the merge process. The domain is BeautyAndTheGeek.IT which is a Google Apps Premier edition. My other domains that are hosted in Google Apps have not transitioned yet and I suspect that Google is first merging Premium accounts, then Educational, Government and it will finish up with the Standard edition of Google Apps, which kind of makes sense (paid customers first!).

Email Invitation

Invitation email to the Admin

The day before yesterday I received an email inviting me to participate in the new infrastructure change and merge the Google Apps accounts with the Google Accounts of my users.

The merge process works as follows:

- If your Google Apps account email address has never been used to access Google products such as Google Reader, Google Voice, Google Code etc. then your Google Apps account will be converted to a Google Account.

- If your Google Apps account email address has been used to access Google products such as Google Reader, Google Voice, Google Code etc. then your Google Apps account is conflicting with your Google Account. This page in Google Apps Administrator Help provides an overview of conflicting accounts. In short both of your accounts will merge and the data will be associated with your Google Apps account, which will now be the same as your Google Account.

There are certain limitations to this merge (as one would expect :)). These are as follows:

If the offline feature is enabled in GMail, the user has to synchronize before the transition. If they don't, they will lose their offline messages.
Offline Google Calendar doesn't work for Google Apps accounts that have transitioned.
The following Google products don't work with Google Apps accounts that have transitioned:
- Android Market for Developers
- Google Extra Storage (bummer)
- Health (bummer again)
- PowerMeter
- Profiles
- Web History
- YouTube
The purchase of additional Google Storage is currently unavailable for Google Apps accounts. No storage can be purchased for a Google Apps account, although it will be available later this year. If a user account has already purchased extra storage (like I have) in an existing Google Account, the storage will remain in that account and will not transfer to the respective Google Apps account after the transition.
Delegating email only works with the same account type (i.e. Google Apps)
Any Picasa Web Album, Profile, or Wave usernames cannot be moved from an existing account to your Google Apps account.

Resources: Transition readiness checklist, Conflicting accounts, Early adopters, Additional storage for Google Apps, Google Apps Administrator Help Center

The process

Google Apps - Dashboard Warning

Dashboard Warning

Once I logged in my Google Apps Administration area, the Dashboard presented a new notice at the top. In short the notice states that new services will be available to the Google Apps accounts. This will be achieved with the transition of the Google Apps Account to a Google Account. The update is free and Google will automatically roll it out on September 30, 2010.

Resource: Google Apps core suite

Google Apps - Understand Transition

Understanding the transition

Clicking the "Get Started" button at the bottom of the notice will start a wizard that helps with the transition of the domain's accounts to Google accounts of your users (some or all). The first screen contains information that aids in understanding the transition and what is involved. Moving to the new infrastructure will update your control panel and give you control over which Google services your users can access with their accounts.

Resource: Transition readiness checklist, Google Apps Administrator Help Center

Google Apps - New Services

Review New Services

Based on the services that are available for your domain user accounts, the screen above might be slightly different. It does however give you (the administrator) the ability to enable or disable services for your users. Note that turning off a service will disallow your users to sign up for that service using their Google Apps account for your domain. It will however not stop them from using a totally different Google Account (personal for instance) to access that service.

Google Apps - Notify Conflicting Accounts

Notify Conflicting Accounts

Confirming the new services, the wizard will display a new screen, that will allow you to notify those users that have conflicting accounts. This screen provides information on how many users have conflicting accounts (in my case only 1), what happens to those accounts (the merge process) and what should you do as the administrator. Google will not provide information on which of your users have conflicting accounts. What they will however do, is offer you a temporary email address and an email template, that you can use to email your users. The email template is shown in the "Google Apps - Email to User" section below.

Google Apps - Select Users

Select Users

Clicking "Continue", brings up the User Selection screen. Here the administrator can select whose account will be transitioned for now. You can choose to pilot test this transition with a small set of users or everyone. The difference is that if you choose a small set of users to pilot test this transition, you can revert their accounts back to what they were prior to this step. If you choose "Everyone" from this screen, the change will be across the organization and cannot be undone. Reminder here that the transition will happen either way by the end of September, 2010. Finally you have the option to inform the users when their account transition is complete. This generates an email (in English) to the user with relevant information. You can see the email in section "Google Apps - Email to User" below.

Resource: Early adopters

Google Apps - Confirmation

Confirmation

Clicking "Continue"again will bring up the final screen which is the confirmation page. Google is very thorough and I like the confirmation screen. The information in this screen is a summary of what the wizard collected in previous steps. You (the administrator) will have to confirm in several places that you have understood the process, read the agreement and then accept the whole process. Clicking "I accept. Start the transition." will make things happen :)

Resources: Transition readiness checklist

Google Apps - Transition in progress

Transition in Progress

Navigating back to the dashboard, you will see a message notifying you that there is a transition in place for the accounts of your domain and can take up to 24 hours to be completed. This serves as a reminder on what has just happened. The notice will disappear once the transition is completed.

Google Apps - Email to User

Email to User

If you (the administrator) chose to notify your users of this transition (see step Select Users above), they will receive an email like the one shown above. If you chose to notify your users with a different message then that message should have been sent to the temporary email address that Google has provided (see step Notifying Conflicting Accounts above)

Resources: Data ownership, Transition readiness checklist, Google Apps Administrator Help Center

Google Apps - Email to User after Transition

Email to User after transition

Finally another email will be sent to the user summarizing what has happened with their account. This email contains links to additional resources.

Resources: List of products, New sign in option, Data ownership, What is a browser?, Using multiple accounts, Help Center for Admins, Google Apps Administrator Help Center

Google Apps - User Login message for Account Merge

Message to user after login for account merge

Once the transition operation is completed, the next time the user logs in their account, they will see the screen above. In short this screen informs the user of what has happened and provides links to online resources for help. The user must click "I accept. Continue to my account." for them to have access to their account. This is a one off process.

Resources: Google Apps Administrator Help Center, Google Security and Privacy, Using multiple accounts, Terms of Service

Conclusion
Without a doubt this is a huge step forward for Google and for us as admins or users. The ability to have a single sign on to use their products not only helps them but us by simplifying everything. That also means that if a hacker guesses that my password is "password1" they have access to all my Google related services... but oh well :)

Joking apart though, I am really excited that this change has finally occurred. I am guessing that the next steps would be to give users or admins the ability to purchase additional storage for a single account is something that in my view will make quite a lot of money for Google. A lot of people will want to keep all their emails and are nearing or have reached their email quota. Administrators would be happy to pay for certain users that have depleted their email storage allocation without converting their whole Google Apps edition to the Premier one. After all, if your organization has 100 users and 5 of them are near capacity, even if you pay $20 per user it will cost $100 per year. If however the whole account is changed to a Premier account (with 25Gb per mailbox) the cost will go up to $5,000.

Some questions come to mind:

Will the user be able to purchase storage for all services or just for email?
Will the 1Gb limit from Docs be lifted and will it be combined with storage for mail?
$20 a year gives you 80Gb in Picasa. Will Google follow that model or change it? According to the Additional storage for Google Apps page it appears that it will be more expensive now to have more space.
If storage is so cheap (Picasa) will it be extended to Google Documents? It appears that 7.5Gb is enough for the vast majority of users as far as email is concerned, but 1Gb for Docs is not enough. For a society moving towards a full electronic document storage this could help a lot.

Still a long way to go until everything clears up. I am not sure that Google has all the answers yet but they are moving forward and this is the most encouraging news of all!

Subversion Backup [How-To]

2010-08-01T13:11:00.000-04:00

I will start this post once again with the words of a wise man:

There are two kinds of people, those who backup regularly, and those that never had a hard drive fail

So the moral of the story here is backup often. If something is to happen, the impact on your operations will be minimal if your backup strategy is in place and operational.

There are a lot of backup scenarios and strategies. Most of them suggest a backup once a day, usually at the early hours of the day. This however might not work very well with a fast paced environment where data changes several times per hour. This kind of environment is usually a software development one.

If you have chosen Subversion to be your software version control software then you will need a backup strategy for your repositories. Since the code changes very often, this strategy cannot rely on the daily backup schedule. The reason being is that, in software, a day's worth of work usually costs a lot more than the actual daily rate of the programmers.

Below are some of the scripts I have used over the years for my incremental backups, that I hope will help you too. You are more than welcome to copy and paste the scripts and use them or modify them to suit your needs. Please note though that the scripts are provided as is and that you must check your backup strategy with a full backup/restore cycle. I cannot assume responsibility of something that might happen in your system.

Now that the 'legal' stuff are out of the way, here are the different strategies that you can adopt. :)

svn-hot-backup
This is a script that is provided with Subversion. It copies (and compresses if requested) the whole repository to a specified location. This technique allows for a full copy of the repository to be moved to a different location. The target location can be a resource on the local machine or a network resource. You can also backup on the local drive and then as a next step transfer the target files to an offsite location with FTP, SCP, RSync or any other mechanism you prefer.

#!/bin/bash

# Grab listing of repositories and copy each
# repository accordingly

SVNFLD="/var/svn"
BACKUPFLD="/backup"

# First clean up the backup folder
rm -f $BACKUPFLD/*.*

for i in $(ls -1v $SVNFLD); do
    if [ $i != 'conf' ]; then
        /usr/bin/svn-hot-backup --archive-type=bz2 $SVNFLD/$i $BACKUPFLD
    fi
done

This script will create a copy of each of your repositories and compress it as a bz2 file in the target location. Note that I am filtering for 'conf'. The reason being is that I have a conf file with some configuration scripts in the same SVN folder. You can adapt the script to your needs to include/exclude repositories/folders as needed.

This technique gives the ability to immediately restore a repository (or more than one) by changing the configuration file of SVN to point to the backup location. If you run the script every hour or so then your downtime and loss will be minimal, should something happens.

There are some configuration options that you can tweak by editing the actual svn-hot-backup script. In Gentoo it is located under /usr/bin/. The default number of backups (num_backups) that the script will keep is 64. You can choose 0 to 'keep them all' but you can adjust it according to your storage or your backup strategy.

One last thing to note is that you can change the compression mechanism by changing the parameter of the --archive-type option. The compression types supported are gz (.tar.gz), bz2 (.tar.bz2) and zip (.zip)

Full backup using dump
This method is similar to the svn-hot-backup. It works by 'dumping' the repository in a portable file format and compressing it.

#!/bin/bash

# Grab listing of folders and dump each
# repository accordingly

SVNFLD="/var/svn"
BACKUPFLD="/backup"

# First clean up the backup folder
rm -f $BACKUPFLD/svn/*.*

for i in $(ls -1v $SVNFLD); do
    if [ $i != 'conf' ]; then
        svnadmin dump $SVNFLD/$i/ > $BACKUPFLD/$i.svn.dump
        tar cvfz $BACKUPFLD/svn/$i.tgz $BACKUPFLD/$i.svn.dump
        rm -f $BACKUPFLD/$i.svn.dump
    fi
done

As you can see, this version does the same thing as the svn-hot-backup. It does however give you a bit more control over the whole backup process and allows for a different compression mechanism - since the compression happens on a separate line in the script.

NOTE: If you use the hotcopy parameter in svnadmin (svnadmin hotcopy .....) you will effectively be duplicating the behavior of svn-hot-backup.

Incremental backup using dump based on revision
This last method is what I use at work. We have our repositories backed up externally and we rely on the backup script to have everything backed up and transferred to the external location within an hour, since our backup strategy is an hourly backup. We have discovered that sometimes the size of a repository can cause problems with the transfer, since the Internet line will not be able to transfer the files across in the allocated time. This happened once in the past with a repository that ended up being 500Mb (don't ask :)).

So in order to minimize the upload time, I have altered the script to dump each repository's revision in a separate file. Here is how it works:

We backup using rsync. This way the 'old' files are not being transferred.

Every hour the script loops through each repository name and does the following:

Checks if the *.latest file exists in the svn-latest folder. If no, then it sets the LASTDUMP variable to 0.
If the file exists, it reads it and obtains the number stored in that file. It then stores that number incremented by 1 in the LASTDUMP variable.
Checks the number of the latest revision and stores it in the LASTVERSION variable
It loops through the repository, dumps each revision (LASTDUMP to LASTVERSION) and compresses it

This method creates new files every hour so long as new code has been added in each repository via the checkin process. The rsync command will then pick only the new files and nothing else, therefore the data transferred is reduced to a bare minimum allowing easily for hourly external backups. With this method we can also restore a single revision in a repository if we need to.

The script that achieves that is as follows:

#!/bin/bash

# Grab listing of folders and dump each
# repository accordingly

SVNFLD="/var/svn"
BACKUPFLD="/backup"
CHECKFLD=$BACKUPFLD/svn-latest

for i in $(ls -1v $SVNFLD); do
    if [ $i != 'conf' ]; then
        # Find out what our 'start' will be
        if [ -f $CHECKFLD/$i.latest ]
        then
            LATEST=$(cat $CHECKFLD/$i.latest)
            LASTDUMP=$LATEST+1
        else
            LASTDUMP=0
        fi

        # This is the 'end' for the loop
        LASTREVISION=$(svnlook youngest $SVNFLD/$i/)

        for ((r=$LASTDUMP; r< =$LASTREVISION; r++ )); do
            svnadmin dump $SVNFLD/$i/ --revision $r > $BACKUPFLD/$i-$r.svn.dump
            tar cvfz $BACKUPFLD/svn/$i-$r.tgz $BACKUPFLD/$i-$r.svn.dump
            rm -f $BACKUPFLD/$i-$r.svn.dump
            echo $r > $CHECKFLD/$i.latest
        done
    fi
done

Conclusion
You must always backup your data. The frequency is dictated by the rate that your data updates and how critical your data is. I hope that the methods presented in this blog post will complement your programming and source control should you choose to adopt them.

Android 2.2 (Froyo) [Review]

2010-06-21T12:31:00.000-04:00

Android 2.2 (Froyo)

The newest version of the Android Operating System has been codenamed Froyo (as in Frozen Yogurt).

Although I have heard of the name, I was not following very closely the development of the said OS, so I did not know what to expect. The presentation on Day 2 at Google I/O was more than impressive so I had to get my hands on it :)

After the GoogleIO presentation, I was regularly checking (Settings – About Phone – System Updates) my phone to see what kind of version I am running and whether the update was waiting for me. Unfortunately I was disappointed every time, therefore I resigned to the idea that it will update when it is pushed to my phone.

A few days later though, reports on the blogosphere started appearing of users now running on Froyo. The update started rolling out so it was a matter of time for me. Most of the reports were coming from California, so if I were to look at the geography, it would take the update quite a bit of time to reach West Virginia :)

Through my regular research on the Internet, I found a few interesting blog posts that claimed that one can update the Nexus One without waiting for the update to be rolled out. Having nothing to lose, I decided to try it on my phone and see what happens.

Upgrade to Froyo

Android 2.2 (Froyo)

Below is how I upgraded my Nexus One to Froyo. I had a couple of failed attempts, loads of Googling but I finally managed to get it to work. Luckily a couple of days later an article appeared on Lifehacker which confirmed that the steps I took were the correct ones!

I connected my Nexus One to my computer using the USB cable.

The phone was detected and I mounted the SD card from the phone (bring down the notification area and select the USB connection – select mount to mount the SD card)

When I performed these steps I had to navigate to a URL on the Android website (see article on Lifehacker).

I copied the file and pasted it on my newly mounted drive which was the SD card of my Nexus One

The file is 45Mb and when the copy was completed, I unmounted the SD card from the computer initially (click the status area and select Safely remove <X> drive) and then from the phone (from the notification area select USB connection and select Unmount)

The update instructions below are the original work of SimonNWalker and can be referenced here.

With the update file uploaded on the Nexus One, all I had to do is shut down the device so that I can reboot it in recovery mode. The steps I took are as follows:

Figure 1: Skating Androids

I clicked and held the power button until the menu appeared. I selected 'Power Off', confirming that I wish the phone to power off.

Once the phone powered off completely, I pressed and held the trackball button down and then pressed the power button. A new screen appeared (which, like myself, you probably have never seen before) with three androids on skateboards at the bottom (Figure 1) and several options at the top (Figure 2).

Figure 2: Android FastBoot

I navigated using the power and the volume buttons. I selected the first option (Bootloader) using the volume buttons and pressed the power button to activate it.

A new menu appeared with the following options:

FASTBOOT

RECOVERY

CLEAR STORAGE

SIMLOCK

I navigated to RECOVERY and selected it

The familiar Nexus One X appeared on the screen. A few seconds later, a little android with a big exclamation mark in a triangle replaced the X.

The next step I took was to press simultaneously the power button and the volume up button.

This brought a new menu at the top (Figure 3).

Figure 3: Recovery

The options available were:

reboot system now
apply sdcard:update.zip
wipe data/factory reset
wipe cache partition

Using the trackball, I navigated to the second option and pressed the trackball button down.

At this point the update started and some information was flashing on the screen for a while, where some files were patched, some deleted, new ones copied and others replaced accordingly. The whole process took roughly 5 minutes from start to finish.

Features that I found

Please note that these are my own observations. Some of the features here might have been present in the previous version of Android (Eclair) and I simply did not notice them. If that is the case please let me know and I will correct the post accordingly.

Phone search (Search throughout the phone itself)

Figure 4: Phone Search

This is one of the updates that I just love. Froyo allows you to search a lot of content that is stored on your phone. The content includes your contacts, sms messages, applications, twitter feeds and many more. With all this power it is very easy to find the information you are looking for with the press of a few keys on the virtual keyboard (Figure 4).

Since "with great power comes great responsibility", Froyo allows you to select what can be searched (Figure 5). Go to Settings - Search - Searchable Items and you will find the following options:

- Web (web search, bookmarks and browser history)

- Aldiko (only if you have it installed)

- Apps (names of installed applications)

- Contacts (names of your contacts)

Figure 5: Search Areas

- Finance (stock tickers, company names) (only if you have it installed)

- Google Sky Map (only if you have it installed)

- Messaging (text in your messages)

- Music (artists, albums and tracks)

- Twitter (searchable tweets) (only if you have it installed)

I am sure that the search extends to other applications. Your results might vary based on the apps you have installed.

Tethering - Wi-Fi Hotspot

Figure 6: Wi-Fi Hotspot

When I first heard about this feature during the presentation of Froyo at Google I/O, I was really excited.

I spend every day 3 hours on the train, where I mostly work on my notebook. Having the ability to tether my phone and work online is essential. A couple of years back when I bought my first iPhone, I managed to jailbreak it and installed iPhoneModem on it to achieve the functionality that I wanted (tethering). It wasn't fancy (after all don't expect miracles with AT&T's EDGE network) but it worked. I could get my emails reply quickly and disconnect.

When I got my Nexus One, I bought PDA Net to achieve the same result. It worked too but again in a very basic mode but it was draining the battery very fast and was running the phone very hot.

After I upgraded to Froyo, I uninstalled PDA Net and have been working with the built in functionality ever since.

Figure 7: Configure Wi-Fi Hotspot

To access the Tethering options, you will need to go to Settings - Wireless & Networks - Tethering & portable hotspot. The options shown on Figure 6 appear, which allow you to switch the Wi-Fi hotspot on or off. A USB tethering option also exists for those that want to keep their phone charged with the power that comes from their notebook or attached device. Once the hotspot is enabled, a blue icon appears at the top notification bar. The configuration of the hotspot is really easy and can be seen in figures 7 and 8. For those wondering why I chose the SSID of my hotspot to be one of the most infectious (Windows) viruses ever, it is exactly for that reason. Apart from the WPA2 encryption, the name itself is a deterrent for anyone that might get ideas in stealing bandwidth.

Everything works perfectly apart from AT&T's EDGE network, which is really slow. The Nexus One I have can support T-Mobile's 3G network. I haven't switched to T-Mobile yet since there was no need. I fear though that I will not be able to escape the inevitable.

Figure 8: Wi-Fi Hotspot settings

On June 2nd AT&T issued a press release where they showed us once again that they don't give a damn about their customers since they lie, deceive and overcharge for a mediocre (at best) service. You can read the press release and draw your own conclusions but to me it seemed once again a slap in the face.

No more unlimited data plans for new customers, yet it is not clear what will happen to existing customers such as myself. I read somewhere in the blogosphere that those plans will remain as is, but this is AT&T that we are talking about - the same company that claims that calls to 1-800 numbers are free yet they charge you minutes for it.

I have signed for the unlimited data plan, yet AT&T does not allow me to tether. Why? What difference does it make? They still offer the same crappy EDGE network whether I watch a YouTube video on my phone or on my computer. The answer is in the press release. They want more money. As phones get 'smarter' they phone companies get greedier. John Gruber offers a good analysis of the new AT&T data plans.

I do not know how this will evolve but I will definitely continue using the Wi-Fi hotspot on my train ride, whether this will be with AT&T or T-Mobile. Perhaps if I change to T-Mobile I will be able to have better coverage. AT&T in West Virginia is not the greatest carrier. Notable is a recent phone conversation that I had with my wife while I was driving (I am wearing the headsets btw) where the line dropped 9 times :(

Phone/Web buttons at the bottom of every screen and desktop

Figure 9: Phone - Web buttons

This was something that was needed in my opinion (Figure 9). I was pressing way too many buttons to get to the phone, especially if I was not on the main screen. This little shortcut is very well received and thought of. I do not think that there is enough space for an additional 2 buttons (for the future release of Android) but you never know.

Perhaps a future version will allow you to customize those buttons.

Android development though has to be very cautious when releasing functionality. I am sure that Apple is checking everything that Android does with a microscope. With their enormous patent book near by, they will not hesitate to sue Android (or Google for that matter) for patent infringement (see Apple sues HTC).

More accounts to sync with

Figure 10: Sync Accounts

Another area that I noticed improvements, is the synchronization of the online accounts with the phone (Figure 10).

This area can be accessed via Settings - Accounts & sync settings.

I noticed that not only my regular GMail and Google Apps accounts appear but also YouTube as well as Twitter. I have installed the official Twitter application so I am pretty sure that this synchronization appears because of that application.

I have not tried it with Seesmic or any other twitter application, so if you have any additional information please let me know and I will modify this post accordingly.

I would also be very interested to know which other applications offer synchronization capabilities or take advantage of Android's synchronization API. If you have any other applications that synchronize on your phone, please let me know and I will include it in this post too.

Easy account switch (Email)

Figure 11: Email Account Switch

When I switched from an iPhone to an Android based phone, certain things were just "not right", not because there was something wrong with them - it was how I was used to things being done. One of those areas was the email. With the iPhone I was used to a traditional listing of accounts and then once something was selected I would go into the folders and then emails. If I wanted to change accounts I would have to go back two steps and then enter the account that I wanted.

This seems a very logical approach and it is easy to get used to. Although the Android has a better email management interface, it lacked the ability to quickly switch accounts and thus not spend time tapping away going back or forth. In that area the iPhone was better. Note the was. It was not because the user will tap less times, but because you would tap the back button twice which was located in the same area of the screen at all times. For the Android you had to press the menu button and then select Accounts. This was again the same amount of steps but the iPhone approach felt more natural.

With Froyo a new button appears at the top right of your email screen which will allow you to quickly go to the account selection screen. This effectively reduces the steps by one.

Analyzing briefly my emails, I can say that on average I receive 25 emails on my personal account and 35 on my business (I chose the two accounts that I get the most traffic). So if in theory I get two emails every time I check my email, that would mean that I am checking my phone 12 - 17 times a day (assuming again that I get the batches of emails on both accounts at the same time). It would therefore be safe to assume that I check my emails 15 times a day where I need to switch from one account to another.

So the math gives us:

	Taps	Average Email Checks per Day	Week	Month	Year
iPhone	2	15	210	900	10,950
Android	1	15	105	450	5,475

Clearly with the above I am using now half of the screen taps than I used to with Android Eclair or with the iPhone. However the this rough calculation shows how much I was tapping in the past prior to Froyo. Goodbye RSI. :)

Colored labels in your email (like GMail)

Figure 12: Email Colored Labels

If you are like me and use Google Apps or GMail, you are by now accustomed to the colored labels on your emails, that you don't know what you have been doing without them all this time.

With Froyo, this functionality is now available in my mobile device allowing me to visually identify emails of high interest.

For instance, Figure 12 shows my setup. As you can see I mark clients with a green label color and financial institutions (bills mainly) with a red color. When an email reaches my mailbox and is automatically labeled due to a relevant filter, I can easily identify its importance using this color coding.

Having this functionality on my mobile device is invaluable!

Easier navigation between emails - New < > buttons appear on the phone to get you from email to email

Another huge improvement in the navigation part as far as emails are concerned came in the screen where I read a specific email.

Figure 13: Email Navigation

There are two extra buttons at the bottom of that screen which allow navigation to the previous or next message (Figure 13).

Another great tap saving feature!

Better voice recognition

I had some problems with this feature from time to time primarily due to my accent – a blend of Greek – English British – German and English American. After a few tries on the new system, I can say that there is improvement since it recognized now queries that it had failed in the past.

The voice search will probably never be able to detect everything that everyone is saying due to the different accents and voices of people but it is getting pretty close to perfect in my case.

One thing that I love about voice search is the voice navigation. I have purchased the car dock for my Nexus One and I use the voice navigation almost everywhere I go. Understanding that I want to go to Rockville, MD instead of Rock Creek is awesome!

Marketplace – Update all, autoupdate for each application

Figure 14: Update All

This feature was missing and was probably one of the ones that were mostly requested by the users. The Update All to the installed applications.

Luckily the Android developers have heard our pleas and Froyo now features an Update All button at the bottom of the Downloads section in the Marketplace application (Figure 14).

Adding to this functionality, the user now has the ability to automatically update selected (or all) applications. When clicking on one of the applications to update, a checkbox appears which allows for automatic updates. If the checkbox is checked, the next time the selected application has an update, the phone will download it and install it.

There will always be a notification regarding the action in the notification area, but unless you know where the application is coming from (and you trust the source) you should keep this checked off. I know I might be getting a bit paranoid here but that is what I did.

Different notification for Text Messages (trackball)

I am not sure if this existed in Eclair but I just noticed it. When an email arrives, the trackball will start glowing briefly in regular intervals with a white color to visually notify me about the email(s) waiting for me.

If I receive a Google Voice message though, the trackball will still start glowing but this time it will be with a green color. This way I know that a text message is waiting for me.

I have sent a text message to my AT&T number and did see the trackball glowing but this time it was only white. It appears that the green trackball notification is a feature of Google Voice on Froyo or again it was always a Google Voice feature and I hadn't noticed, at which point I am getting excited for nothing :)

New icons for USB Connect/Disconnect and USB debugging

Figure 15: USB Debugging

These ones I loved them the first time I saw them. If USB debugging is enabled (Settings - Applications - Development - USB Debugging), the minute the phone is connected to the computer using the USB cable, a new icon will appear in the notification area (Figure 15). It appears that it is an android bug of sorts :)

Figure 16: USB Off

Figure 17: USB On

When the phone is connected to the computer via the USB cable, the user has the ability to use the SD card as a storage device. Once the relevant entry in the notification bar is tapped, the screen with the "Turn on USB storage" will appear. If I switch the USB storage on, the screen changes slightly (Figures 16 and 17).

These were two really cool (in my view) new screens that engage the user even more in exploring their device!

Camera control

Figure 18: Camera Controls

A new enhancement appeared in the Camera application (Figure 18).

The new menu that changes position based on the orientation of your phone (horizontal or vertical) allow for zooming, flash control, white balance control, geolocation and exposure.

The options available are:

Zoom: 1x, 1.2x, 1.4x, 1.7x, 2x
Flash mode: Auto, On, Off
White balance: Auto, Incandescent, Daylight, Fluorescent, Cloudy
Store location: Off, On
Focus mode: Auto, Infinity
Exposure: -2, -1, 0, +1, +2

Unfortunately these controls only appear when taking photos and not when shooting video. I am sure however that this functionality (and more) will be extended to the video capturing aspect of the camera application.

Select text

Figure 19: Select Text

This is an area where the iPhone was far better than the Eclair and unfortunately still is with Froyo. The gap though has decreased significantly.

With Froyo I can now select text from say an email and paste it somewhere else - even a different application. The Select Text option is hidden under the More menu button and once selected, it creates a small mouse pointer. That is the start of where it will start selecting (Figure 20).

I simply point to the top left area I want to copy from and drag my finger diagonally to end up at the bottom right of the area I want to select. This will select the text in a very appealing pink color and as soon as I lift my finger from the screen it will copy the text on the keyboard (Figure 21).

Figure 20: Mouse pointer

The technique on the iPhone is better but not by much - as I wrote the gap has decreased significantly. On the iPhone you have a magnifying glass where you can pinpoint exactly where you want to start copying (or inserting text - same functionality).

Again unfortunately this functionality (the one with the magnifying glass for selecting text) is patented by Apple Inc. and will not be seen on an Android based phone but I am sure that the Android developers will come up with something that will give us the same if not better user experience.

FLASH Support

Steve Jobs never kept a secret of his thoughts about Flash support on an iPhone. Although many people were hopeful that something will be worked out with Adobe in the end, Mr. Jobs shut the door on Adobe. Adobe then invested time and resources to work something out and right before they were able to release their solution Apple changed the iPhone Developer License Agreement to allow only applications written in Objective-C, C, C++ or Javascript and executed by the OS Webkit engine. Adobe had enough and issued a 'screw you apple' (not officially of course).

Figure 22: Flash on Android - PacMan

The presentation on Day 2 at Google I/O included the news that Flash will be allowed and supported on Android based phones that run Froyo (or newer versions).

I have to admit, the first version of Flash that I installed (it is still in BETA) was really slow. However three versions later, I am happy to announce that it works as well as a desktop machine. I have not tried to load a heavy flash based website but my brother in law's website (www.dnm.gr) loads just fine and you can see all the information that you need to see :)

I am sure that in the coming months we will see a lot more progress in that area.

Conclusions
The Android 2.2 (Froyo) is a huge step forward. It provides users with a lot of functionality that transforms a phone to a multifunction communication device. The only thing that we are missing now is proper coverage from the national carriers (AT&T this one is for you) and without having to sell our first born children to pay for the monthly bills (AT&T this one is for you too).

Update
There has been another blog post on AndroidAndMe explaining how to update Froyo to the latest build (this one is a patch not the real thing).

Update 2010-07-04
Happy 4th of July! It came with the update for Froyo (officially now) on my cellphone. I am now running the FFR91 build (and so does my wife).

2010 Memorial Day [USA][Celebrations][Personal]

2010-05-31T12:21:00.000-04:00

Today, the Memorial Day is celebrated across the USA. There are similar celebrations in Greece, not the same day as in the US but remembering and honoring the veterans for their sacrifices for the freedoms we and our families enjoy. I received an email this morning which I thought was too precious to be archived in my mail folders. Since I hate mass mailing and chain emails, I thought I would duplicate the email here for my archiving but also for everyone else to see. Please note that I am not the original author of the email and I do not know if it has appeared as a blog post or an article someplace. I do not own the images presented below either so if there is a copyright issue, please let me know and I will remove them immediately. If you know who the original author is, please let me know so that I can give them the mention they deserve.

MEMORIAL DAY

It is the VETERAN, not the preacher, who has given us freedom of religion.

It is the VETERAN, not the reporter, who has given us freedom of the press.

It is the VETERAN, not the poet, who has given us freedom of speech.

It is the VETERAN, not the campus organizer, who has given us freedom to assemble.

It is the VETERAN, not the lawyer, who has given us the right to a fair trial.

It is the VETERAN, not the politician, Who has given us the right to vote.

It is the VETERAN who salutes the Flag,

It is the VETERAN who serves under the Flag,

ETERNAL REST GRANT THEM O LORD,
AND LET PERPETUAL LIGHT SHINE UPON THEM.

No matter what the country of your origin, there are always some people that step up in difficult situations. Those men and women gladly give their lives for the better good, so that others can enjoy a better future. Those people are the Veterans that we must always remember and honor.

There is plenty of material regarding Memorial Day on the Internet. A site worth visiting is http://www.usmemorialday.org/ and two videos worth watching on YouTube are http://youtu.be/afd_sDNYbpY and http://youtu.be/wK0T4pVHP28