DrSlump @ Pollinimini.net

Zend Framework InlineScript helper hack

2011-02-08T00:00:00+01:00

Pepe -_blacky_- Jiménez has been kind enough to act as special guest in this blog and write about a hack he found, when including Javascript chunks into PHP files, while using Zend Framework’s InlineScript view helper.

If you are looking for a talented web frontend programmer I can only speak good words about this bloke.

Why this hack?

Syntax highlighting is essential for developers. If you are familiar with Zend Framework you will know that InlineScript view helper doesn’t need the <script> tag. The reason is simple, that tag is automatically included at the view rendering stage. There is one big disadvantage however when you are developing in your favorite IDE… No syntax highlighting for your JavaScript piece of code.

When I noticed that, I searched for a solution at Google. No luck this time… So I started thinking how to solve this issue. As far as I know, IDEs detect JavaScript language in source files thanks to the <script> tag, so that should be a good start. If we could find a way to keep the tag in the source file and remove it before rendering it the problem would disappear.

How it works?

The InlineScript inherits from HeadScript who is responsible for introducing JavaScript content (literal source code or file). Taking a look at the HeadScript __call method, the createData function is the best candidate to solve this tag issue. If we look at Zend Framework’s source code, the 3rd parameter is the one containing our JavaScript code (potentially including the <script> tag).

Once we have this function located is very easy to remove the <script> tag. We just need to create a custom view helper extending Zend Framework’s one in order to override the createData method. At this point take a look at the source code below to see how it works.

I hope this post will prove useful to other developers who have been in the same situation as me.

Note: Don’t forget that custom view helpers prefix and paths have to be properly configured in your application to be automatically loaded and used.

Code

    <?php
    /**
     * Remove <script> tags
     *
     * @author      Jose Luis Jiménez <jljc.work@gmail.com>
     */
    class Pepe_Zend_View_Helper_InlineScript extends Zend_View_Helper_InlineScript
    {
      /**
       * Create data item containing all necessary components of script
       *
       * @param  string $type
       * @param  array $attributes
       * @param  string $content
       * @return stdClass
       */
      public function createData($type, array $attributes, $content = null)
      {
        // Replace <script> tag
        if (!empty($content)) {
          $content = preg_replace('@^\s*<script[^>]*>@i', '', $content);
          $content = preg_replace('@</script>\s*$@i', '', $content);
        }

        // Call parent
        return parent::createData($type, $attributes, $content);
      }
    }

PHP post_max_size issue

2010-08-03T00:00:00+02:00

PHP’s ability to manage uploaded files has always left mixing feelings in me, on one hand it’s really trivial to implement file upload functionality on PHP scripts but in the other, managing big files uploads (just anything above a couple megabytes) is far from reliable.

Recently a couple solutions became popular to implement progressive uploads in PHP, PECL’s uploadprogress extension and recent APC versions. But anyway, the default PHP behaviour when uploading a file is to wait until it’s been fully transferred to the server before executing the php script.

This design decision from the PHP folks greatly simplifies the most common case, since you always get the full file when your PHP code runs, but also means that big files can cause some serious trouble. To defend itself, PHP has a few ini options that tune its ability to work with big files.

upload_max_filesize tells PHP the maximum size an uploaded file can have, files bigger will be reported as an error in the $_FILES array. max_input_time control the maximum time PHP can spend receiving data from the client, this is important because if you have many visitors with slow connections (or a malevolent attacker) a lot of PHP processes will be idling in memory collecting data slowly, consuming your server resources and potentially provoking your server to become unresponsive. Finally, we have post_max_size, which is a bit tricky, it limits the maximum amount of data a request from the client can hold. The tricky part is that if that limit is reached PHP won’t error out or signal the problem anyhow to your script, instead it will continue normal script execution but won’t populate $_POST or $_FILES superglobals, leaving them empty.

So if your post_max_size is set to 2Mb and a client uploads a 3Mb file, nor $_POST neither $_FILES will have any content. Even if you set your upload_max_filesize to 2Mb or below, if size of the file (or files) is bigger you will find yourself with those superglobals empty.

There is however a way to detect this condition, like shown in the following code snippet.

<?php

if (in_array($_SERVER['REQUEST_METHOD'], array('POST', 'PUT'))) {
  if (empty($_POST) && empty($_FILES)) {

    // Get maximum size and meassurement unit
    $max = ini_get('post_max_size');
    $unit = substr($max, -1);
    if (!is_numeric($unit)) {
      $max = substr($max, 0, -1);
    }

    // Convert to bytes
    switch (strtoupper($unit)) {
      case 'G':
        $max *= 1024;
      case 'M':
        $max *= 1024;
      case 'K':
        $max *= 1024;
    }

    // Assert the content length is within limits
    $length = $_SERVER['CONTENT_LENGTH'];
    if ($max < $length) {
      throw new Exception('Maximum content length size (' . $max . ') exceeded');
    }
  }
}

With that code in place you can gracely detect the issue and act accordingly by showing the user an error page explaining what just happened for example.

PHP command runner class

2010-05-31T00:00:00+02:00

PHP offers multiple methods to execute external applications, for simple things they are more than enough but sometimes we need more flexibility. The following class acts as a wrapper around proc_open.

It implements a fluent interface to ease its configuration and non-blocking pipes for stdout and stderr. The output can be obtained at the end of the program execution or incrementally by using a callback function.

<?php
$cmd = Command::factory('/usr/bin/svn');
$cmd->option('--username', 'drslump')
    ->option('-r', 'HEAD')
    ->option('log')
    ->argument('http://code.google.com/drslump/trunk');
    ->run();
if ($cmd->getExitCode() === 0) {
    echo $cmd->getStdOut();
} else {
    echo $cmd->getStdErr();
}

Incremental updates can be accomplished with a callback function, like in the following example (PHP 5.3+):

<?php
$cmd = Command::factory('ls');
$cmd->setCallback(function($pipe, $data){
        if ($pipe === Command::STDOUT) echo 'STDOUT: ';
        if ($pipe === Command::STDERR) echo 'STDERR: ';
        echo $data === NULL ? "EOF\n" : "$data\n";
        // If we return "false" all pipes will be closed
        // return false;
    })
    ->setDirectory('/tmp')
    ->option('-l')
    ->run();
if ($cmd->getExitCode() === 0) {
    echo $cmd->getStdOut();
} else {
    echo $cmd->getStdErr();
}

Some more features:

StdIn data can be provided to the process as a parameter to run()
Set environment variables for the process with setEnv()
Second argument to option() and argument to argument() are automatically escaped.
Options separator is white space by default, it can be changed by manually setting it as third argument to option() or setting a new default with setOptionSeparator().
The proc_open wrapper is exposed as a static method for your convenience Command::exec()

And finally the class which makes all that possible :)

Subversion svnsync post-commit hook work around

2010-05-11T00:00:00+02:00

Just a quickie for future reference. Svnsync is a tool that comes with Subversion that allows to create read-only mirrors of Subversion repositories.

Creating a mirror is quite easy and usually involves the following steps:

First we create a standard subversion repository.
```
 mkdir mirror
 svnadmin create mirror
```
Now we have to create a pre-revprop-change hook that only allows the svnsync write access to it. Remember that the mirror is read-only. Put the following contents in mirror/hooks/pre-revprop-change.
```
 #!/bin/sh

 if [ "$3" = "svnsync" ]; then exit 0; fi
 echo "Only svnsync user may edit revision properties through svnsync" >&2
 exit 1
```
To activate the hook we need to give it executable permissions.
```
 chmod +x mirror/hooks/pre-revprop-change
```

Configure the synchronization

 svnsync init --username svnsync \
     file:///path/to/mirror \
     https://url.to/repository

Finally perform the synchronization! You can put this command in the cron system for example to ensure the mirror is kept up-to-date.
```
 svnsync sync file:///path/to/mirror
```

There is a problem though with this setup if we want to use a post-commit hook. svnsync works by first committing a synced revision with its own user and then, in a second step, modifies the revision properties to change the author and date to those of the original repository. This means that post-commit hooks which need to obtain the revision author or date cannot be property run, since the reported author and date will be wrong.

The solution is to use a different hook, post-revprop-change, which is triggered each time a revision property is changed. We can have something as the following script to emulate a normal post-commit hook.

  #!/bin/sh

  REPOS="$1"
  REV="$2"
  USER="$3"
  PROPNAME="$4"
  ACTION="$5"

  # Only do actual work when the "svn:date" property is modified, which seems
  # to be the last revision property modified by the svnsync process.
  if [ "$ACTION" = "M" -a "$PROPNAME" = "svn:date" ]; then
    AUTHOR=`svnlook author $REPOS -r $REV`
    DATE=`svnlook date $REPOS -r $REV`
    LOG=`svnlook log $REPOS -r $REV`

    echo "$REPOS@$REV: $LOG by $AUTHOR at $DATE"
  fi

New header banner

2010-03-13T00:00:00+01:00

Blackies Pepe and Jdbd have provided this site with a much needed header banner.

Kudos to them!

Better keyboard handling in Terminal.app

2010-02-28T00:00:00+01:00

Apple OSX’s default terminal application, Terminal.app, has matured quite a bit over the last years. It offers a good balance between compatibility with modern shells, nice font rendering and a functional user interface with support for tabs. For people comming from other operating systems however, it’s a bit frustrating getting used to mac’s keyboard shortcuts.

The default behaviour of home, end and page keys is specially traumatic, so after googling a fair bit¹ I came up with the following recipies.

Key	Action	Behaviour
home	`\033[1~`	Move to the start of the line
end	`\033[4~`	Move to the end of the line
page up	`\033[5~`	Search previous in the history
page down	`\033[6~`	Search next in the history
option left	`\033b`	Move one word left
option right	`\033f`	Move one word right
forward delete	`\033[3~`	Delete next char

To change the behaviour of a key on Terminal.app we need to access the preferences and navigate to the keyboard tab. There we can modify and create new key mappings. To input \033 just press the Esc key.

Now for they to work properly we need configure readline by either editing as root /etc/inputrc or just creating an user file at ~/.inputrc

# Be 8bit clean
set input-meta on
set output-meta on
set convert-meta off

# allow the use of Home/End keys
"\e[1~": beginning-of-line
"\e[4~": end-of-line

# allow the use of Delete/Insert keys
"\e[3~": delete-char
"\e[2~": quoted-insert

# page up/down to search history
"\e[5~": history-search-backward
"\e[6~": history-search-forward

Time to restart your terminal session and get all the previous changes to work.

By the way, if you work with tabs there are two lesser known keyboard shortcuts to change between them: Cmd+Shift+Left and Cmd+Shift+Right will change the active tab to the left or the right.

As a bonus, we can modify all Cocoa based apps (Safari, Word, Mail …) to behave a bit more as our newly configured terminal by creating a custom keyboard binding in ~/Library/KeyBindings/DefaultKeyBinding.dict:

{
    "\UF729" = "moveToBeginningOfLine:"; 
    "\UF72B" = "moveToEndOfLine:"; 
    "$\UF729" = "moveToBeginningOfLineAndModifySelection:";
    "$\UF72B" = "moveToEndOfLineAndModifySelection:";
    "\UF72C" = "pageUp:"; 
    "\UF72D" = "pageDown:"; 
    "$\UF72C" = "pageUpAndModifySelection:"; 
    "$\UF72D" = "pageDownAndModifySelection:"; 
}

http://tech.inhelsinki.nl/gnu_developement_under_mac_os_x/
↩

Consolidate e-mail accounts in GMail

2010-02-25T00:00:00+01:00

Over time GMail has been giving options to easily consolidate email accounts with them, however they offer it under its conditions, which might not always match your requirements.

For example, they allow to fetch mail from a remote account with POP3 only, not IMAP. Moreover, the frequency by which they fetch new mails is not very reliable, going from 5 minutes to upto an hour!

Recently they introduced the posibility to use a remote SMTP server to deliver emails from consolidated email addresses, fixing the issue with Outlook mail clients displaying the infamous “on behalf of” tag. However they only support remote SMTP servers offering secured connections (TLS) or at least secured logins (STARTTLS).

To solve the mail forwarding issue, a veteran program by the name of fetchmail can be easily used. Here is my config to fetch email via the IMAP protocol and forward it using a sendmail like command line MDA tool.

In /etc/fetchmailrc

set daemon 120

poll MY.IMAP.SERVER imap
    username "MY_REMOTE_USERNAME" password "MY_REMOTE_PASSWORD"
    smtpname "MY_GMAIL_USERNAME+MY_TAG@gmail.com"
    keep
    mda '/usr/sbin/sendmail MY_GMAIL_USERNAME@gmail.com'

Fetchmail will even allow to specify which IMAP folders to fetch, so you can configure it to also forward your Sent folder for example.

Since I didn’t want to install a full SMTP server just for forwarding the email, I went with msmtp, which is a SMTP client for relaying to a SMTP mailhub or smart host. Note that we can’t use GMail’s SMTP server for this task, since it’ll rewrite the From header from the messages, meaning that all the emails will look like they come from ourselves.

In /etc/msmtprc

defaults
tls off
logfile /var/log/msmtp.log

account MY_ACCOUNT
host SMTP.HOST.NET
port 25
auth login
user SMTP_USERNAME
password SMTP_PASSWORD
from SMTP_EMAIL_ADDRESS

account default : MY_ACCOUNT

msmtp should install a symlink under /usr/sbin/sendmail so it gets somewhat compatible with it for the most simple tasks. Note that it won’t do local delivery but instead connect to the configured SMTP server to relay the email messages.

With this setup I can get new messages almost instantly in GMail. Now I needed to solve the use of a custom SMTP server in GMail, to avoid the “on behalf of” issue with Outlook recipients.

Since the SMTP server of my consolidated account didn’t offer TLS neither STARTTLS, I had to create a secure tunnel for it in my hosting server with stunnel. You can learn to generate a certificate for the SSL connections from its site and documentation. What follows is just the section of the configuration to secure the SMTP tunnel.

[ssmtp]
accept = 587
connect = REMOTE.SMTP.SERVER:25
protocol = smtp

Now in GMail settings I can simply configure a remote SMTP server for my consolidated email by using the address of the server running the tunnel on port 587, which upon a connection will proxy the request to the actual remote SMTP server.

Finally I have a really consolidated external email address in GMail. The setup is quite easy to implement and light on resources.

Migrating the blog to Jekyll

2010-01-30T00:00:00+01:00

Since I barely post anything new in this blog any more I thought that migrate it to a new system might bring some more life to it. It started years ago running on Textpattern, then I migrated to Drupal since I wanted to experiment with it a bit and now it’s being generated with Jekyll.

The former two use a database for persistence of the content while the later, Jekyll, generates static files from the content which are then uploaded to the web server hosting this blog. The change has been quite nice, since now I don’t need to login into a CMS administration web panel and get lost in the little details of each publishing system. My workflow is improved because I’m forced now to write the content using my preferred text editor, even when I’m offline, allowing me to concentrate on the content instead of the tool.

Even if publishing gets a bit more complicated, involving rsync‘ing the new content on my laptop with the published html files in the server, there is a great advantage in having a system which generates a static rendition of the site. Since no database or interpreted language is needed, the runtime can be served with a very simple and efficient setup. It’s allowed me to bring down the server steady memory usage from 120Mb of my previous LAMP setup to a mere 30Mb of the current nginx one.

Netbeans and Ubuntu 64bits

2009-11-23T00:00:00+01:00

I’ve been having some very annoying problems with Netbeans IDE running under Ubuntu Jaunty and Karmic. The Java virtual machine process would start to consume memory, probably due to a bug in Netbeans, at a very fast rate, after just 20 minutes of moderate use it would have consumed about 1Gb of RAM. This means that, on my 2GB laptop, the system starts to paginate the memory on disk, severely affecting the overall performance of the system.

It was a strange behavior since one of the reasons why I settled on Netbeans recently is that it was more responsive and less demanding on resources than Eclipse. I certainly didn’t have any problem with it on Windows (Xp and Vista) or Apple’s OSX.

The culprit seems to be the JDK used. I installed the Sun version of the JDK via the official apt repositories, since my system is 64bits it installed the 64bits version of the JDK. Unfortunately there is something wrong with it, at least when used for Netbeans, since changing to the 32bit version of the JDK seems to have solved the problem. At least the memory consumption stays below the 300Mb after a couple of hours.

These are the steps I took to install the 32bit JDK on Ubuntu:

Obtain java-package: sudo apt-get install java-package
Download the 32bit binary, just the JDK, from Sun

Create a deb package from the JDK binary we just downloaded:

 DEB_BUILD_GNU_TYPE=i486-linux-gnu DEB_BUILD_ARCH=i386 fakeroot make-jpkg jdk.xxx.bin

Install the created deb package
```
 sudo dpkg -i sun-xxx.deb
```
Now edit your netbeans.conf file and point netbeans_jdkhome to the newly installed JDK directory. You can know it by checking the installed Java environments with sudo update-alternatives -config java

Optimizing Javascript and CSS downloads with Zend Framework

2009-08-27T00:00:00+02:00

Modern web sites and applications require the use of many javascript and css files. If not done with care this can affect the user’s experience. One of the easiest methods to improve the performance is by concatenating and minimizing the individual files into bundles, reducing this way the number of downloads and the total size to be downloaded.

One problem with this approach though is that during development phases it’s important to work with the original source files, so it’s easier to debug or perform changes in them. From the developer’s point of view it’s ideal to work with small files.

Zend Framework offers an interesting way to include external javascript and css files in your pages. The headLink and headScript view helpers allow to include files in the page at any point in the execution which will be finally added to the head section of your layout just before sending the page to the browser. Since all the files are actually added to page at the end of the request we can override those helpers and filter their contents to replace the individual files with our bundles.

Fortunately, both helpers implement the Iterator interface, which means that’s very easy to attach a filter mechanism to them. First thing we need is a filtering iterator as shown here:

<?php
/** vim: ff=unix fenc=utf8 tw=80 ts=4 sw=4 et ai
 *
 * Implements a filter iterator designed to modify the default behaviour of Zend
 * view helpers HeadLink and HeadScript to support the runtime replacement
 * of individual javascript and css files for its corresponding bundles.
 *
 * License
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 * THE SOFTWARE.
 *
 * @copyright   Copyright (c) 2009 Iván Montes
 * @author      Iván -DrSlump- Montes <drslump@pollinimini.net>
 */
class BundlesFilterIterator extends FilterIterator
{
    /** @var array */
    protected $_files = array();
    /** @var array */
    protected $_included = array();

    /**
     * Constructor
     *
     * @param Iterator $iterator
     * @param array $bundles
     */
    public function __construct($iterator, $bundles)
    {
        // Reverses the array placing the individual files as keys and the
        // bundle files as values
        foreach ( $bundles as $bundle => $files ) {
            foreach ($files as $file) {
                $this->_files[$file] = $bundle;
            }
        }

        parent::__construct($iterator);
    }

    /**
     * Performs the filtering
     *
     * @return bool
     */
    public function accept()
    {
        // Get the current item
        $item = $this->getInnerIterator()->current();

        // Check if it is an external javascript or css file
        $type = 'js';
        $src = isset($item->attributes['src']) ? $item->attributes['src'] : null;
        if (empty($src)) {
            $type = 'css';
            $src = isset($item->href) ? $item->href : null;
        }

        if (!empty($src) &&
            // The individual file is in a bundle
            isset($this->_files[$src]) &&
            // Check if it is a normal include
            empty($item->attributes['defer']) &&
            empty($item->attributes['conditional']) &&
            empty($item->attributes['conditionalStylesheet'])) {

            // Get the bundle file
            $bundle = $this->_files[$src];

            // If its bundle was not yet included do so now
            // The HeadXXXX helpers will also check for this but still...
            if (empty($this->_included[$bundle])) {
                $this->_included[$bundle] = true;
                if ($type === 'js') {
                        $item->attributes['src'] = $bundle;
                } else {
                        $item->href = $bundle;
                }
                return true;
            }

            // Otherwise we do nothing with this item
            return false;
        }

        return true;
    }
}

Now we need to override the view helpers to add the filtering iterator

<?php
class DrSlump_View_Helper_HeadLink extends Zend_View_Helper_HeadLink {
    /**
     * Override the default iterator with a filtering one
     *
     * @return BundlesFilterIterator
     */
    public function getIterator()
    {
        $iterator = $this->getContainer()->getIterator();
        $bundles = Zend_Registry::get('bundles');

        return empty($bundles) ? $iterator : new BundlesFilterIterator( $iterator, $bundles );
    }
}

class DrSlump_View_Helper_HeadScript extends Zend_View_Helper_HeadScript {
        /**
         * Override the default iterator with a filtering one
         *
         * @return BundlesFilterIterator
         */
        public function getIterator()
        {
                $iterator = $this->getContainer()->getIterator();
                $bundles = Zend_Registry::get('bundles');
 
                return empty($bundles) ? $iterator : new BundlesFilterIterator( $iterator, $bundles );
        }
}

Now we just need to map our small individual files with the bundle files in which they are contained:

<?php
$bundles = array(
    // For jQuery we use a public CDN
    'http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js' => array(
        '/js/jquery-1.3.2.js'
    ),
    '/resources/bundle-plugins.js' => array(
        '/js/jquery/jquery.event.drag.js',
        '/js/jquery/password_strength_plugin.js'
    ),
    '/resources/bundle.css' => array(
        '/css/reset.css',
        '/css/global.css',
        '/css/forms.css'
    )
);

// Store the bundles configuration in the registry so that it can be fetched by the view helpers
Zend_Registry::set('bundles', $bundles);

When defining the bundles mapping we’re assigning the bundle URL as key to an array of the individual files URLs we’re using in the code.

Note: This code does NOT create the bundles. There are several tools available for that task. The purpose of the examples above is to show how to use those bundles in an application without having to think about them when coding or even apply them to an existing application without having to modify the source code.

Runtime, annotations based, lexer generator

2009-08-14T00:00:00+02:00

Just a quick post to publish this class. It allows to easily create lexers (or lexical scanners or tokenizers) by either extending the class or pointing it to a class or object.

Takes advantage of PHP’s Reflection API to explore the document blocks of your class looking for special annotations describing patterns (regular expressions) and rules.

Some features:

“First to match” and “longest match” lexing modes.
States based lexer with support for a states stack
Rules can consume, ignore, repeat in a different state or repeat skipping the rule just matched
Implements the iterator interface so it can be used in foreach() loops for example
Reports lexing failures indicating the position in the text plus the line and column
Caches the internal lexer information so the annotations are only parsed once
Allows to save and restore the internal lexer information so it can persisted and cached between requests

Finally HTML5 goodness for the real world

2009-04-10T00:00:00+02:00

Firefox 3.1b3 has arrived and it’s stuffed with a ton of new web technologies. Most of the stuff comes from the WhatWG working drafts, which should mean that major browser vendors (Mozilla, Microsoft, Opera, Apple and Google) are aware of the specifications and will incorporate them at some point.

I love this release of Firefox because it is really a giant step forward for web developers, stuff like web workers (a simple threading scheme for web applications), cross site XMLHttpRequest via access control in the server, native JSON support (really fast!), Drag-n-Drop events and Offline persistance.

Together with the inclusion by default of the TraceMonkey javascript engine, these features put Firefox again as the ultimate web development platform.

My new programming font: Inconsolata

2009-03-16T00:00:00+01:00

Just a short post to announce a jewel I’ve just found. The Inconsolata font is a great monospace font suitable to use in text terminals and code editors or IDEs.

It’s replacing now the ancient Mac’s Monaco which has served damn well over the years but I feel like I needed a change and Inconsolata feels bolder and more vibrant than Monaco or Microsoft’s Consolas.

Update: This font was initially conceived for printing so it doesn’t perform well at small sizes, so to get it at its full glory I recommed a size of at least 12 points over a dark background.

A solution to PHP's new namespace separator moaning

2008-11-09T00:00:00+01:00

I’ve been quite busy lately but I’ve obviously been exposed to all the fud about the PHP internals decission of choosing the back-slash as the namespace separator.

So to keep everyone happy I’ve spent a few minutes working on a solution to the problem and here it goes:

<?php
class NsFixerFileStream {
    protected $fp;
    protected $separator = ':::';
    protected $isPHP = false;
    public $context;

    function stream_open($path, $mode, $options, &$opened_path)
    {
        if (!$this->context) {
                $this->context = stream_context_get_default();
        }
        $opts = stream_context_get_options($this->context);
        if (!empty($opts['file']) && !empty($opts['file']['separator'])) {
                $this->separator = $opts['file']['separator'];
        }

        $this->isPHP = pathinfo($path, PATHINFO_EXTENSION) === 'php';

        stream_wrapper_restore('file');

        $this->fp = fopen($path, $mode);

        stream_wrapper_unregister('file');
        stream_wrapper_register('file', get_class($this));

        return !empty($this->fp);
    }

    function stream_read($count)
    {
        $data = fread($this->fp, $count);
        if ($this->isPHP && !empty($data)) {
                $data = str_replace($this->separator, '\\', $data);
        }

        return $data;
    }

    function stream_write($data)
    {
        return fwrite($this->fp, $data);
    }

    function stream_tell()
    {
        return ftell($this->fp);
    }

    function stream_eof()
    {
        return feof($this->fp);
    }

    function stream_seek($offset, $whence)
    {
        return fseek($this->fp, $offset, $whence);
    }

    function stream_stat()
    {
        return fstat($this->fp);
    }
}


// Replace default stream wrapper with out custom one
stream_wrapper_unregister('file');
stream_wrapper_register("file", "NsFixerFileStream")
    or die("Failed to register protocol");

// Define the default namespace separator
stream_context_set_default(array(
        'file' => array(
                'separator' => ':::'
        )
));

For the least versed on PHP’s stream wrappers, what it does is replace the standard file i/o stream with one that replaces on the fly everything you like to be a namespace separator with an ugly back-slash. Just place this on a file on your server and add its location to your auto_prepend_file ini directive.

Now you can spend your time writing stuff like the following instead of blogging about how the ugly back-slash in your code can ruin your way of life :)

<?php
namespace DRSLUMP:::test;

class foo {
    static public function bar(){
        :::printf('Hello %s!', 'world');
    }
}

:::DRSLUMP:::test:::foo::bar();

Tags emulation for Zend_Cache

2008-08-06T00:00:00+02:00

Managing a cache is a complex task, at first you start placing random stuff on it, when you need to squeeze a few more hits per second you keep adding stuff to the cache. At the end of the day, if you’ve not planned it carefully, you might end up with a nightmare of stale data in your pages or even a completely broken site.

One easy way to apply some logic to the cached data and help in the invalidation of cached stuff is the use of tags. Cache tags are a simple way to group cached items so it gets easy to remove/invalidate them if needed. For example, if we have an online shop and we are caching paginated list of items by their category, then we might have keys like list-{category}-{startOffset}, if we add a new item to our shop we have to invalidate all the cached partial lists (since it could be at any position in the list). If we apply a simple tag, like list-{category} to the cached lists we can group them and proceed to their invalidation when a new product is added or moved to a given category, so our online data is always fresh.

The Zend Framework implements tag support in its cache component, but unfortunately not APC neither the high performance Memcached do support grouping or tags. For the later one there is a fork/patch, memcached-tag, however it does not seem to be very mature yet as to use in production.

Still not everything is lost, we can emulate tag support by keeping ourselves a relation between cache keys and tags. The first thing that comes to mind is to self-store this relation in the cache, however APC and Memcached may drop a cache item without further notice, they are caches not persistency layers, so we need to find a persistent store to keep this information. The solution below uses a PDO connection (MySQL, Sqlite, Oracle…) to keep the relation between keys and tags in a table, this table is updated each time there is a cache write (with tags) and when cleaning by tag, so it shouldn’t hurt much performance if your cache hit is high (the overhead won’t matter much anyway if you get a lot of misses).

By the way, the code is licensed under the BSD License, so feel free to use it at will as long as you keep the copyright notice in place.

Download it from my public subversion repository.

If we only have a single server we can use a simple Sqlite database. For distributed memcached servers we’ll need a server based database system like MySql (Note: To improve performance in MySQL it’s desirable to use the Memory table engine if we don’t need the cache to survive our MySQL instance).

Following is a simple usage example for the class (Remember to rename the file to the usual ZF directory structure Zend/Cache/Backend/Tags.php).

<?php
// Setup the cache
$frontOpts = array(
   'lifetime' => 3600
);
$backOpts = array(
  // Setup the real cache backend
  'backend' => array(
    'class' => 'Zend_Cache_Backend_Apc',
    'options' => array(
    ),
  ),
  // Configure the persistent store for the keys-tags mapping
  'pdo' => array(
    'dsn' => 'mysql:host=127.0.0.1;dbname=cachedb',
    'user' => 'dbuser',
    'password' => 'dbpass',
  ),
  // Set to true if you want to delete non tagged keys
  'track_untagged' => false,
  // Set this to true if you don't want to have duplicates in the table,
  // it could improve performace a bit
  'duplicates_ok' => false
);

// Create the cache object
$cache = Zend_Cache::factory( 'core', 'tags', $frontOps, $backOps );

// Store some stuff in the cache
$cache->save( 'My cached data #1', 'cacheA', array('tagA') );
$cache->save( 'My cached data #2', 'cacheB', array('tagA') );
$cache->save( 'My cached data #3', 'cacheC', array('tagB') );

// Clean all the keys with tagA
$cache->clean( Zend_Cache::CLEANING_MODE_MATCHING_TAG, array('tagA') );

Update 2009-03-07

Jan ‘Mithrandir’ Pedersen has provided in the comments a modification of the class based on Zend_Db_Table instead of PDO. Here is the code:

<?php
/**
 * Zend_Cache Tags emulator for backends not supporting them natively
 *
 * It'll use a Zend_Db_Table_Abstract object to persist the mapping
 * between cache keys and its tags.
 *
 * @author ZooCMS Crew
 * @credits Iván -DrSlump- Montes <drslump AT pollinimini DOT net>
 * @copyright Copyright (c) 2009 ZooCMS Crew
 * @copyright Copyright (c) 2008 Iván -DrSlump- Montes <http://pollinimini.net>
 * @version 1.0
 * @license BSD
 */

class Zoo_Cache_Backend_Tags extends Zend_Cache_Backend implements Zend_Cache_Backend_Interface
{
    /**
     * Available options
     *
     * =====> (Zend_Cache_Core) backend:
     * A Zend_Cache_Backend object or an associative array declaring its configuration:
     * 'class' => (string) The full class name of the backend
     * 'options' => (array) The array of options passed to the class constructor
     *
     * =====> (array) table:
     * An associative array with the Zend_Db_Table_Abstract class data:
     * 'class' => string) The full class name of the backend
     * 'options' => (array) additional configuration parameters to use with Zend_Db_Table_Abstract::__construct()
     * It is assumed that the table will have an 'cacheid' and a 'tag' column
     *
     * =====> (book) track_untagged:
     * If true it will keep track of untagged items so they can be purged by the NOT_MATCHING_TAG mode
     *
     * =====> (book) duplicates_ok:
     * If true it won't make sure that there are no duplicate entries in the database.
     *
     * @var array available options
     */
    protected $_options = array(
        'backend' => null,
        'table' => null,
        'track_untagged'=> false,
        'duplicates_ok' => false
    );

    /**
     * Zend_Cache_Core object
     *
     * @var Zend_Cache_Core object
     */
    private $_backend = null;

    /**
     * Table object
     *
     * @var Zend_Db_Table_Abstract object
     */
    private $_table = null;

    /**
     * Constructor
     *
     * @param array $options associative array of options
     * @throws Zend_Cache_Exception
     * @return void
     */
    public function __construct($options = array())
    {
        parent::__construct($options);

        if ( is_array($options['backend']) ) {

            $backendClass = $options['backend']['class'];

            $this->_backend = new $backendClass($options['backend']['options']);

        } else if ( $options['backend'] instanceof Zend_Cache_Backend_Interface ) {

            $this->_backend = $options['backend'];

        } else {
            Zend_Cache::throwException('The backend option is not correctly set!');
        }

        if (!is_array($options['table'])) {
            Zend_Cache::throwException('The table option is not correctly set!');
        }
    }

    /**
     * Test if a cache is available for the given id and (if yes) return it (false else)
     *
     * @param string $id Cache id
     * @param boolean $doNotTestCacheValidity If set to true, the cache validity won't be tested
     * @return string|false cached datas
     */
    public function load($id, $doNotTestCacheValidity = false)
    {
        $result = $this->_backend->load( $id, $doNotTestCacheValidity );

        // If not in cache make sure to delete the tag mapping to handle cache expiration
        if ( $result === false && !$this->_options['duplicates_ok'] ) {
            $where = $this->getTable()->getAdapter()->quoteInto('cacheid = ?', $id);
            $this->getTable()->delete($where);
        }

        return $result;
    }

    /**
     * Get Zend_Db_Table_Abstract object for persistence
     */
    protected function getTable() {
        if (!$this->_table) {
            $tableClass = $this->_options['table']['class'];
            $this->_table = new $tableClass($this->_options['table']['options']);
        }
        return $this->_table;
    }

    /**
     * Test if a cache is available or not (for the given id)
     *
     * @param string $id Cache id
     * @return mixed|false (a cache is not available) or "last modified" timestamp (int) of the available cache record
     */
    public function test($id) {
        return $this->_backend->test($id);
    }

    /**
     * Save some string datas into a cache record
     *
     * Note : $data is always "string" (serialization is done by the
     * core not by the backend)
     *
     * @param string $data Datas to cache
     * @param string $id Cache id
     * @param array $tags Array of strings, the cache record will be tagged by each string entry
     * @param int $specificLifetime If != false, set a specific lifetime for this cache record (null => infinite lifetime)
     * @return boolean True if no problem
     */
    public function save($data, $id, $tags = array(), $specificLifetime = false)
    {
        $result = $this->_backend->save( $data, $id, array(), $specificLifetime );

        if ( $result ) {
            // Makes sure to always store the key for notMatchingTag cleaning mode
            if ( $this->_options['track_untagged'] && !count($tags) ) {
                $tags = array('');
            }

            if ( count($tags) ) {
                foreach ( $tags as $tag ) {
                    $result = $this->getTable()->fetchAll(
                                $this->getTable()->select()
                                    ->where('cacheid = ?', $id)
                                    ->where('tag = ?', $tag)
                    );

                    if (!$result->count()) {
                        $row = $this->getTable()->createRow(array('cacheid' => $id, 'tag' => $tag));
                        $row->save();
                    }
                }
            }
        }

        return $result;
    }

    /**
     * Remove a cache record
     *
     * @param string $id Cache id
     * @return boolean True if no problem
     */
    public function remove($id)
    {
        if ( !$this->_options['duplicates_ok'] ) {
            $where = $this->getTable()->getAdapter()->quoteInto('cacheid = ?', $id);
            $this->getTable()->delete($where);
        }

        return $this->_backend->remove($id);
    }

    /**
     * Clean some cache records
     *
     * Available modes are :
     * 'all' (default) => remove all cache entries ($tags is not used)
     * 'old' => remove too old cache entries ($tags is not used)
     * 'matchingTag' => remove cache entries matching all given tags
     * ($tags can be an array of strings or a single string)
     * 'notMatchingTag' => remove cache entries not matching one of the given tags
     * ($tags can be an array of strings or a single string)
     *
     * @param string $mode Clean mode
     * @param array $tags Array of tags
     * @return boolean True if no problem
     */
    public function clean($mode = Zend_Cache::CLEANING_MODE_ALL, $tags = array())
    {
        if ( $mode == Zend_Cache::CLEANING_MODE_MATCHING_TAG ||
             $mode == Zend_Cache::CLEANING_MODE_NOT_MATCHING_TAG ) {

            if ( !is_array($tags) ) {
                $tags = array($tags);
            }

            // Quote the tags and convert to a string
            $tags = array_map(array($this->getTable()->getAdapter(), 'quote'), $tags);

            // Build the query for the desired matching mode
            if ( $mode == Zend_Cache::CLEANING_MODE_MATCHING_TAG ) {
                $items = $this->getTable()->select()->where('tag IN (?)', $tags);
            } else {
                $items = $this->getTable()->select()->where('tag NOT IN (?)', $tags);
            }

            foreach ( $items as $item ) {
                $this->remove( $item->cacheid );
            }

            return true;

        } else {

            return $this->_backend->clean($mode, $tags);

        }
    }


    /**
     * Magic call handler to forward to the real backend the method calls
     *
     * @param string $method The method called
     * @param array $args The arguments used packed as an array
     * @return mixed
     */
    public function __call( $method, $args )
    {
        return call_user_method_array( $method, $this->_backend, $args );
    }
}

A really simple include filter for Drupal

2008-08-06T00:00:00+02:00

I’ve been using Texy! lately to write in my Drupal powered blog, it’s a great Markdown/Textile replacement and it even has a syntax highlighting addon which helps a lot to publish source code in the posts.

I’ve found however a problem. I usually keep my source files in my subversion repository and sometimes I want to show the syntax highlighted code of one of those files in the post. I was getting tired to copy-pasting the contents so I’ve implemented a dead simple Drupal input filter which dynamically includes a local or remote file as simple text in the content. I guess it can be useful too without Texy!

The syntax is very easy to use and remember:

{include http://myserver.com/myfile.ext}

The module will use the Curl extension if available to fetch remote files, so it can work on hosts like DreamHost, which don’t allow URLs via nomal file functions.

Here is the .info

name = Include Filter
description = This module allows to include a text file in a node.
package = Filters
version = 1.0
core = 6.xCopy to clipboard

And the .module

<?php

function includefilter_filter($op, $delta = 0, $format = -1, $text = '') {
    switch ($op) {
        case 'list':
            return array('Include filter');

        case 'description':
            return t('Allows to include a text file in the content. Give it a negative weight so it\'s processed before all other filters');

        case 'prepare':
        case 'process':
            $text = preg_replace_callback('!{include\s+([^}]+)}!', '_includefilter_replace', $text);
            return $text;

        case 'no cache':
            return false;
        default:
            return $text;
    }
}

function _includefilter_replace($m) {
    if ( preg_match('!^https?://!i', $m[1]) && function_exists('curl_init') ) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $m[1]);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
        $txt = curl_exec($ch);
        curl_close($ch);
    } else {
        $txt = @file_get_contents($m[1]);
    }

    if ( $txt === false ) {
        return '*IncludeFilter: Error including "' . $m[1] . '"*';
    }

    return $txt;
}

Note: Remember to give this filter a negative weight so it gets executed before any other filter in your input format.

RISON, a compact JSON specially suited for URIs

2008-07-28T00:00:00+02:00

I was looking for a way to efficiently pass data between cross-domain iframes using the location hash trick. The first thing that I checked was JSON but the payload gets huge when url encoded, then I tried with base64 encoding which helps but not very much. Finally I found RISON which in short is nothing more than a reformulation of JSON’s control characters to make them more URI friendly.

It’ll basically use ( instead of {, !( instead of [ or ! instead of \ to escape stuff. The result is quite good with savings of a 50% on average when url encoding the result. The only problem I had was that the source code no longer seems to be maintained and the only copy I could find was in the repository of a project in google code.

So I’ve implemented my own version taking as example JSON’s official reference implementation API and rolling in a really simple token based parser. It passes all the test fragments from RISON’s home page and the performance is quite good for moderately sized payloads.

The code is licensed under the liberal MIT License and can be downloaded from my subversion repository.

Nokia's N810 on the mail

2008-06-14T00:00:00+02:00

After being trying to get an iphone for a couple of months and haven’t succeed at it, I sat down and did an extensive research on the market offerings for my needs. Finally I found what I needed and to my surprise it wasn’t the phone with the Apple logo.

I spend more than two hours each day on the train which gets absolutely packed at rush hours, so my plan to work on the laptop while traveling is not a feasible option (Even with my smallish Dell XPS M1330). Checking email and rss feeds on my Sony Ericsson K800i cellphone works but with the huge amount of tunnels in the trip the data connection keeps falling and the screen is not that big as to make it a nice experience. So I decided to try with the Apple’s phone which should offer a great experience while on the move with its large touch screen and the ability to sync imap accounts for offline reading of the inbox and nice rss reading. Since I couldn’t obtain one I searched on the net for alternatives and I felt in love with Nokia’s N810 internet tablet device.

The N810 is not a smart phone, it just offers Wifi and Bluetooth radios so it needs to tether with a phone to access mobile data networks. It does offer however a wonderful touch screen sporting a 800×480 resolution with an amazing 225dpi (quite good for ebook reading), a full qwerty keyboard and a Linux based (Maemo) operating system with a ton of available third party application and games. What I like the most about the device is that its environment is really open, which will allow me to tinker with it and suit it best to my needs.

So yesterday I placed the order on Nokia’s online shop for it and I should be enjoying this wonderful device next week. A big big big thanks to Cristina for giving me this great present!

A web development environment (III) - Firefox

2007-12-26T00:00:00+01:00

Firefox is a great browser and is also a great development tool in itself. There are some problems though with the numerous extension available to developers. They are a great resource when programming but they slow down (and in some scenarios they might even break) our normal web browsing.

So the idea is to have a normal Firefox installation to use as our browser and then a separate install (and profile) used only for development purposes. This way we can install tons of development related addons without worrying about slowing down our normal browsing. Of course this is only useful if you use Firefox as your primary browser.

Download the Firefox installer
Launch the installer
Set the destination folder to c:\dev\Develfox
Uncheck the options to create shortcuts on desktop, start menu…
Uncheck the option to launch Firefox on finishing the setup
Go to the installation directory and start a command prompt there
- Run Firefox with the following command: firefox.exe -no-remote -ProfileManager
  - Create a new profile and name it Devel
  - Exit the profile manager
- Create a new batch file name develfox.bat with the following contents. It’ll copy on each run the firefox executable to develfox.exe so we can tell it apart from our normal browser on the task manager (we copy it on each run so that automatic updates are correctly applied).
```
@echo off
copy "c:\dev\develfox\firefox.exe" "c:\dev\develfox\develfox.exe" > null
start "" "c:\dev\develfox\develfox.exe" -no-remote -P "Devel"
```
Still on the installation directory go to the chrome folder and create in there a directory named icons and inside that one another called default. Put in there a custom icon with the name main-window.ico to be used as the application icon. Check out IconBase for some nice ones.
- Create a shortcut on your desktop or another suitable place and use the following properties:
  - Target: c:\dev\develfox\develfox.bat
  - Start in: c:\dev\develfox
  - Change the icon to c:\dev\develfox\chrome\icons\default\main-window.ico
- Launch the shortcut to execute our new Firefox installation. It’ll be a vanilla one so now it’s time to install our development addons ( Firebug, Webdev toolbar, Yslow, Clearcache, Venkman… ) and also a custom theme so we can easily know in which firefox we are at each moment.
- To further customize our new Firefox installation we can install the Firesomething addon, which allows to change the window title to anything we want.

Update 2008–11–03

A great extension for Flash/Flex development is FlashSwitcher which allows to easily change the Flash player version used in Firefox. One problem though is that it’ll interfere with your default Firefox installation. To solve this you can edit its settings dialog and point the Firefox Plugin directory to c:\dev\develfox\plugins instead of the default location. This way the Flash player changes will only affect our development firefox installation.

A web development environment (II) - Cygwin

2007-12-26T00:00:00+01:00

Following with the series I will introduce in this chapter Cygwin.

Cygwin is an excellent collection of Unix utilities and programs ported to Windows. While its emulation of an Unix like environment is quite good I actually don’t use it. I install it just for the helpful utilities it has. Although I do occasionally use its shell to perform some command line tasks which on Windows’ cmd.exe would be a real nightmare to perform.

Lets set it up and then we can see some tips and tricks.

Download the Cygwin installer
Move the installer (setup.exe) to c:\dev\cygwin and launch it
Set the Root Directory to c:\dev\cygwin and the Local Package Directory to c:\dev\cygwin\packages
Choose at least the following packages to install. You won’t probably need all of them but in my experience these are the most commonly used and they don’t take much space actually:

Admin

shutdown

Archive

unzip, zip

Base

bash, findutils, grep, gzip, run, tar, sed, termcap, terminfo

Database

sqlite3

Devel

binutils, automake, cvs, cygport, gcc, make, subversion

Editors

nano, vim

Graphics

ImageMagick

Interpreters

m4, perl, python, ruby

Net

curl, openssh, openssl, rsync

Shells

bash, bash-completion

Text

less, tidy

Utils

bzip2, cygutils, diffutils, gnupg, patch, scree

Web

cadaver, curl, httping, links, lynx, wget
Now wait for all packages to download and install (it will take a while)
Once the installation is complete we can launch the Cygwin system by running c:\dev\cygwin\cygwin.bat
On the first run it’ll create a new configuration and home directory for our user. Follow the instructions given on the terminal to complete the installation.

Ok, so Cygwin is now setup and it’s time to learn a few tips.

On a standard installation Cygwin will create a home directory for the user in c:\dev\cygwin\home\[USER]. However we might want to re-use our Windows folder so we have a unique location for our stuff. To do it we just need to create an environment variable (in System Preferences) called HOME with the path to our profile folder (c:\Users\[USER] in Vista). Now we have to edit /etc/passwd and change the home folder to our Windows profile one (/cygdrive/c/Users/[USER]), we can move the contents of the old home folder to the new one with the following commands:

$ mv /home/[USER]/* /cygdrive/c/Users/[USER]/.
$ mv /home/.* /cygdrive/c/Users/[USER]/.

By default the Windows drive letters are accessible in /cygdrive/[DRIVE], that is ok but it’s used often so we can save a few key strokes by creating symlinks at the root of the file system. Create one for each of your drive letters so you can access them as /[DRIVE]/my/path/to/a/file.

$ ln -s /cygdrive/c /c

As we have seen Cygwin uses Unix like paths so to launch Windows programs from the shell we have to make an extra step and that is to convert between paths. Fortunately there is a little tool called cygpath. When used without modifiers it will translate a Windows path to a Cygwin one and when used with the -w or the -m modifiers it will do the opposite conversion.

Another nice tool is cygstart which launches a program (or the default program associated with a document). The good about it is that it will launch the program and detach the process from the console.

For commonly used programs it’s a bit uncomfortable to use cygpath and cygstart, like for example to launch our favourite text editor. In this example I’m going to use Notepad as an example. We just have to change our bash init script (~/.bashrc) to include a function to launch Notepad easily from the command line. Add the following to your .bashrc file:

notepad () {
    cygstart "c:/Windows/System32/notepad.exe" "`cygpath -w $1`"
}

Now we can edit the contents of a text file just doing notepad /var/log/setup.log. This is a great way to really integrate Cygwin with Windows, which actually makes the Cygwin system ready for real work.

Another useful utility is cdargs which while not an official part of the Cygwin system can be compiled from source. To install it we just have to download its latests source tarball and unpack it into your Cygwin home directory. Then issue the following commands:

$ cd cdargs-1.XX
$ ./configure
$ make
$ make install
$ cp contrib/cdargs-bash.sh /etc/profile.d/.
$ source /etc/profile.d/cdargs-bash.sh

Explaining how to use this tool is beyond the scope of this article. There is a nice mini tutorial at Linux.com and Google will give you plenty of information about this tool.

There is a lot of software compatible with Cygwin beyond the packages included in the official repository. The Cygwin Ports project offers a few dozens of applications already compiled and ready to be installed. It’s well worth a look. Besides there are many Unix/Linux applications which can be compiled to be used in Cygwin even if they don’t redistribute the binary packages.

A web development environment (I) - Introduction

2007-12-26T00:00:00+01:00

This will be a series of posts dedicated to document my current development setup. This setup tries to be the one of a Ninja. It won’t end up with a bunch of boring application windows typical of a bald engineer. It also won’t be a group of black terminal windows running vi as a pseudo operating system like the average geek wannabe. I’m in the search of the Ninja feeling, with white on black text editors, translucent terminals and a dose of eye candy so that I feel special without having to resort to browsing youtube with Lynx.

After many years I am finally happy with my current development setup. It fits my needs and this series of articles should serve myself as a reminder on how to setup the different pieces. However I guess that it will also be of inspiration for other developers who haven’t already found their peace of mind with their environment.

I’ll focus on Microsoft Windows systems but most of the stuff can be easily implemented in other operating systems. Actually much of the things I do for windows come standard in several operating system distributions, it’s just that I happen to like Windows (which obviously harms my geek status).

The main building blocks for this environment are the following ones:

Cygwin: A collection of very useful Unix utilities for Windows.
coLinux: Wonderful virtualization software to run Linux side by side Windows.
Debian: The only Linux distro I know a bit about. It’s fantastic so I don’t have the need to search for a better one.
PHP5: Currently the programming language of my choice.
Console2: An excellent terminal emulator with lots of eye candy.
Firefox: The browser that came to rule them all.
A lightweight text editor: Everyone has its favorite
A heavy IDE: Komodo and Eclipse to the rescue
Several utilities to ease our lives: TortoiseSVN, kdiff3, wincachegrind, total commander…

Before we begin I would like to note the following:

All tools are installed in C:\dev. I don’t like polluting the standard Windows installation with the development tools. Besides, it allows to easily backup most of the environment.
I’m using a 32bit version of Windows Vista since coLinux is not compatible with 64bit versions.

Xdebug bookmarklet

2007-11-26T00:00:00+01:00

Nothing really new since there is already a firefox extension which does just the same, but I also wanted to support other browsers.

So this bookmarklet toggles the cookie Xdebug uses to check if it should debug the running script or not!.

Toggle Xdebug

Drag it to your bookmarks (or favorites or whatever your browser calls it) toolbar so it’s easy to access. Assign it a keyboard shortcut, if your browser has that function, and you’re done.

By default it’ll prompt for the idekey to be used by Xdebug, it’s a good idea to modify the idekey variable at the start of the bookmarklet’s source with your idekey or with default if you’re not using a DBGp proxy.

Introducing DBGp Path Mapper intercepting proxy

2007-11-25T00:00:00+01:00

My current IDE of choice, Komodo, brings native support for the DBGp protocol to debug source code either locally or from a remote server. ~~It has however some issues with its path mapping implementation~~¹.

Since it’s not the first time I stumbled upon this kind of issue (my previous Eclipse PDT experience was troublesome too) I finally tried to solve it. The solution is an intercepting proxy server which listen for connection from the debugger to modify the messages it sends to the IDE with custom path mappings.

To define the path mappings we use a file with a set of paths on each line. The path tuples are separated by the => operator, placing the path understood by the debugger on the left side and on the right the path as understood by the IDE. The file looks like this.

# Mapping for projects
file:///var/www/projects => file:///c:/Users/DrSlump/projects
# this one is for my subdomains
file:///var/www/subdomains => file:///c:/subdomainsCopy to clipboard

Note that the path mappings are case insensitive to avoid problems on Windows.

To run the path mapper server we just need to call it from a terminal like so:

$ php5 dbgp-mapper.php -i 192.168.0.5 -m dbgp.mapCopy to clipboard

The -i option specifies the IP or host of the computer where the IDE is running, while the -m option points to the file where the path mappings are defined. There are also other options which you can check by seeing the built in help using the -h argument.

By default it will bind itself to the port 9000, which is DBGp default one, so if you are running this on the same computer as your IDE you will have to change the port either in this tool or on your IDE.

The source is available via this subversion repository

It seems that Komodo’s uri mapping works properly after version 4.2.0. In my case I was having problems but after rebooting the system it started to work properly.
↩

TextMate, the myth

2007-11-02T00:00:00+01:00

Source code editing and programming tools in general are one of those things I like to be up to date with, you never know when you’re going to find a killer tool which will boost your productivity.

For a couple of years I keep hearing about an awesome editor for Macs called TextMate. People even comment on forums and blogs that they are thinking to switch to Mac just to use it.

Well, I switched to Mac a few weeks ago and one of the first things I installed was this TextMate editor. Given all the hype around it I thought it was going to be my new favourite editor. However there is a lot of myth around this software. For the last year I’ve been using Eclipse on Windows as my heavy work IDE with Notepad++ as casual editor for quick operations. Now on the Mac Eclipse runs quite slowly so I’ve been using TextMate a lot in the last days and while it’s a great replacement for Notepad++ it doesn’t come close to a real IDE. The Bundles are nice and expose a lot of possibilities but they are useless to work on a big project.

When you have to deal with a few hundreds library files (Zend Framework, Doctrine, Pear…) plus another few hundreds of the project then there is nothing else to do that to use a real IDE with professional tools like integrated debugger, a real code parser, a code explorer and powerful refactoring tools (at least a good find and replace dialog with regular expression support). Then there are some extras like bracket pairing, code versioning, code formatting and smart indentation.

Sure some of you could argue that TextMate’s power is on its snippets/bundles customization but lets be serious. I don’t mind writing function or class constructs continuously, it can get tedious but what really hurts your performance is having to leave the file your are editing because you don’t remember the exact spelling of an obscure method from a rarely used class, or you can’t recall what was the order of the arguments for that function that you checked just two hours ago.

Given that Eclipse is running like a dog on my system I’ve searched for an alternative and to my surprise I’ve found a killer one. ActiveState’s Komodo IDE, which I had trialed a few months ago in Windows but didn’t really paid attention to it. It’s dialogs are a bit buggy, the default colour schemes are quite ugly, it takes ages to load, is very expensive and it’s not the easiest to configure to a point where it’s useable but… it’s really customizable and has a ton of features out of the box. Moreover there is Komodo Edit which is free but doesn’t have an integrated debugger, so I’m going to settle on it and will keep Eclipse for remote debugging. Let’s see if ActiveState drops the price so it can get more popular. By the way, it also has support for code snippets and macros which can make most of the stuff that TextMate’s bundles can do.

So right now, I’m using Komodo IDE and TextMate. When the trial periods of both are over I’ll have a combination of Komodo Edit, Eclipse and Smultron for the fast editing.

PS: Excuse the lack of links… google is your friend

Automatic login to SSH with Putty

2007-07-10T00:00:00+02:00

Go to the PuTTY homepage and download PuTTY, PuTTYgen and Pageant. Put them in an easily accessible directory like %PROGRAMFILES%\Putty.

Launch puttygen.exe and click on Generate, then move your mouse in the space below the progress bar to create some entropy for the key randomness. Once the key is generated lets complete the available fields:

Key comment is just a comment so anything will do, putting username-putty seems quite logical.
Key passphrase is actually a password to unlock the key. Put something complex in here but make sure you’ll remember it in the future!!!
Confirm passphrase, here you just input again the password to confirm you know which is it.

Now click on Save private key to store it permanently on your hard disk. Choose a secure place for it, I can’t strength enough how important is that this key remains secret. A good location could be %USERPROFILE%\ssh\putty.ppk.

Do not close PuTTYgen just yet, we need to copy the public key to our server (or servers). Connect to the remote server with your username and password as normal, go to your home directory and create a directory named .ssh if it doesn’t exists already. Create a text file named authorized_keys2 in the .ssh/ directory, inside that file copy the public key from PuTTYgen in a single line, save the file and exit the shell. Make sure the .ssh directory and its contents are only accessible by you (chmod -R og= .ssh). Repeat this step for each server you want to access with your public key.

The next step is to configure PAgeant (PuTTY Authentication Agent) to automatically start with Windows. Just create a shortcut in your Startup folder, the target should be something similar to %PROGRAMFILES%\Putty\pageant.exe "%USERPROFILE%\ssh\putty.ppk", change it to suit the paths you’ve used.

Double click it to avoid having to restart to launch it, a small pop-up window should show asking for your passphrase, type it and press Ok. If you have typed the correct passphrase a small blueish icon should show on you system tray, right click it to start a new session or launch a pre-configured one. If everything is allright PAgeant should take care of authenticate you.

Just a simple tip in case you don’t know, you can specify an username together with the server address by using this syntax: username@server.com. This way the authentication is completely automatic.

There are also another couple of great companion tools from PuTTY: PSCP and PSFTP. The first is a command line tool to transfer files securely between computers, the second is a text mode FTP client using a secure SSH connection. Specially the PSCP can be handy to be used in batch files to automatize some tasks.

PLEASE READ: Be careful with using public keys authentication to connect to a server. The steps I described are ok if you have the access to your computer restricted, do not leave PAgeant running while you’re away from your computer!

Los sitios web deberían ser fáciles de crear

2007-07-03T00:00:00+02:00

Este es un post filosófico que llevo algún tiempo preparando para aclarar mis ideas. Como dice el título, los sitios web (aplicaciones web incluidas) deberían ser fáciles de crear. El sitio web típico tiene unas pocas docenas de páginas, un par de formularios y sindicación RSS como mucho. No parece que sea algo muy complejo, menos aun si tenemos en cuenta que la infraestructura para que funcionen es asequible, contrastada y potente. Algunos (yo me incluía en ese grupo hasta hace unos meses) culpan el hecho de la complejidad de crear un buen sitio web a lo enrevesado del entorno, un entorno no controlado como es un navegador web, donde las sutiles diferencias entre versiones y marcas suponen elevar exponencialmente la complejidad de la creación. La realidad en mi opinión es distinta.

El paradigma de programación más utilizado actualmente para el desarrollo de sitios web es el popular MVC (Model-View-Controller), algo que ya tiene unos cuantos años pero que solo en los últimos dos o tres se ha empezado a utilizar para el desarrollo de webs simples. La idea de ese concepto es que dividimos el desarrollo en tres capas diferenciadas. El Modelo se encarga de hacer el acceso a los datos uniforme, la lógica de negocio queda encapsulada en esta capa. La Vista se encarga de la presentación de la información, formateando los datos obtenidos del Modelo de una manera atractiva para el visitante. Por último el Controlador es quien controla quien puede ver que y bajo que circunstancias, además de ser el pegamento que une todas las piezas. Pues bien, resulta que para un sitio web normalito no necesitamos tantas capas, porque la realidad es que esas capas ya existen en la infraestructura utilizada, como por ejemplo con un stack LAMP.

Apache no es más que un Controlador, MySQL es un Modelo y los ficheros HTML son la Vista. Estamos de acuerdo en que esa infraestructura es muy genérica y que trabajar con ella es como construir una casa con herramientas manuales. Pero por que ocultar totalmente esas herramientas para substituirlas por otras nuevas más lentas y menos flexibles? Hablemos de REST, la filosofía detrás de la World Wide Web.

REST es el diseño en el que se modeló la web, se basa en que hay un conjunto de recursos identificados globalmente (la conocida URL) que pueden obtenerse en diferentes formatos (HTML, XML, PDF…). El diseño es realmente elegante y lo mejor de todo es que es algo probado y contrastado durante casi dos décadas. Mi conclusión es que si hacemos un sitio web este ha de hacerse en base a los principios básicos del medio donde reside, de lo contrario sería como intentar hacer un barco para ir por una carretera, seguro que podemos añadir ruedas pero seguirá pareciendo un barco en vez de un coche.

Así que juntando lo comentado arriba sobre LAMP como framework con el diseño REST, podemos obtener un sistema muy potente, flexible y fácil de utilizar. Para empezar tenemos que aprovechar la infraestructura disponible, y aprovecharla con criterio, como por ejemplo no almacenar las páginas HTML en una base de datos sino en el disco. A primera vista vemos que solo con el servidor web, la base de datos y el HTML estamos como hace 10 años, editando los ficheros con un editor y subiéndolos por FTP para publicarlos. Necesitamos algo que complete esas herramientas y ese algo son los lenguajes interpretados, ya sea PHP, Perl, Python, Java o Ruby. Estos lenguajes nos permitirán crear el pegamento necesario para ofrecer soluciones modernas de una manera simple y escalable. La idea es volver unos 5 años hacia atrás y retomar la costumbre de hacer una web con ladrillos, utilizando scripts prácticamente independientes para crearla. Después de todo PHPNuke fue popular pero no era la solución, aunque detrás de él hayan aparecido miles de sucesores intentando solucionar sus problemas, que lo único que han conseguido es ocultar esos problemas con capas y capas de complejidad añadida.

En un próximo artículo comentaré sobre una posible implementación de todo esto. La idea básica es la de quitar capas de complejidad de nuestros scripts para poder concentrarnos en la funcionalidad y el contenido, eliminando posibles puntos negros de errores y aprovechando la escalabilidad inherente a la web.

Rectangle Packing (2D packing)

2007-06-15T00:00:00+02:00

I’m working on a little thing which I don’t want to speak about just now, lets see if I can get it to do what I want first before making it public. However for that project I needed to pack several small rectangles of varying dimensions into a bigger one without them overlapping.

At first it seem like an easy task but the more I thought about it the more difficult it was. After a bit of research I found out that until today this seems to be a non resolved problem, there are algorithms which offer a pretty decent approximation to the optimal result though. The problem is commonly known as the Bin Packing problem and in this case I needed to work with two dimensional objects (rectangles) so it gets a lot more complicated.

After more googling I found an article by Jim Scott with pseudo code to solve a similar problem, the article explains on how to pack several lightmaps into a single texture. It was the easiest to understand algorithm I had found so I went ahead and implemented it in PHP. I had it in a few minutes but I wanted to fine tune the results by feeding it the rectangles by different sorting criteria, since it seems to be a good way to achieve the best results. Well, I got tired to slowly generating the image in PHP to check the results with every minor variation so I implemented a JavaScript version which allows to see the result in real time.

This algorithm can be used for a wide variety of problems, from packaging small sprites in a big image to allocate tasks in complex projects (use the width for the team developers and the height for the time).

A simple usage example could be as follows:

var coords,
    packer = new NETXUS.RectanglePacker( 512, 512 );

for (var i=0; i<images.length; i++) {
    coords = packer.findCoords( images[i].width, images[i].height );
    alert( 'Image of ' + images[i].width + 'x' + images[i].height +
         ' allocated at ' + coords.x + 'x' + coords.y
    );
}

I’ve put up a demo so you can try it and see the results with different parameters. Tip: the best results seems to be with the blocks sorted by the magic method and reversed.

You can download the source code from this link.

Hex2Dec and Dec2Hex

2007-06-11T00:00:00+02:00

For secret reasons I need to usually convert hex strings to decimal notation. This can be easily done with a scientific calculator, like the one supplied with Windows, but it’s slow and I’m usually in a hurry to convert them :)

So I took a look at GreaseMonkey, which is an extension for Firefox which allows to modify a web page once rendered with a simple javascript file.

To install it you’ll first need to have GreaseMonkey installed and activated (a little monkey face on Firefox’s status bar), then click on this link satkeystringconversor.user.js

Here is the highlighted source code so you can review it easily. It’s a pretty simple script which traverses the DOM performing regular expression tests on text nodes and converting the matched strings into span elements with an onclick behaviour.

// ==UserScript==
// @name           Sat key string conversor
// @namespace      http://pollinimini.net
// @description    Converts satelite key strings from hex to dec or vice versa
// @include        http://www.satscreen.ws/forum/*
// @include      https://www.blogger.com/*
// @include      http://floresd.blogspot.com/*
// @include      http://galeon.com/tusflores/*
// ==/UserScript==
function HEXDEC ( node ) {

  function hex2dec(hex) {
    var dec = [],
      re = /([0-9A-Za-z]{2})\s*/g|>,
      m,
      conv;
    while (m = re.exec(hex)) {
      conv = parseInt(m[1], 16);
      dec.push( (conv < 100 ? ( conv < 10 ? '00' : '0' ) : '') + conv );
    }
    return dec.join(' ');
  }

  function dec2hex(dec) {
    var hex = [],
      re = /([0-9]{3})\s*/g|>,
      m,
      conv;
    while (m = re.exec(dec)) {
      conv = (m[1]-0).toString(16).toUpperCase();
      hex.push( conv.length < 2 ? '0'+conv : conv );
    }
    return hex.join(' ');
  }

  function swapHexDec( e ){
    var target;
    if (!e) var e = window.event;
    if (e.target) target = e.target
    else if (e.srcElement) target = e.srcElement
    if (target.nodeType === 3) target = target.parentNode;
    var tmp = target.firstChild.nodeValue;
    target.firstChild.nodeValue = target.getAttribute('hexdec');
    target.setAttribute('hexdec', tmp);
    if (target.className === 'hexdec')
      target.className = 'dechex';
    else
      target.className = 'hexdec';
  }


  // if no node is specified then start processing on the page body
  node = node || document.body;

  var skipRe = /^(script|style|textarea)$/i|>;
  var re = /((?:[0-9A-Ha-h]{2}\s*){6,})|((?:[0-9]{3}\s*){6,})/g|>;
  // get all nodes from dom
  var nodes = node.all || node.getElementsByTagName('*');
  var i, cnode, span;

  // loop thru all the nodes
  for (i=0; i<nodes.length; i++) {
    // skip if it has no child nodes, it's already processed or the tag is not valid
    if (!nodes[i].childNodes.length ||
      nodes[i].hasAttribute('hexdec') ||
      skipRe.test(nodes[i].tagName)
    )
      continue;

    // loop thru all children
    cnode = nodes[i].childNodes[0];
    while (cnode) {
      // we just care about text nodes
      if (cnode.nodeType === 3) {
        // loop thru all regex matches
        while (m=re.exec(cnode.nodeValue)) {
          // if there is text before the match add it on its own node
          if (m.index > 0)
            cnode.parentNode.insertBefore(
              document.createTextNode( cnode.nodeValue.substr(0, m.index) ), cnode
            );
          // create a new span node for the key string
          span = document.createElement('SPAN');
          // if the first capture parenthesis is found then it's in hex format
          if (m[1]) {
            span.className = 'hexdec';
            span.setAttribute( 'hexdec', hex2dec(m[0]) );
          // otherwise it is in decimal format
          } else {
            span.className = 'dechex';
            span.setAttribute( 'hexdec', dec2hex(m[0]) );
          }
          span.setAttribute( 'title', 'Click to swap betwen hex and decimal notations' );
          span.appendChild( document.createTextNode( m[0] ) );
          //span.onclick = swapHexDec;
          span.addEventListener( 'click', swapHexDec, true );
          cnode.parentNode.insertBefore( span, cnode );

          // remove the already processed text from the node
          cnode.nodeValue = cnode.nodeValue.substring( m.index + m[0].length );
          // reset the regex text pointer
          re.lastIndex = 0;
        }
      }
      // point to next children or null
      cnode = cnode.nextSibling;
    }
  }
}

window.addEventListener( 'load', function () { HEXDEC() }, true );

function addGlobalStyle(css) {
    var head, style;
    head = document.getElementsByTagName('head')[0];
    if (!head) { return; }
    style = document.createElement('style');
    style.type = 'text/css';
    style.innerHTML = css;
    head.appendChild(style);
}
addGlobalStyle( '\
  .hexdec, .dechex { \
    cursor: hand;\
    cursor: pointer !important;\
    border-bottom: 1px dotted #888;\
    padding-left: 32px;\
    background: url(http://pollinimini.net/temp/hex.png) no-repeat left;\
  }\n\
  .dechex {\
    background: url(http://pollinimini.net/temp/dec.png) no-repeat left;\
  }');

By default the following sites are enabled (you can configure it for whatever sites you need):

http://www.satscreen.ws/forum/
http://floresd.blogspot.com/
http://galeon.com/tusflores/
https://www.blogger.com/

It uses a neutral blue coloring for the styling which shouldn’t break most web designs. The images are served from my own server, so they aren’t very reliable, don’t be afraid if one day they stop showing :)

Detecting IE with JS conditional comments

2007-04-06T00:00:00+02:00

I just stumbled upon a short yet excellent post on Dean Edwards blog. It shows how to use IE proprietary javascript conditional comments to detect if we’re running on IE in a secure and very short way.

var isMSIE = /*@cc_on!@*/false;

Further down in the post’s comments someone named Lucky propose the use of a variant to check against IE 5.5+ and IE7. javascript:

var ieScript/*@cc_on=@_jscript_version@*/;
if (ieScript) // IE
if (ieScript>=5.5) // IE 5.5 +
if (ieScript==5.7) // IE 7.0 VistaCopy to clipboard

Moreover Dean also shows in one of the comments how to check for the rendering path used by IE (quirks VS strict modes)

var ieStrict = document.compatMode == 'CSS1Compat';Copy to clipboard

While is not very wise to abuse the sniffing of browsers vendors it’s also true that sometimes it’s required to split the code path in order to support the incompatibilities among browser vendors and versions.

Commenting regular expressions in Javascript

2007-03-20T00:00:00+01:00

I’ve been working a lot lately with regular expressions in Javascript. Being a really powerful tool it’s quite easy to get shoot on your own feet if you are not careful.

After pushing the Javascript built in regular expression engine to its limits I found it lacking some modern features available in most recent versions of regex engines used in many languages like .Net, C++ Boost, Perl, Ruby or PHP. The feature I miss the most is the ability to use look behind assertions, with JS just supporting look ahead ones. Another feature I miss are named groups and then one very useful new feature is the /x modifier (also known as ignore whitespace) which allows regular expression literals to span over multiple lines, ignoring white space, new lines and embedded comments!

See the following example regex to match an url:

/\b(https?|ftp):\/\/([-A-Z0-9.]+)(\/[-A-Z0-9+&@#/%=~_|!:,.;]*)?(\?[-A-Z0-9+&@#/%=~_|!:,.;]*)?/i

Writting it is not difficult but modifiing it or even understanding it becomes an issue. Using the Perl /x modifier it would look like this:

m/\b                              # match at word break
  (https?|ftp)://                 # protocol: http, https or ftp
  ([-A-Z0-9.]+)                   # domain
  (/[-A-Z0-9+&@#/%=~_|!:,.;]*)?   # file path - optional
  (\?[-A-Z0-9+&@#/%=~_|!:,.;]*)?  # query string - optional
/ix;

Much easier to understand, isn’t it? At least I really think so. To imitate this behaviour in Javascript I’ve written this simple function which just concatenates the regular expressions passed as arguments and returns a new one. This allows to break a complex regex into smaller pieces and comment them.

function xRegExp() {
    var cur, re;
    var arr = [];
    var n = arguments.length;

    // check if the last argument contains the regexp modifiers
    if (/^[gmi]+$/|>.test(arguments[n-1]))
        n--;

    // go thru all the passed regexes and concatenate them
    for (var i=0; i<n; i++) {
        if (typeof arguments[i] === 'string') {
            arr.push( arguments[i] );
        } else {
            // when we cast a regexp object to a string
            // the / delims and modifiers are included
            cur = String(arguments[i]);
            arr.push( cur.substring(cur.indexOf('/')+1, cur.lastIndexOf('/')) );
        }
    }

    // create a new empty regexp
    re = new RegExp();
    // compile the concatenated regex to speed it up
    re.compile( arr.join(''), arguments[n] );
    return re;
}

Please note that the same thing can be done by concatenating strings manually, however regexes syntax is terse enough without having to double escape all the tokens. That’s why I preffer to write them out in their literal form (between slashes) instead of using quotes.

Using the given function the above example would look like this. Note that you can use the regexp literal notation or just pass a simple string. The last argument can optionally pass the modifiers for the regular expression.

var re = new xRegExp(
    /\b/,                                 //match at word break
    '(https?|ftp)://',                    //protocol: http, https and ftp
    '([-A-Z0-9.]+)',                      //domain
    '(/[-A-Z0-9+&@#/%=~_|!:,.;]*)?',      //file path (optional)
    /(\?[-A-Z0-9+&@#/%=~_|!:,.;]*)?/,     //query string (optional)
    'i'                                   //Modifiers: case-insensitive
);

Some brief notes on TextAreaPlus

2007-03-08T00:00:00+01:00

Been a quite long time since the last update. I’m a bit ashamed for not keeping the blog updated regularly but I must confess that I’ve been expending my time on other affairs more important to me.

Anyways, in this weeks I’ve been working again on TextAreaPlus, actually I’ve been busy making a full rewrite of the lexing scanner and optimizing the implementation to the limits of my knowledge. The algorithm used is described in the project wiki.

Today I was re-installing the PC at work since it broke havoc yesterday. After finishing the installation of almost all softwares I decided to have a look for a notepad replacement which was more feature packed than Scite and a bit faster than Notepad++ to perform some minor editing of config files mainly. In the search I found a couple of new editors which I didn’t knew about but which seem to be gaining momentum. Both seem to be based on a successful Mac text editor called TextMate, which I have heard about from Ruby-on-Rails evangelists but given my allergic reactions to anything mac related lately I didn’t check before. Today, trying those two new editors I got interested in TextMate and I’ve been learning about it in the last hour.

So long history short, I thought I had invented the sliced bread with my implementation and design ideas for TextAreaPlus but it turns out that Allan Oddgaard, the TextMate developer, already implemented it in a very similar way a couple of years ago. My feelings are a bit conflictive right now, in one way I feel sad because I just reinvented the wheel but in the other I feel reassured to have come up with a solution which turns out to be proved and successful.

I’ve written a couple hundred words with a couple of links in this post using a lame browser textarea, in moments like this I wish I had already a production ready version of TextAreaPlus. Lets hope I don’t keep wishing for it too long.

Comprobar un NIF, NIE o CIF

2007-02-01T00:00:00+01:00

Después de postear un simple código para comprobar un NIF, me he mirado un poco más el tema y para mi sorpresa no hay mucha información sobre esto.

Así que aquí presento una pequeña función para comprobar si el formato de un NIF, NIE o CIF es válido.

<?php
function checkIdentidadFiscal( $str )
{
    //normalizamos el formato
    $str = preg_replace( '/[^0-9A-Z]/i', '', $str );

    // El formato es de un NIF o un NIE
    if (preg_match('/X?[0-9]{8}[A-Z]/i', $str))
    {
          //para no duplicar código, eliminamos la X en el caso de que sea un NIE
          $str = preg_replace('/^X/i', '', $str);

          //calculamos que letra corresponde al número del DNI o NIE
          $stack = 'TRWAGMYFPDXBNJZSQVHLCKE';
          $pos = substr($str, 0, 8) % 23;
          if (strtoupper( substr($str, 8, 1) ) == substr($stack, $pos, 1) )
              return true;
    }
    // El formato es el de un CIF
    else if (preg_match('/[A-HK-NPQS][0-9]{7}[A-J0-9]/i', $str)) //CIF
    {
          //sumar los digitos en posiciones pares
          $sum = 0;
          for ($i=2; $i<strlen($str)-1; $i+=2) {
              $sum += substr($str, $i, 1);
          }

          //Multiplicar los digitos en posiciones impares por 2 y sumar los digitos del resultado
          for ($i=1; $i<strlen($str)-1; $i+=2) {
              $t = substr($str, $i, 1) * 2;
              //agrega la suma de los digitos del resultado de la multiplicación
              $sum += ($t>9)?($t-9):$t;
          }

          //Restamos el último digito de la suma actual a 10 para obtener el control
          $control = 10 - ($sum % 10);

          //El control puede ser un número o una letra
          if ( substr($str, 8, 1) == $control ||
               strtoupper(substr($str, 8, 1)) == substr('JABCDEFGHI', $control, 1 ))
              return true;
          }

          return false;
    }
}

El algoritmo para comprobar el CIF lo he sacado de aquí.

Introducing FLV4PHP

2006-09-11T00:00:00+02:00

This weekend I found some time to work in a new project. In the last post I wrote about streaming video for Flash players, I went thru some different options for that task and one of them was the HTTP progressive download of FLV files.

There is an interesting idea to make seeking possible when using a simple HTTP download (I think Google Video does just this), it has been already mentioned in a few places.

There are some tools already which can inject the needed metadata into an flv file, namely, FLVTool2 and flvmdi. The ffmpeg project can create FLV files and there seem to be already patches to support the creation of metadata entries too.

Over the years I’ve come to love to develop readers for obscure (not standard neither well documented) file formats, it’s quite a challenge to do it but since the file format spec is not public you have a very good excuse when bugs are found :)

So I’ve started to work on a system to serve video (and also audio only) files to Flash based players with support for seeking. The idea is to just limit ourselves to use PHP and Flash technologies, no shell scripts or compiled programs, since they are difficult to get running on most shared hosts.

Right now the FLV analyzing library is finnished and the sample metadata extraction tool can process a 140Mb FLV in just 1.3 seconds on my cheap DreamHost test system.

So take a look at the project page, right now I’m looking for an Action Script guru to help out with the development of the player. I’ve tried to get my brains around Flash but I haven’t got much success :(

By the way, all this work will be used for an upcoming project I’m planning. Am I the only one who think that current web photo gallery systems suck?

URL argument separator

2006-08-11T00:00:00+02:00

Quite so often a web developer starts thinking what’s wrong with the guys who make the rules of how Internet works. And if we dig a bit in the problem, we can many times see that the ‘non-sense’ is there not because the specs were flawed but because the popular implementations didn’t follow completely the spec.

The ‘non-sense’ I’ve been thinking about lately is why the hell we use the ampersand & to separate the arguments in a URL. My problem with using the & as separator is that in todays world of XML crazyness, the ampersand must be handled with care. It seems that the specs said that any character could be used after the initial query char mark ? but the # which marks the start of the fragment. Actually, in the current specs for URI/URL syntaxis they recommend the semicolon ;

So the decision to choose the ampersand as the defacto standard was taken by the guys who took the lead with the CGI scripts and the dynamic web content generation.

Since the specs recommend the use of the semicolon I thought “ok, no problem, I’ll just use it from now on” with a grin in my face. If you are thinking the same keep reading because it’s not that easy. My server-side language of choice right now is PHP, so I build up a simple test script to check out if the semicolon worked ok, but it didn’t. After consulting php.net for a bit more of information I found the solution, there are two ini directives which define the argument separators, one for input and the other for output (I guess this last one is used to manage the session id without cookies). arg_separator.input and arg_separator.output are the names of the directives. The first one takes a string as argument and every char in that string becomes an argument separator, the second one defines what will PHP use when building URLs as separator.

The bad news is that the default value is just the ampersand &. The good news is that both directives can be defined per directory, so we can put them in our .htaccess in a shared host. If you control the host just edit the php.ini file. Logically we can’t modify them with ini_set() because the URL is already processed when our script starts execution (see below for a solution to this).

php_value arg_separator.input "&;"
php_value arg_separator.output ";"

Note that I left both: the ampersand & and the semicolor ; as valid argument separators, I’m far too used to the ampersand and I don’t want to expend my whole life debugging an application just because I forgot that I decided that the semicolon was a better suit :)

Even if we can’t use the ini_set() function to modify the PHP behaviour for this, we can still make use of a different argument separator by processing the $_GET and $_REQUEST arrays and splitting them by our preffererd argument separator. The following code snippet show do it but please don’t use it in production environments, it’s probably flawed!

<?php
// first join the PHP splitted arguments
$get = '';
foreach ($_GET as $k=>$v) {
          $get.= $k . '=' . $v . ';';
          // remove from the arrays, we'll add the correct ones later
          unset($_GET[$k]);
          unset($_REQUEST[$k]);
}

// split the query string using our desired char
$args = split(';', $get);
// process the query string and rebuild the _GET and _REQUEST array
foreach ($args as $arg) {
          // now separate the key from the value and assign them to the arrays
          list($k,$v) = explode('=', $arg);
          if (! empty($k)) {
              $_GET[ $k ] = $v;
              $_REQUEST[ $k ] = $v;
          }
}

Changing the argument separator does not only help out when working with XML based technologies but also to give your site URLs a geeky look ;)

Doing Ajax? you'll need transactions!

2006-07-10T00:00:00+02:00

In an embarrassing attempt to bring some more visits to this blog I’ll start to post about buzzwords. After some months of fighting against it I’ve finally understood that there is no point in teaching people that AJAX, Web 2.0 and the like are just silly buzzwords about quite old technologies. So from now on I’ll get into the bandwagon and start using those buzzwords.

Getting into the stuff that matters, lets start with the basics. As you should already know, the web works on top of the HTTP protocol, which by design is based on a request-response paradigm. It’s a proven design which works like a charm for the typical web, where static pages are loaded from an URL, however the Web is evolving to a new kind of pages. The current trend is to make web pages which act like the typical thin client, where the bulk of the operations and domain logic (model) is performed on the server but the application flow (controller) and display (view) is handled on the client side with the help of JavaScript and DOM.

Looking at the whole picture, you’ll need perspective here, AJAX web pages are Web Service Brokers. This exposes a big problem, since we’re sill using server side software focussed (read optimized) on the traditional web, while the client side is moving to support the thin client paradigm. The main trouble here is the design concept of the web where each page is an atomic unit, so we need to take some considerations when developing ajaxified web pages if we don’t want to hurt performance seriously.

Most AJAX frameworks expose some simple examples to demonstrate their functionality. They work ok, they do quite cool things, but looking at them a bit closer there are some grey areas. Besides the popular live-search, update on checkbox click or the plain spellchecker, there is a lot more functionality promised by the AJAX hype. Take for example a datagrid widget, like an Excel spreadsheet, there are some complex processes running on the background when working with it. When editting a cell we need to send the new value to the server, check if it worked ok and fetch any other updates on calculated cells which could have been affected by that change. So we have several weak points (mainly by the inherent network unreliability) which would need checks. In the previous example of the datagrid, imagine that the fetch of the affected cells fails, the application state on the client side and on the server side will be different and this is a major concern when working with thin clients.

To solve this problem we have to use transactions (already popular on the RDBMS world). Transactions will allow us to emulate the web principle that each page/action is atomic. In the case that something goes wrong we can rollback the changes and our application state will be kept the same on both sides.

Extending on this idea, we can even optimize the request-response cycle by using an Unit of Work pattern for example. We compose a list of actions and send it when finished to the server instead of sending individual actions. Network latency is always an issue and the asynchronous nature of AJAX is not used too often in practice, when we make a change we expect an inmediate update. By packing the actions in Units of Work we can optimize the client-server interaction and support transactions quite easily.

Remember that for the tipical RPC the web server needs to spawn (or fork) a new process, load the dynamic language interpretter, parse thousends of lines of code, perform a DB connection and do the actual work. Even when having a properly setup backend (opcode cache, fast-cgi), the overhead of a RPC is huge. Packing actions together seems like a great solution for performance problems.

Como escribir números

2006-07-07T00:00:00+02:00

Vaya por delante que el tema no está demasiado estandarizado a nivel internacional. Lo que realmente supone un problema a la hora de interpretar números en un mundo globalizado como el actual.

Una cosa si que hay en común, según el Sistema Internacional, no se deben separar los millares, en el caso de que haya de hacerse para mejorar su lectura, se hará mediante el uso de un espacio de no rotura, que viene a ser un poco más estrecho que un espacio normal. Este espacio puede introducirse en un editor de textos como Microsoft Word mediante la combinación Ctrl+Shift+Espacio (Ctrl+Espacio para OpenOffice). El código unicode es U+00A0 y en HTML es el conocido &nbsp.

Para los decimales la cosa es algo más complicada. Hay dos grandes zonas geográficas con dos estilos distintos. La notación anglosajona usa un punto, mientras que en la mayoria del resto de paises se usa la coma.

Así pues todas las siguientes formas son correctas:

123456.78
123 456.78
123456,78
123 456,78

A la hora de implementar una aplicación, donde se deben introducir números, lo mejor es soportar todos los formatos en la entrada de información y normalizarlos a la hora de operar con ellos. Si nuestra aplicación solo operará en una región, podemos limitarnos al uso de ese formato específico, aunque hoy en día, y especialmente en apliaciones web, nunca es bueno suponer que solo será usada en una región límitada.

Interfacing ODBC from Linux

2006-06-02T00:00:00+02:00

Our billing software runs on Windows platforms and uses ODBC to store its data. Recently we called the makers of that software since we wanted to extract some Business Intelligence from our records.

It turned out that they where going to release a new program for that purpose. We asked to evaluate the beta version and to my surprise it just created an Microsoft Excel file with a dynamic table report.

So I worked on the needed SQL query to extract the records we wanted and exported the data into a CSV to be loaded in an Excel worksheet with a dynamic table report. It worked ok, however the sales and marketting people didn’t want to contact a technician every time they needed the reports to be updated.

The problem was that our application server is a Debian box and since we didn’t have to much time to expend on this, we looked for the easiest solution. I found ODBC Socket Server which seems unmaintained but the last version works ok on our Windows Server 2003 machine. It’s a service which acts as a proxy for ODBC databases in the local machine, so they can be accessed from other machines using a socket connection and a very simple XML grammar. Once everything was in place I wrote a simple PHP program to fetch the needed records (a few thousand rows). To my surprise the available PHP libraries to interact with ODBC Socket Server didn’t perform too well. They stored the full response in memory and then build up an array, since we need a few thousend rows, the amount of memory required was too high for our little Debian system.

The following code is a pretty simple set of classes to work with ODBC Socket Server. They fetch the data on demand, using a SAX type xml parser to discard the already processed records. It’s not a really thought of or tested solution, however I’m making it public since it’s far better than existing solutions.

<?php
/*
  The ODBCSock class is an utility object to operate with the ODBC Socket Server
  from http://odbcsock.sourceforge.net

  It's compatible with ODBC Socket Server's XML formats 0 and 2. In our tests
  xml format 2 is significally faster than the others.
  Should run on PHP4 and PHP5, however it's only been tested on PHP5

  This code is copyright (c) 2006 Netxus Foundries
  v1.0 28-May-2006 : Ivan -DrSlump- Montes <imontes@netxus.es>

  This program is free software; you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by
  the Free Software Foundation; either version 2 of the License, or
  (at your option) any later version.

  This program is distributed in the hope that it will be useful, but
  WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
  or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
  for more details.
  You should have received a copy of the GNU General Public License along
  with this program; if not, write to the Free Software Foundation, Inc.,
  51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
*/

class ODBCSock {
  var $handle = null;
  var $server = 'localhost';
  var $port = '9628';
  var $username = '';
  var $password = '';
  var $database = '';

  function ODBCSock( $server, $user, $passwd, $database = false )
  {
    $server = explode(':', $server);
    if (count($server) > 1)
      $this->port = array_pop( $server );
    else
      $this->port = 9628;
    $this->server = implode(':', $server);

    $this->handle = fsockopen( $this->server, $this->port, $errNo, $errStr, 30 );
    if ( ! $this->handle ) {
      return null;
    }

    $this->username = $user;
    $this->password = $passwd;

    if ($database !== false)
      $this->selectDB( $database );
  }


  function selectDB( $database )
  {
    $this->database = $database;
  }


  function & query( $query )
  {
    $pl = array();
    $pl[] = '<?xml version="1.0"?>';
    $pl[] = '<request>';
    $pl[] = '<connectionstring>DSN=' . HTMLSpecialChars($this->database) . ';UID=' . HTMLSpecialChars($this->username) . ';PWD=' . HTMLSpecialChars($this->password) . ';</connectionstring>';
    $pl[] = '<sql>' . HTMLSpecialChars( $query ) . '</sql>';
    $pl[] = '</request>';

    fwrite( $this->handle, implode("\r\n", $pl) );

    $rs = new ODBCSock_Result( $this );
    if ($rs->sax->error) return FALSE;
    else return $rs;
  }
}


class ODBCSock_Result {
  var $rows = array();
  var $conn;
  var $lastRow = 0;

  function ODBCSock_Result( &$conn )
  {
    $this->conn =& $conn;

    $this->sax = new ODBCSock_SAX( $this );
    $this->parser = xml_parser_create();
    xml_set_object( $this->parser, $this->sax );
    xml_parser_set_option( $this->parser, XML_OPTION_CASE_FOLDING, false );

    xml_set_element_handler( $this->parser, 'startElement', 'endElement' );
    xml_set_character_data_handler( $this->parser, 'content' );

    while ( $this->_read() ) {
      if ($this->sax->error || $this->sax->parsing)
        break;
    }
  }

  function _eof()
  {
          return feof($this->conn->handle);
  }

  function _read()
  {
          xml_parse( $this->parser, fread( $this->conn->handle, 4096 ), feof($this->conn->handle) );
  }

  function addRow( $row )
  {
          $this->rows[] = $row;
  }


  function fetch( $row = false )
  {
    if ($row === false) {
      $row = $this->lastRow;
      $this->lastRow++;
    } else {
      $this->lastRow = $row+1;
    }

    do {
      $this->_read();

      if (isset($this->rows[$row]))
        return $this->rows[$row];

    } while ( !$this->_eof() );
  }
}

class ODBCSock_SAX {

  var $error = false;
  var $parsing = false;

  var $curElement = '';
  var $curColumn = '';
  var $curColumnIdx = 0;
  var $curRowIdx = 0;

  var $rs = null;
  var $row;
  var $labels = array();

  function ODBCSock_SAX   ( &$rs )
  {
    $this->rs =& $rs;
  }

  function startElement( $parser, $name, $attrs )
  {
    $this->curElement = $name;

    if ( $name == 'result' ) {
      if ($attrs['state'] == 'success')
        $this->parsing = true;
      else
        $this->error = true;
    } else if ( $name == 'row' ) {
      $this->row = array();
      $this->curColumnIdx = 0;
    } else if ($name == 'column') {
      if ($attrs['name']) {
        if ($curRowIdx == 0)
                $this->labels[] = $attrs['name'];

        $this->curColumn = $attrs['name'];
      } else {
        $this->curColumn = $this->labels[$this->curColumnIdx];
      }

      $this->row[ $this->curColumn ] = '';
      $this->curColumnIdx++;
    }
  }

  function endElement( $parser, $name )
  {
    if ( $name == 'row' ) {
      $this->rs->addRow( $this->row );
      $this->curRowIdx++;
    }
  }

  function content( $parser, $data )
  {
    if ($this->curElement == 'column') {
      $this->row[ $this->curColumn ] .= $data;
    }
  }
}

The Binary Search algorithm

2006-05-23T00:00:00+02:00

Long time since I last wrote here so I’ll take the opportunity to introduce (just joking) the Binary Search algorithm.

I was looking for a safe way to find the line of text where a user clicks the mouse. I needed this for the TextArea+ editor. Fortunately Mozilla supports the DOM3’s compareDocumentPosition() function, so I took advantatge of it. Since I’m using a UL with an LI child element for each line of text, I can say that UL.childNodes is an ordered array, which fits perfectly for the Binary Search algorithm.

The algorithm is really simple (I’m sure most of you have used it before), we just need to find the middle point between two points and compare its value with the one we’re looking for. We do this recursively until we either find the value or the right point becomes smaller than the left one, which means that the value is just not there.

Here comes a really simple example of this algorithm (without using recursive calls)

var left = 0;
var right = 100;

while (left <= right) {

    middle = floor( (right-left)/2 ) + left;
    if (haystack[mid] == needle) {
        alert('Found!');
        break;
    }
    if ( needle < haystack[middle] )
        right = middle-1;
    else
        left = middle+1;
}

The Binary Search algorithm performs at O(log n) time, but for TextArea+ I use a number of further optimizations. Since it’s likely that the user will click closely to where the cursor currently is, I just check the N precedding and following lines before. If this fails I check the current viewport and if still not found, then just do a full search. And this is just for the Mozilla code path! hell, how I hate cross-browser coding!

This post is meant to remind us how important is to learn already known and proved algorithms and that even when having a good algorithm, there is always room to optimize by taking advantge of our specific problem.

Programmatic reflection effect

2006-04-28T00:00:00+02:00

Today Haripako pointed us to Thomas Fuchs (of script.aculo.us fame) DHTML image reflection effect

It uses 1px tall divs with an adjusted top offset for the containing image and the opacity property to perform it. Well, you just need to know some basic arithmetics to create this kind of effects, but I guess that if people find this kind of stuff so cool is just because most of them only do maths to check their wages :)

So here is some PHP to perform the same effect on the server side

<?php
$file = 'test.png';

// Configuration params
$mirrorSize = 1.35;   /* 1 .. 2 */
$startAlpha = 80;     /* 0 .. 127 */

// Adjust params to supported limits
$mirrorSize = max( 1, min($mirrorSize, 2) );
$startAlpha = max( 0, min($startAlpha, 127) );

// Load the source file
$imgSrc = ImageCreateFromPng( $file );

// Create the reflect portion
$imgRfl = ImageCreateTrueColor( ImagesX( $imgSrc ), floor(ImagesY( $imgSrc ) * ($mirrorSize-1)) );
ImageAlphaBlending($imgRfl, true);
ImageFill( $imgRfl, 0,0, ImageColorAllocate( $imgRfl, 255,255,255 ) );

$offset = ImagesY( $imgSrc ) - ImagesY( $imgRfl );

// In this loop we flip the source image and apply a fade out
for ($y = 0; $y < ImagesY( $imgRfl ); $y++)
{
    ImageCopy( $imgRfl, $imgSrc, 0,$y, 0,ImagesY( $imgSrc )-$y-1, ImagesX( $imgRfl ), 1 );
    $alpha = floor( ($y*(127-$startAlpha)) / ImagesY( $imgRfl ) );
    $color = ImageColorAllocateAlpha( $imgRfl, 255,255,255, 127-$startAlpha-$alpha );
    ImageFilledRectangle( $imgRfl, 0, $y, ImagesX( $imgRfl )-1, $y+1, $color );
}

// Create the image buffer which will be outputted
$imgDst = ImageCreateTrueColor( ImagesX( $imgSrc ), ImagesY( $imgSrc ) * $mirrorSize );
ImageAlphaBlending($imgDst, false);

// Paste into the output buffer the original image and the reflect portion
ImageCopy( $imgDst, $imgSrc, 0, 0, 0, 0, ImagesX( $imgSrc ), ImagesY( $imgSrc ) );
ImageCopy( $imgDst, $imgRfl, 0, ImagesY( $imgSrc ), 0, 0, ImagesX( $imgRfl ), ImagesY( $imgRfl ) );

ImageDestroy( $imgSrc );
ImageDestroy( $imgRfl );

// Output the image
header('Content-type: image/png');
ImageSaveAlpha( $imgDst, true );
ImagePng( $imgDst );

ImageDestroy( $imgDst );

I’ve tried to keep it very simple, so it’ll only work with PNG files. Moreover it’s hardcoded for white backgrounds, althought it can be easily changed.

If you use this be sure to cache the result, since the cpu cycles to process images is quite high.

Implementing a line based code editor

2006-04-06T00:00:00+02:00

To my surprise I couldn’t find much info on the net about implementing text editors. So I had to figure it out on my own, however as often happens, while implementing it I found some info when looking for related stuff.

There seems to be two common ways to implement text editors, those which are line-based and those which are block-based. I started with the block-based approach, dividing the text buffer in tokens and storing the position and color attributes in each block. Soon I found that this approach gets pretty complex quite fast. So I turned the code into a line-based editor. It’s almost like the block based with the main difference that it makes an intermediate division based on lines of text. This allows to simplify the code at the expense of a bit of overhead.

For each line of text we need to store the actual text (or the coordinates to fetch it from a buffer), the offset relative to the start of the text, the parser state at the start of the line and block definitions for that line. Since we have broken down the text in lines, for each block we just need to store the offset relative to the start of line which contains it, the length of the block and the style of it. The following figure illustrates the whole concept.

To make all this work we just need a function which parses a single line of text. This function would take as arguments a string with the line of text and the parser context at the start of that line. It will return the blocks found and the parser context at the end of the line.

The code would roughly look like this:

txtLines = split('\n', text);
lines = new Array();
context = new Context();
offset = 0;
for (i=0; i < count(txtLines); i++) {
   lines[i].source = txtLines[i];
   lines[i].offset = offset;
   lines[i].state = state;
   lines[i].blocks = parseLine( txtLines[i], context );
   offset += strlen( txtLines[i] );
}

The advantatge of the line based approach is that it’s very easy to reparse the text to display any changes on the syntax highlighting. To do so we just need to parse each line from the modified one to the end until the parser context matches

context = line[i].context;
do {
   lines[i].blocks = parseLine( lines[i].text, context );
   i++;
} while ( context != line[i].context );

Most of the time, buffer changes will only affect a line and most lines will only be a few chars long, so we have a really optimized solution with only a few lines of code.

In overall, writing a fast text editor is quite simple if we only optimize for the common cases. In fact, I’m having more problems optimizing the display routines than the buffer updating ones.

Browser based syntax highlighting code editor (part ||)

2006-03-26T00:00:00+01:00

In a previous post I was moaning about the lack of a proper browser based (dhtml powered) syntax highlighting code editor.

I took the aproach descrived in that post of using a translucent textarea over the syntax highlighted code to achieve the result with the minimum of effort. Well, it hast turned out that it won’t work as fast as spected. The performance is horrible, at least in Firefox, so I’ve abandoned this aproach.

However, as I already have the code to highlight the code in real-time, I’m trying to implement the code editor following a more natural aproximation, which is to capture the keyboard input and process it. To speed up development time I’m focussing just in Firefox, but I guess it’ll be portable to IE, Opera 8.x and Safari quite easily.

To capture the input from the user we need to handle some events: onFocus, onBlur and onKeypress. Unfortunatly Gecko only allows onKeypress events on input and textarea fields (IE doesn’t have this limitation). To sort this problem out we just need to create a non-visible text input field (style.display: none) which receives the focus when the element containing the editor does. This way we can trick Gecko and make use of onKeypress evens (in IE this won’t work, since it doesn’t fire events on non-visible elements). The drawback with this is that the selection, if any, is lost when changing the focus to the input field, however it can be easily solved by recreating the selection range after switching focus.

The syntax highlighting algorithm uses regular expressions, since they should be much faster than implementing a traditional tokenizer/lexer in javascript. This means that, at least for now, the highlighting can’t be nested (“Hello $userName” is rendered in one color, as string, without applying another style to the variable $userName). Basicly, what the code does is create an internal structure divided in lines which in turn have N blocks. Each block defines an offset from the start of the line and a style (normal, comment, string …)

// comment
$text = "string test";

becomes

[
    {
        offset: 0,
        blocks : [
            {
                    offset: 0,
                    style: 'comment',
            }
        ]
    },
    {
        offset: 10,
        blocks: [
            {
                offset: 0,       // $text
                style: 'var',
            },
            {
                offset: 5,       //  =
                style: 'normal',
            },
            {
                offset: 8,       // "string text"
                style: 'string',
            },
            {
                offset: 22,      // ;
                style: 'normal',
            }
        ]
    },
]

This data structure is specially suited to parse source code, where most tokens can’t span over multiple lines. However, it’ll be very limited for languages like XML until we don’t support a stack based tokenizer (just a simple finite state machine).

That’s it for now, next installment will probably focus on how to handle the keyboard events :)

document.createElement('INPUT')

2006-03-22T00:00:00+01:00

Just a quick post about yet another Internet Explorer glitch

When you want to add a new input element to a page, you must define the type attribute before including the new element in the document tree.

var inp = document.createElement('INPUT');
inp.setAttribute('type', 'checkbox');
document.body.appendChild(inp);

I was going crazy because I had the 2nd and 3rd lines swapped, which makes Internet Explorer display the input element as type text and gives an exception when trying to change the type.

In my coding style, even if it’s not the more efficient one, I try to append the new elements just after creating them so I don’t forget to do it.

Ideally, to optimize the execution speed, we should always add the element when it’s completely defined, to avoid browser redraws. Even if we need to add a bunch of elements, it’s better to create them in a document fragment, outside the document tree, and when we are done just insert the whole fragment.

My take on a flexible persmission system

2006-03-17T00:00:00+01:00

Since I started programming using data sources and more specially web apps, I’ve always been looking for a flexible enough permission system to use in my projects. No matter how complicated the permission system implemented was that the client would ask for a given combination which isn’t supported by it >:(

Just recently a fellow developer told me about OpenLdap permissions system. I’ve been using a central repository (I call it registry) for all entities in my database schemas, this allows me to do a very light-weight pseudo object oriented data layer on top of a relational database like MySQL. This system is a bit similar to an LDAP datastore, after all it’s nothing more than a hierarchical object database. However, studying more in detail OpenLdap’s permission system I had all of the sudden the following idea.

Permission checks must be flexible and very fast, say you have a database with 10 thousend contacts which can be readed by the users in the same group as the owner and edited by the administration group and by the owner. Obviously you can’t waste 2 seconds just to query the database for the contacts an user is able to see in a listing. So I tried to simplify the whole concept, starting by what is a permission, when applied to an object/entity/resource a permission can be None, Read, Write or Delete. Things like Print or Export are in fact the Read permission, only with a different presentation layer. Since it’s only four possible values we can represent it with a simple integer like 0, 1, 2 and 4 respectively. So now, in this system, you have a registry with all the entities and another table with all the users. The only missing piece is to relate the entities with the users, for that the simplest thing to do (and the more flexible) is just to create a relation table which has a composite primary key of the entity id and the user id, as an attribute we put the access level we described as an int above. Now, we said we had 10.000 contacts or entities for this matter, and say we have 100 users, the resulting relation table could be upto 10.000×100 = 1 million rows! That is quite a lot of rows, even if it’s quite doubtfull that we will use the maximum of rows, it’s clear that this system has a drawback and this is that it won’t scale well with a huge number of users. I can live with it actually, at least it serves my purposes. If someone needs an application for some hundreds of users he may probably spend some bucks on a powerfull server :)

So now we have a system which allows for really flexible permissions but which only requires a simple table join to check them. Obviously it’s not operative right now, since we need a way to build the ACL table. In this task is where the OpenLdap method comes in handy. Basicly it allows to create rules based on the entity attributes (aka entity fields), so we can create a simple parser for this rules and populate the ACL table accordingly. Take the following example acl, it defines that all users can read any entity and the owner can write to it:

 access to *
       by self  write
       by users read
       by *     none

I’m not saying we need to implement OpenLdap’s syntax, it probably doesn’t fit too well a relational model, however it helps to ilustrate how we can achieve a very complex permissions system. One thing to note is that when the permission definition is modified, the whole ACL table needs to be recalculated. However, when we add a new entry we just need to calculate that new entry, so the overall performance is quite good.

In my opinion what this method offers compared to other commonly used permission systems in web applications is that it calculates the permission when creating/updating objects (or modifying the permission definitions) instead of performing complex (many times also slower) calculations when selecting data, which is most cases is the most frequent operation.

El uso de RAM del Exchange Store.EXE

2006-03-16T00:00:00+01:00

Ya me he encontrado en varios servidores que el proceso Store.EXE del Microsoft Exchange usa cantidades desmesuradas de memoria.

En realidad esto lejos de ser un problema, es una característica del Exchange desde la versión 2000 creo. Para explicar porque hace esto de chupar memoria como si fuera un vampiro biteriano, primero hay que explicar como funciona el sistema de almacenamiento del Exchange.

Cuando recive un nuevo correo/contacto/fichero, vamos lo que sea que tenga que ser almacenado en los buzones, lo primero que hace es escribirlo en un fichero de logeado. De esta forma no tiene que estar esperando a que la base de datos este disponible para meter el mensaje, así que simplemente guarda la nueva petición en un fichero de log y vuelve a la espera de nuevas entradas. Por otro lado, un subproceso se encarga de ir procesando esos logs y va introduciendo las nuevas entidades en la base de datos. Como no tiene prisa por volver a escuchar nuevas peticiones, puede introducir los elementos en la base de datos de una forma adecuada. Adicionalmente, este sistema tiene la ventaja de que si se colgara el servidor (se va la luz, peta el exchange, vuelcas el café en el teclado…), no se perdería nada, ya que los datos se han guardado en los fichero de log, y aunque la base de datos quede corrupta por una operación no finalizada, Exchange puede restaurarla a un estado optimo haciendo uso de esos mismos logs.

Todo este rollo, nos viene al pelo para explicar el porque del alto consumo de RAM del proceso Store.EXE. Lo que hace es guardar los datos en el log y al mismo tiempo mantenerlos en memoria, a modo de cache, así que usará toda la memoria que tengamos disponible para crear un cache lo más eficiente posible. Por supuesto, los ingenieros de Microsoft no son tan tontos como para pensar que por ser quienes son tienen derecho a ‘secuestrar’ toda la RAM de un servidor. Lo que hace este proceso es usar una técnica llamada Dynamic Buffer Allocation, la cual va usando toda la memoria que quiere pero vigila el uso de la memoria virtual (cache de disco), si detecta que empieza a usarse la memoria virtual, el proceso comenzará a reducir su cache para dejar RAM libre a otras aplicaciones.

Como nota adicional, en mi busqueda de información he leido que para calcular la RAM necesaria para un Exchange (version 2003) se puede usar esta formula: 256Mb + 1Mb * Buzón Logicamente cuanto más RAM tenga mejor rendimiento, debido al uso del mencionado cache. Por cierto, para servidores con más de 1Gb se recomienda usar las opciones /3GB /USERVA=3030 en el boot.ini, y cambiar el valor del registro HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\HeapDeCommitFreeBlockThreshold a 0×00040000, para evitar la fragmentación de la memoria.

Browser based syntax highlighting code editor

2006-02-21T00:00:00+01:00

I’ve been looking for a working source code / plain text editor which can be run on a browser. There are dozens of WYSIWYG html editors out there (see here for a comprehensive list). However, for pure source code or just plain text I just found two options: Helene and About:Edit.

The first one, Helene, is open source software but it’s not ready for production use. The other one, About:Edit, looks really cool but it’s commercial which is good but I can’t afford to buy a license, so it’s not a realistic option in my case.

Thinking a bit about the main problem of that kind of editors, it seems to be that browsers don’t support natively the editing of hightlighting code, even if it can be emulated with the WYSIWYG editors it doesn’t feel like it’ll be very efficient and/or clean. So I took another approach, which is a bit of a hack but I think that can solve my needs pretty well. The hack is just to use modern browsers absolute positioning and opacity settings. We just place a div below a textarea, with the same sizes, then put the opacity of the textarea to 30% and we set the text color to white. Then we just need to update the underlying div with the changes in the textarea, all the text editing, cut&paste, selection … is handled natively by the browser but we can style the contents (like syntax highlighting) in the div.

It doesn’t look perfect, since the half transparent textarea makes the colors look too soft but the speed is great and it’s really easy to implement.

Expect a simple demo sometime next week, I’m a bit too busy right now too expend my time on this kind of experiments :(

Goodbye MacroCSS, long live CSSing

2006-02-13T00:00:00+01:00

I’ve just spend an hour chatting with Daniel Mota (aka IceBeat) about his project CSSing. I’ve been following its developement for some months now, in fact I started my own version, MacroCSS, since he didn’t have released the source code at that time.

I didn’t have the time to continue the development of MacroCSS, I did manage though to implement a SAC based parser for CSS which turned out to be quite powerful. IceBeat was searching with a post in his blog for a replacement for his regular expression based parser. I pointed him to MacroCSS and he was quite impressed with it.

After chatting with him I’m quite happy, not just because he liked my parser but because we almost share the same vision for this project. Since as I said, I don’t have the time to continue the development of MacroCSS on my own, I’m deprecating it and I’ll focus on helping IceBeat with CSSing in what I can.

I really think this project could be a real hit for the web development comunity, it fits really well with other ‘web 2.0’ (hell, how much I hate that term) technologies like ajax (I hate this one even more). So if anyone is reading this and wants to help out, he/she will be more than welcome.