Mika Tuupola

HTML5 Drag and Drop Multiple File Upload

2009-10-12T14:59:00Z

Previously I experimented with drag and drop file upload with Google Gears. Recently FireFox 3.6 (codenamed Namoroka) ~~was the first to implement File API~~. It enables JavaScript to interact with local files.

Correction: Ionut G. Stan pointed out that File API was actually available already in FireFox 3.0. What Namoroka introduced is the drag and drop interface for the files. Sorry for the confusion.

Here is how you can implement drag and drop multiple file upload with native JavaScript. No plugins needed. Just plain ~~old~~ new HTML5. Again there is a working demo. You will need FireFox 3.6 to test it. Full source code can be at GitHub.

PHP Upload Script

PHP script for receiving the files is the same as last time. It accesses uploaded files using $_FILES superglobal. For this demo we assume users upload only image files.

<?php

$error_message[0] = "Unknown problem with upload.";
$error_message[1] = "Uploaded file too large (load_max_filesize).";
$error_message[2] = "Uploaded file too large (MAX_FILE_SIZE).";
$error_message[3] = "File was only partially uploaded.";
$error_message[4] = "Choose a file to upload.";

$upload_dir  = './tmp/';
$num_files = count($_FILES['user_file']['name']);

for ($i=0; $i < $num_files; $i++) {
    $upload_file = $upload_dir . basename($_FILES['user_file']['name'][$i]);

    if (!preg_match("/(gif|jpg|jpeg|png)$/",$_FILES['user_file']['name'][$i])) {
        print "I asked for an image...";
    } else {
        if (is_uploaded_file($_FILES['user_file']['tmp_name'][$i])) {
            if (move_uploaded_file($_FILES['user_file']['tmp_name'][$i], 
                $upload_file)) {
                /* Great success... */
            } else {
                print $error_message[$_FILES['user_file']['error'][$i]];
            }
        } else {
            print $error_message[$_FILES['user_file']['error'][$i]];
        }    
    }
}
?>

Setup Drop Target

Also JavaScript code is almost identical with the Google Gears version. Since only FireFox is supported we can leave Mozilla and IE initialization out. We only need to attach drop event to div with id #dropzone. Event will trigger function called upload().

$(function() {

    /* Cannot use $.bind() since jQuery does not normalize native events. */
    $('#dropzone')
        .get(0)
        .addEventListener('drop', upload, false);

    function upload(event) { 
        /* Uploading will here. */
    }

}

Upload Files With Native JavaScript

JavaScript is used to build RFC2388 string. This string is POST:ed using XMLHttpRequest to our PHP script. PHP will see the request as if it was POST:ed from normal multipart/form-data form.

There are three differences to Gears version:

RFC2388 string is built using JavaScript string variable instead of Gears BlobBuilder API.
Data is POST:ed using native XMLHttpRequest instead of Gears HttpRequest API.
Information on dropped files are accessed using event.dataTransfer variable instead of desktop.getDragData() function call.

function upload(event) {

    var data = event.dataTransfer;

    var boundary = '------multipartformboundary' + (new Date).getTime();
    var dashdash = '--';
    var crlf     = '\r\n';

    /* Build RFC2388 string. */
    var builder = '';

    builder += dashdash;
    builder += boundary;
    builder += crlf;

    var xhr = new XMLHttpRequest();

    /* For each dropped file. */
    for (var i = 0; i < data.files.length; i++) {
        var file = data.files[i];

        /* Generate headers. */            
        builder += 'Content-Disposition: form-data; name="user_file[]"';
        if (file.fileName) {
          builder += '; filename="' + file.fileName + '"';
        }
        builder += crlf;

        builder += 'Content-Type: application/octet-stream';
        builder += crlf;
        builder += crlf; 

        /* Append binary data. */
        builder += file.getAsBinary();
        builder += crlf;

        /* Write boundary. */
        builder += dashdash;
        builder += boundary;
        builder += crlf;
    }

    /* Mark end of the request. */
    builder += dashdash;
    builder += boundary;
    builder += dashdash;
    builder += crlf;

    xhr.open("POST", "upload.php", true);
    xhr.setRequestHeader('content-type', 'multipart/form-data; boundary=' 
        + boundary);
    xhr.sendAsBinary(builder);        

    xhr.onload = function(event) { 
        /* If we got an error display it. */
        if (xhr.responseText) {
            alert(xhr.responseText);
        }
        $("#dropzone").load("list.php?random=" +  (new Date).getTime());
    };

    /* Prevent FireFox opening the dragged file. */
    event.stopPropagation();

}

And there you have it. Native drag and drop multiple file upload.

Drag and Drop File Upload With Google Gears

2009-10-02T09:27:00Z

Google Gears is an extension which adds new features to you browser. It lets your browser to interact with the desktop. You can store data locally to an SQLite based database. Browser will also have a worker pool for running JavaScript code on the background.

Update 20091007: I updated the tutorial and demo to support dragging and dropping multiple files for uploading simultaneously.

Below is how you can do basic drag and drop file upload. Gears, PHP and jQuery are needed. If you want to try there is a working demo. All source code for the demo can be found from GitHub.

PHP for Receiving the Files

PHP has easy way of accessing uploaded files using $_FILES array. For demo’s sake I am assuming we should upload only image files. Nothing special here. Just you plain old upload script.

<?php

$error_message[0] = "Unknown problem with upload.";
$error_message[1] = "Uploaded file too large (load_max_filesize).";
$error_message[2] = "Uploaded file too large (MAX_FILE_SIZE).";
$error_message[3] = "File was only partially uploaded.";
$error_message[4] = "Choose a file to upload.";

$upload_dir  = './tmp/';
$num_files = count($_FILES['user_file']['name']);

for ($i=0; $i < $num_files; $i++) {
    $upload_file = $upload_dir . basename($_FILES['user_file']['name'][$i]);

    if (!preg_match("/(gif|jpg|jpeg|png)$/",$_FILES['user_file']['name'][$i])) {
        print "I asked for an image...";
    } else {
        if (is_uploaded_file($_FILES['user_file']['tmp_name'][$i])) {
            if (move_uploaded_file($_FILES['user_file']['tmp_name'][$i], 
                $upload_file)) {
                /* Success... */
            } else {
                print $error_message[$_FILES['user_file']['error']];
            }     
        } else {
            print $error_message[$_FILES['user_file']['error']];
        }    
    }    
}
?>

Initialize Gears and Setup Drop Target

Gears will be initialized after DOM is ready. We need desktop support for dragging and dropping. HTTPrequest support is used for uploading the files.

We will also setup event handlers for file dropping. Target is a div with id #dropzone. Different browsers use different event names. Unfortunately jQuery does not normalize native dropping event name. Thus we need to initialize the event separately for each browser. We also need to use native DOM element fetched with .get(0). When drop happens upload() function will be called.

When dropping a file default action for browser would be top open it. With Safari and IE this can be prevented by false from dragover event. With FireFox you need to call event.stopPropagation() later in the upload() code.

$(function() {

    /* Google Gears support. */
    var desktop = google.gears.factory.create('beta.desktop');
    var request = google.gears.factory.create('beta.httprequest');

    /* We cannot use $.bind() since jQuery does not normalize the */
    /* native events. */
    if ($.browser.mozilla) {
        $('#dropzone')
            .get(0)
            .addEventListener('dragdrop', upload, false);
    } else if ($.browser.msie) {
        $('#dropzone')
            .get(0)
            .attachEvent('ondrop', upload, false);
        $('#dropzone')
            .get(0)
            .attachEvent('ondragover', function(event) { 
                event.returnValue = false; 
            }, false);                
    } else if ($.browser.safari) {
        $('#dropzone')
            .get(0)
            .addEventListener('drop', upload, false);        
        $('#dropzone')
            .get(0)
            .addEventListener('dragover', function(event) { 
                event.returnValue = false; 
        }, false);
    }
};

Upload Files With JavaScript

There are two ways of uploading files with JavaScript. I will emulate normal HTML multipart/form-data type of form. PHP has native support for uploading files this way. JavaScript side will be slightly more complicated though.

Gear blobbuilder is used for building a string according to RFC2388. Most of the code in upload() function is about building the multipart/form-data string. This string is then submitted to PHP script with Gears httprequest.

If upload.php returns something we assume it is an error and we alert() it. Otherwise we update the content of #dropzone with a thumbnailed list of images. This list is generated by list.php script.

function upload(event) {

    var data = desktop.getDragData(event, 'application/x-gears-files');

    var boundary = '------multipartformboundary' + (new Date).getTime();
    var dashdash = '--';
    var crlf     = '\r\n';

    /* Build RFC2388 string. */
    var builder = google.gears.factory.create('beta.blobbuilder');

    builder.append(dashdash);
    builder.append(boundary);
    builder.append(crlf);

    for (var i in data.files) {

        var file = data.files[i];

        /* Generate headers. */
        builder.append('Content-Disposition: form-data; name="user_file[]"');
        if (file.name) {
            builder.append('; filename="' + file.name + '"');
        }
        builder.append(crlf);

        builder.append('Content-Type: application/octet-stream');
        builder.append(crlf);
        builder.append(crlf); 

        /* Append binary data. */
        builder.append(file.blob);
        builder.append(crlf);

        /* Write boundary. */
        builder.append(dashdash);
        builder.append(boundary);
        builder.append(crlf); 
    }

    /* Mark end of the request. */
    builder.append(dashdash);
    builder.append(boundary);
    builder.append(dashdash);
    builder.append(crlf);        

    request.onreadystatechange = function() {
        switch(request.readyState) {
            case 4:
                /* If we got an error display it. */
                if (request.responseText) {
                    alert(request.responseText);
                }
                $("#dropzone").load("list.php?random=" +  (new Date).getTime());
                break;
        }
    };

    /* Use Gears to submit the data. */
    request.open("POST", "upload.php");
    request.setRequestHeader('content-type', 
        'multipart/form-data; boundary=' + boundary);
    request.send(builder.getAsBlob());

    /* Prevent FireFox opening the dragged file. */
    if ($.browser.mozilla) {
        event.stopPropagation();
    }

}

Other Demo Code

For demo’s sake there is a simple script which produces thumbnails of uploaded images. It uses ImageResize class taken from Peter Gassners Image Resize Frog CMS plugin.

<?php

require_once 'ImageResize.php';

foreach (glob("./tmp/*") as $source) {
    $file = pathinfo($source);
    if (!preg_match('#-100x100\.([a-z]+)$#i', $source)) {
        $destination = $file['dirname'] . '/' . $file['filename'] . 
                               '-100x100.' . $file['extension'];
        ImageResize::image_scale_cropped($source, $destination, 100, 100);
        printf('<a href="%s"><img src="%s" widht="100" height="100" /></a>', 
                 $source, $destination);
    }
}
?>

Known Issues

Gears is currently broken with FireFox 3.5.x. Hopefully issue will be fixed with next release. If you find any other issues or have feedback feel free to leave comment.

Estonian Internet Awards 2009

2009-09-21T13:57:00Z

All photos by Annika Haas, Fotobrigaad.

It was talked about in several different places. Last thursday it became reality. First Estonian Internet Awards winners were announced. These are my comments as a member of jury.

The Winners

There was two world level entries. Edicy which was judged as an online service (not just a website). Fraktal is setting a benchmark for Estonian startups on how things should be done. Stop Killing Dogs which was great example of a campaign where social media usage has a point. At the same time it was clever self promotion of Uniteddogs.com. Why a great self promotion? Because was not trying to promote too much.

Rest of shortlisted entries were good. Nothing earth shattering. Good basic work. I especially liked Velvet’s Rahvusmeened.ee and Estonian Air Tantsupidu banner.

What Was Missing?

There are couple of good projects I think should have been submitted. SeeMe by Oskando was missing. Maybe it was released before 2008? Unitedcats.com and Uniteddogs.com as a service were missing too. Metrix would have been a great non-profit candidate.

More Catchy Name?

Competition should be branded with a name. Now it is hard to descibe what you won. “I won bronze in campaigns category of Estonian Internet Awards”. This is a bit awkward. It would be much easier to say “I won bronze jaaniuss in campaigns category.”

About Categories

I think next year the categories could be adjusted. Too many categories seemed to cause quite much confusion. Most of subcategories were merged together anyway. My suggestion for next years categories are:

B2C services
B2B services

Service could be a corporate website, online bank, online newspaper, online shop etc. I’m sure you get the point.

B2C campaigns
B2B campaigns

Campaign can be a mini- or microsite, banner, email newsletter or combination of all of them.

Best online ad

Banner or a banner like advertisement. Can be also a banner submitted as part of project in campaigns category.

Non-profit

Because not everyone has a large budget.

Best design

To keep those full Flash websites with sound and flying stuff from polluting other categories.

Best results

Because results are important. Sometimes campaign can look awful but have spectacular results. This is the category for them.

Thanks to All!

Big thanks to J.Margus Klaar of ADC Estonia and Hando Sinisalu and Silja Oja of Best Marketing for organizing the event. Big up to Robin Gurney of Altex Marketing being the chairman of the jury. Also hi-five to all other jury members. It was nice to finally meet you all in real life.

Verify Estonian Isikukood With JavaScript

2009-09-04T11:37:00Z

Yesterday I had to verify Estonian national identification number with JavaScript. The formula is pretty simple. You just have to read it through couple of times to understand. If you need other languages Dmitri Smirnov has examples in PHP, Java and Ruby.

Logging API for Frog Dashboard

2009-05-18T09:17:00Z

Dashboard plugin for Frog CMS now provides simple API for other developers to log their events. Whenever you want to log something to dashboard just trigger a log event. Include your message as parameter.

Observer::notify('log_event', 'Something was done by :username.');

In your message you can include string :username to log the user name.

Priority and Ident

You can also provide priority and ident for the log message. Priority is used to inform about severity of the message. Ident is used to identify sender of the message. Plugin developers should use the ident of their plugin. For example Funky Cache plugin uses funky_cache as ident.

Observer::notify('log_event', 'Something else was done by :username.',
                               DASHBOARD_LOG_NOTICE, 'pluginname');

Priorities are defined in Dashboard code.

define('DASHBOARD_LOG_EMERG',    0); /* system no longer available */
define('DASHBOARD_LOG_ALERT',    1); /* immediate action required */
define('DASHBOARD_LOG_CRIT',     2); /* critical condition */
define('DASHBOARD_LOG_ERR',      3); /* error condition */
define('DASHBOARD_LOG_WARNING',  4); /* warning messages */
define('DASHBOARD_LOG_NOTICE',   5); /* normal, but significant, condition */
define('DASHBOARD_LOG_INFO',     6); /* general informative messages */
define('DASHBOARD_LOG_DEBUG',    7); /* debugging information */

If your code uses constants and Frog is in debug mode and Dashboard plugin is not installed this will cause PHP to display “Use of undefined constant” warnings. If you want to be sure your user will never see these warnings you can use number instead of constant:

Observer::notify('log_event', 'Something else was done by :username.',
                               4, 'pluginname');

You could also define the constants yourself. But make sure they are not already defined (Dashboard installed). Defining constants twice will cause warnings.

Download

Download the latest tarball or checkout from GitHub.

Bloggers Meeting at Altex

2009-04-20T13:21:00Z

Altex together with Pärnu Turunduskonverets organized a small bloggers meeting. Aim was to discuss same subjects which will be present in the real conference.

How to achieve big things in Internet with small amount of money?
How to convince your boss to invest in online marketing?
How to achieve competitive edge during recession?
Who will survive recession better – online or offline stores?

DISCLOSURE: People who attended the bloggers were kindly asked (but not required) to mention and link to the conference page in a blog entry. Least we can do as a thanks to nice food and beverages.

Money Money Money

Us having short attention span discussion varied a bit. With money talk we figured out everything costs something. Blogging might seem like its free but when you put pricetag to your time it is not so cheap anymore. There was also some discussion of age old problem people thinking that because Internet equals computers it must equal cheap to do.

One issue which raised was how Estonian generally avoid using Paypal. It is for some reason found to be cumbersome to use (which I and others did not agree). There might be trust issues. Since Paypal is not any of the local banks (which are owned by Swedes anyway) it is not trusted.

In off topic discussion some had noted that there are problems sending properly Google tagged banners links to media. Long links usually get broken because advertising section cut and pastes only half of the URL.

Conference

But this blog post was supposed to be about the conference. On friday 15th of May the interesting parts are.

SMALL BUDGET, BIG BENEFIT – EXPERIENCE FROM SOCIAL NETWORKS
Hille Hinsberg, Estonian goverment & Epp-Kristiina Keerov, Altex

WHAT DO PEOPLE DO IN THE INTERNET
Anni Ronkainen, Google

HOW TO SELL E-MARKETING TO YOUR BOSS
Craig Hanna, e-consultancy.com

WILL RECESSION KILL OFFLINE OR ONLINE BUSINESS?
Aivar Paalberg, Enter IT

State of Kuldmuna Digital and What To Do About It?

2009-04-09T15:08:00Z

Last weekend Kuldmuna awards were given out. Someone is going to be offended by saying this. Level of most of entries shortlisted in Internet design and Digital advertising categories were shamefully low. I know there have been better projects in Estonia during last year. How did this happen?

DISCLAIMER: I work in advertising agency myself. I have been working in advertising agency or online department of advertising agency for last 11 years. Good to remember when you work in advertising and feel offended by reading this.

Kuldmuna Is an Advertising Competition

This is the main reason. Kuldmuna is about advertising by advertising people. You can see it from the beginning. When submitting your project only names of project manager, copywriter, AD and creative director are asked. Same problem continues in the website and in the Kuldmuna book.

It might be surprising to some but in web projects there is much more. Frontend coder, backend coder and Flash designer are first to come into mind. With larger projects information architect, UI designer and UX designer might be involved. If nothing else please just add one field called Technical designer for these people.

Judging With Advertising Mindset

Entries get judged with advertising mindset. In worst case just by showing a screenshot or presentation of a site. Judges don’t always have time or possibility to click through the site. We are not even talking about checking code quality here. It ends up being a beauty competition.

Easiest way to win is to make full Flash site with flying stuff and sounds. Even better if you invent some new “innovative” navigation. This is sometimes done in purpose. Agencies do separate projects which are good in Internet sense. Other projects which are just done for winning awards.

Smart agencies have learned what to submit to advertising competition. Good online projects do not get submitted to an advertising competition. They won’t win anything anyway.

What To Do About It?

We had exactly the same problem in Finland. Solution was Grand One. We could do exactly the same here in Estonia – Our own online awards organized and judged by online people.

I am willing to invest my own time and money for this. I just need some native Estonian speaker to join me (I do speak Estonian, just suck at writing it).

Starting Steps

I already figured out name for the awards. I have discussed the name with Estonian colleagues and everyone approved the idea. Name of the competition would be “Au tööle!”.

To start things quickly Au tööle! Twitter channel was set up. Purpose of this channel is to have place which lists what online projects agencies are doing. Yes, there is pixel.ee but only some projects are submitted there. To add your project send a private tweet or email lisa@autoole.ee. Send the following info:

Name of the project
URL to the project
Name of the company(ies) who created it

This solves another problem we have. Lots of projects get done without other agencies knowing about them. It is nice to know what other are doing. Please send recent projects. Do not spam with everything you have done during last year.

If Twitter channel becomes we could start giving out smaller awards. Once a month might be too often. Every other month is a good start. Winner would be chosen by three judges. New judges are chosen each time. Judges will come from different agencies and there should be at least one designer and one coder.

Beginning of next year would be the first gala. Up to here everything has been free. Companies submitting projects to main event would have to pay a fee. There should be one category for non-profits or bloggers where entrance fee is free. Categories would be:

Best campaign
Best online service
Best design
Best online ad (banners etc)
Grand Prix

How Do You Feel About It?

What do you think? You think I am crazy? It will never work? You would like to help? One thing is for sure – something needs to be done. If you have any ideas or suggestion leave a comment (in english or estonian).

Lazy Load Inside Container

2009-02-13T10:21:00Z

Matthew Crumley contributed nice patch to Lazy Load plugin which makes it work with containers. If you have a container which has a scrollbar.

#container {
    height: 600px;
    overflow: scroll;
}

Images which are not visible are not loaded until you scroll to them. Check demo page for horizontal and vertical scrolling.

To use new feature you can give the container as jQuery object.

$("img").lazyload({         
     placeholder : "img/grey.gif",
     container: $("#container")
 });

Mathew also patched a bug where IE was not always loading images. To upgrade download the latest source, minified or packed.

Search jQuery API Docs from Spotlight

2009-02-09T21:56:00Z

Yesterday Priit Haamer of Fraktal notified me about Spotlight searchable Ruby on Rails documentation he had made. Absolutely brilliant idea. When I saw it I knew I have to do same thing for jQuery.

Why Is It cool?

You can just hit Apple + Space to enter spotlight and search for jQuery function.

Press and hold Apple + Control + D over any function in TextMate (or Safari, iChat, Mail etc.) to get info popup.

Docs are in your harddrive so they are always accessible. Even when you or docs.jquery.com are offline.

It is native, based on OS X dictionary. No need to install AIR or any other applications.

Install From Zip

Download jQuery.dictionary.zip file. Unzip it and move to folder ~/Library/Dictionaries or /Library/Dictionaries. Enable it from Dictionary preferences.

Sometimes it takes a while for Spotlight to see new dictionary. You can speed things up by restarting it.

>killall Spotlight

According to Priit you should also drag new dictionary to be first on the list. This was not true in my case. However if you have problems it is worth trying.

Install From Git

If you like to live in bleeding edge install from Git and build your own.

>git clone git://github.com/tuupola/jquery_dictionary.git
>cd jquery_dictionary
>make
>make install

Acknowledgements

Thanks to Priit Haamer for the original idea. David Serduke for the the Python script used exporting jQuery wiki to XML format. Jörn Zaefferer whose XSLT stylesheets were used as basis for XSLT stylesheet which converts exported jQuery docs to OS X dictionary XML source

Mephisto Style Asset Management for Frog CMS

2008-12-23T15:50:00Z

One of the things I like about Mephisto is the asset management. Especially the way how you insert image URLs by dragging thumbnail from sidebar to content area. Frog CMS lacked easy way for inserting images (or any other files). Obvious thing to do was implement Mephisto style asset management as a plugin.

It is not one to one copy on how Mephisto does it. Instead of tagging assets you can categorize them by uploading to different folders. There is also extra pulldown to select which assets to show in sidebar while editing.

Plugin depends on Image Resize and jQuery plugins to work. Full installation instructions on project page.

Send Emails from Frog CMS

2008-11-28T15:04:00Z

I needed a mailer backend which can handle complicated forms with any number of arbitary form fields. I also needed to be able to fully control the layout of sent emails. Something similar as oldie but goldie cgiemail.

Here comes Email Template plugin for Frog CMS.

It provides new page type called named Email template. You can POST your forms to this page type. Page contains the layout of the mail including the headers. Template then parses POST:ed data and sends the email.

To: Somebody <somebody@example.com>
From: <?php print $_POST['name'] ?> <<?php print $_POST['email'] ?>>
Subject: Frog Mail

1. Contact info

Name.............: <?php print $_POST['name'] ?> 
Company..........: <?php print $_POST['company'] ?> 
Email............: <?php print $_POST['email'] ?> 

2. Message

<?php print $_POST['message'] ?>

--
Sent by <?php print $_SERVER['REMOTE_ADDR'] ?>

Plugin assumes your PHP mail() function works properly. Currently it only supports plain text emails. Download and installation instructions at the project page. All feedback welcome.

Ultrafast Frog With Funky Cache

2008-11-21T15:51:00Z

Funky caching is technique popularized by PHP.net site. It was first mentioned by Rasmus Lerdorf in 2002 PHPCon slides (page 25). Content is cached as static file on the first access. All following requests are served using the cached static file. Editing a page will automatically expire cached files. Page is then re-cached on the next hit.

Frog CMS is PHP port of Rails based Radiant CMS. Radiant is great but there is always the hosting problem. Even with mod_rails existing it is still easier to get quality PHP hosting. Both Frog and Radiant are the only CMS’es I can say I like. Expression Engine I can live with. Edicy looks really promising. Everything else I rather not touch.

How About the Speed?

Frog is reasonably fast. Simple ApacheBench running on localhost (to rule out network latency) shows it can deliver around 90 requests per second.

> ab -n 1000 -c 20 http://dev.example.com/about_us.html

Time taken for tests:   10.600109 seconds
Requests per second:    94.34 [#/sec] (mean)
Time per request:       212.002 [ms] (mean)
Time per request:       10.600 [ms] (mean, across all concurrent requests)
Transfer rate:          255.66 [Kbytes/sec] received

I have seen slower. This would not stand Digg effect though.

File Cache Plugin

Using Gilles Doge’s File Cache plugin gives improvement. Test server gives 150 requests per second. Problem is requests are still routed to backend. PHP logic decides whether to show cached contents or not.

> ab -n 1000 -c 20 http://dev.example.com/about_us.html

Time taken for tests:   6.707378 seconds
Requests per second:    149.09 [#/sec] (mean)
Time per request:       134.148 [ms] (mean)
Time per request:       6.707 [ms] (mean, across all concurrent requests)
Transfer rate:          617.08 [Kbytes/sec] received

This still would not stand Digg effect.

Enter Funky Cache Plugin

Something needed to be done. Since I have had good experiences with Mephisto which uses funky caching I decided to give it a try. This is where Frog first showed its power. Software was totally new to me. Still their plugin API was so clear to understand I was able to create first working version in one evening. Second evening was spent polishing the code.

So does it work? With Funky Caching enabled test server delivered 1750 requests per second.

> ab -n 1000 -c 20 http://dev.example.com/about_us.html

Time taken for tests:   0.568727 seconds
Requests per second:    1758.31 [#/sec] (mean)
Time per request:       11.375 [ms] (mean)
Time per request:       0.569 [ms] (mean, across all concurrent requests)
Transfer rate:          7284.69 [Kbytes/sec] received

That should be enough to withstand both Digg effect and Slashdotting at the same time.

Yes Gimme Some!

Installation instructions at project page. If you want to peek under the hood check the source at GitHub. Code is still beta quality. Please leave a comment if you find a bug or you have improvement ideas.

Introduction to Marker Clustering With Google Maps

2008-11-04T20:13:00Z

Static Maps API has URL length limit of around 2048 characters. You can hit this limit quickly when adding lot of markers. You can keep URL short by clustering markers together.

Square Based Clustering

Clustering is usually done by dividing map to squares. Square size depends on map zoom level. Markers inside a square are then grouped into cluster. This technique has some limitations. Look at the following image.

Two markers are close to each other. In fact they are so close they are overlapping. Both markers are also the only marker inside their square. Because markers are in separate square they wont be clustered.

Distance Based Clustering

We can also group markers together based on their distance from each other. We could cluster all markers inside 10 kilometer radius together. There is one problem with this approach. Kilometers (and miles) have different meaning in different zoom levels. In zoomed in map it might mean 100 pixels. In zoomed out maps one kilometer might be only one pixel.

There is only one distance unit which does not have this problem: pixels in current zoom level. One pixel on screen is always one pixel on screen. For example we want to cluster all markers which are 20 pixels from each other. I chose 20 pixels because it happens to be the distance after which markers start to overlap each other.

Now the two markers would be clustered since they are inside 20 pixel radius.

Distance Between Two Coordinates on Earth

Distance between two points on earth can be calculated in several ways. Haversine formula is reasonably accurate and widely used. It assumes earth is spherical (in reality earth is slightly ellipsoid). This causes accuracy to be +-2 km when calculating distances of around 20.000 km. 6371.0 km is used as average radius of earth.

Below is PHP implementation of Haversine formula:

function haversineDistance($lat1, $lon1, $lat2, $lon2) {
    $latd = deg2rad($lat2 - $lat1);
    $lond = deg2rad($lon2 - $lon1);
    $a = sin($latd / 2) * sin($latd / 2) +
         cos(deg2rad($lat1)) * cos(deg2rad($lat2)) *
         sin($lond / 2) * sin($lond / 2);
         $c = 2 * atan2(sqrt($a), sqrt(1 - $a));
    return 6371.0 * $c;
}

But didn’t wee need distance in pixels instead? For that we can use Pythagoras’ theorem. Pythagoras’ theorem uses cartesian (pixel) coordinates. Some Mercator magic can be used to convert latitude and longitude to pixel x and y values.

You might wonder where did number 268435456 come from? It is half of the earth circumference in pixels at zoom level 21. You can visualize it by thinking of full map. Full map size is 536870912 × 536870912 pixels. Center of the map in pixel coordinates is 268435456,268435456 which in latitude and longitude would be 0,0.

define('OFFSET', 268435456);
define('RADIUS', 85445659.4471); /* $offset / pi() */

function lonToX($lon) {
    return round(OFFSET + RADIUS * $lon * pi() / 180);        
}

function latToY($lat) {
    return round(OFFSET - RADIUS * 
                log((1 + sin($lat * pi() / 180)) / 
                (1 - sin($lat * pi() / 180))) / 2);
}

function pixelDistance($lat1, $lon1, $lat2, $lon2, $zoom) {
    $x1 = lonToX($lon1);
    $y1 = latToY($lat1);

    $x2 = lonToX($lon2);
    $y2 = latToY($lat2);

    return sqrt(pow(($x1-$x2),2) + pow(($y1-$y2),2)) >> (21 - $zoom);
}

Now we have all needed mathematics in place. What to do with them?

Cluster Markers Together

Let’s write example clusterer function. It takes three parameters:

Array of lat and lon locations.
Distance in pixel inside which markers will be clustered.
Current map zoom level.

Function will return another array where coordinates closer than $distance are clustered together.

function cluster($markers, $distance, $zoom) {
    $clustered = array();
    /* Loop until all markers have been compared. */
    while (count($markers)) {
        $marker  = array_pop($markers);
        $cluster = array();
        /* Compare against all markers which are left. */
        foreach ($markers as $key => $target) {
            $pixels = pixelDistance($marker['lat'], $marker['lon'],
                                    $target['lat'], $target['lon'],
                                    $zoom);
            /* If two markers are closer than given distance remove */
            /* target marker from array and add it to cluster.      */
            if ($distance > $pixels) {
                printf("Distance between %s,%s and %s,%s is %d pixels.\n", 
                    $marker['lat'], $marker['lon'],
                    $target['lat'], $target['lon'],
                    $pixels);
                unset($markers[$key]);
                $cluster[] = $target;
            }
        }

        /* If a marker has been added to cluster, add also the one  */
        /* we were comparing to and remove the original from array. */
        if (count($cluster) > 0) {
            $cluster[] = $marker;
            $clustered[] = $cluster;
        } else {
            $clustered[] = $marker;
        }
    }
    return $clustered;
}

We can now test clusterer function with array of coordinates.

$markers   = array();
$markers[] = array('id' => 'marker_1', 
                   'lat' => 59.441193, 'lon' => 24.729494);
$markers[] = array('id' => 'marker_2', 
                   'lat' => 59.432365, 'lon' => 24.742992);
$markers[] = array('id' => 'marker_3', 
                   'lat' => 59.431602, 'lon' => 24.757563);
$markers[] = array('id' => 'marker_4', 
                   'lat' => 59.437843, 'lon' => 24.765759);
$markers[] = array('id' => 'marker_5', 
                   'lat' => 59.439644, 'lon' => 24.779041);
$markers[] = array('id' => 'marker_6', 
                   'lat' => 59.434776, 'lon' => 24.756681);

$clustered = cluster($markers, 20, 11);

print_r($clustered);

If you run the code you can see how marker_3, marker_4 and marker_6 are clustered together. This can better be visualized as map screenshot before and after clustering. Blue marker is a cluster.

Real Life Usage?

Obviously making an array of coordinates into a new array of coordinates is not really usefull. However the first clusterer for Static Maps I committed to GitHub uses previously described technique. Clustering a static map takes only two extra lines of code. First create a cluster. Then add it to the map object. Rest is taken care automatically.

$clusterer = Google_Maps_Clusterer::create('distance');
$map->setClusterer($clusterer);

You can see it in action in capital cities of the world map. City locations are parsed from KML file. Note that in closer zooms locations are slightly off. Coordinates have only two decimals of latitude and longitude.

Currently I have demo code only for Static Maps. Serverside clustering for Google Maps API will follow soon. Thats a promise. Cross my heart.

Simple Zoom and Pan Controls With Static Maps

2008-10-24T20:54:00Z

Most of the code from previous Static Maps experiments is now put into one clean package. Previously I showed you how to work with markers and bounds. Now we go forward and add zoom and pan controls. It takes only few lines of code. If you just started reading the series check the theory how it works. As a bonus lets add infowindows / bubbles too.

Note! Image above is just a screenshot. You can test final result in the demo.

Create Map and Some Markers

Start by creating new map object and set the size. We also need to give our API key. Markers are positioned on map using location object. Location can be latitude and longitude represented by Google_Maps_Coordinate object. Location can also be map x and y represented by Google_Maps_Point object.

Because we put markers to map the center is calculated automatically. There is no need to call $map->setCenter(). We can also calculate the closest possible zoom with $map->zoomToFit().

require_once 'Google/Maps.php';

$map = Google_Maps::create('static');
$map->setSize('540x300');
$map->setKey(API_KEY);

$coord_1 = new Google_Maps_Coordinate('58.378700', '26.731110');
$coord_2 = new Google_Maps_Coordinate('58.379646', '26.764090');

$marker_1 = new Google_Maps_Marker($coord_1);
$marker_2 = new Google_Maps_Marker($coord_2);

$map->addMarker($marker_1);
$map->addMarker($marker_2);
$map->zoomToFit();

Add Zoom and Pan Controls

Controls are created using Google_Maps_Control::create() factory method. After creating a control you must attach it to a map. This alone is not enough. When panning and zooming new map center or zoom value is passed in query string. Last line passes the values from URL to the map object.

$zoom = Google_Maps_Control::create('zoom');
$map->addControl($zoom);
$pan = Google_Maps_Control::create('pan');
$map->addControl($pan);

$map->setProperties($_GET);

Note! Image above is just a screenshot. You can test working controls in the demo.

Add Infowindows / Bubbles

Infowindows (or bubbles as they are often referred) are represented by Google_Maps_Infowindow object. You can set the content in constructor or using $bubble->setContent() method. Bubbles have a marker attached to them. Clicking the marker will open the infowindow. As with all other items you must attach them to map.

$bubble_1 = new Google_Maps_Infowindow('Foo Bar');
$bubble_2 = new Google_Maps_Infowindow('Pler pop');

$bubble_1->setMarker($marker_1);
$bubble_2->setMarker($marker_2);

$map->addInfowindow($bubble_1);
$map->addInfowindow($bubble_2);

Note! Image above is just a screenshot. You can test working infobubbles in the demo.

Where’s the Source?

If you want to play around with code you can get from github. Patches, improvements and suggestion are welcome.

git clone git://github.com/tuupola/php_google_maps.git

wget http://github.com/tuupola/php_google_maps/zipball/master

Simple Static Maps With PHP

2008-10-10T15:46:00Z

Lately I have been playing with Google Static Maps API a lot. Writing the same things again and again is tedious job. I decided to put the code together as one clean extendable package. Writing object oriented interface for generating URL is trivial. Real meat is having working zoom and pan controls on static map ~~with just 9 lines of code~~ (demo now includes also clickable markers and infowindows).

Code is still alpha quality. API might change any time. But here is a quick walkthrough of current features. We will build the map you see above step by step.

Create a Map Object

Map object is created using Google_Maps::create(‘static’) factory method. If no markers are set you also need to set the center of the map.

require_once 'Google/Maps.php';

$map = Google_Maps::create('static');

$map->setSize('540x300');
$map->setCenter(new Google_Maps_Coordinate('58.368488', '26.768908'));
$map->setZoom(8);
$map->setKey(API_KEY);

Add some markers

Location on map can be in two ways. Latitude and longitude represented by Google_Maps_Coordinate object. Or pixel x and pixel y location represented by Google_Maps_Point object. You can use both when creating a marker.

$coord_1 = new Google_Maps_Coordinate('58.378700', '26.731110');
$coord_2 = new Google_Maps_Coordinate('58.368488', '26.768908');
$coord_3 = new Google_Maps_Coordinate('58.268488', '26.768908');

$marker_1 = new Google_Maps_Marker($coord_1);
$marker_2 = new Google_Maps_Marker($coord_2);
$marker_3 = new Google_Maps_Marker($coord_3);

$marker_1->setColor('green');
$marker_2->setColor('blue');
$marker_3->setColor('orange');

$map->setMarkers(array($marker_1, $marker_2, $marker_3));

Automatically Calculate Zoom and Center

Google Static Maps API can automatically calculate zoom and center for you. However there is no way to know what zoom level it chose. If you need to know automatically calculated zoom and center use $map->zoomToFit() method.

Here we also add new marker using pixel coordinates. Note that we also clear the center of the map we set in the beginning. This allows map to recenter the map according to markers on the map.

$point_1  = new Google_Maps_Point('308107197', '160958681');
$marker_4 = new Google_Maps_Marker($point_1);
$map->setCenter(false);
$map->addMarker($marker_4);
$map->zoomToFit();

Show Marker Bounds

Sometimes you need to be able to visually see bounding box where all the markers fit in.

$map->showMarkerBounds();

Show Map Bounds at Chosen Zoom Level

You can also calculate and show map bounds at any zoom level. Example below displays map bounds at zoom level 8. Map itself is at zoom level 7.

$map->setZoom(7);
$map_bounds = $map->getBounds(8);
$map->setPath($map_bounds->getPath());

Where’s the Source?

If you want to play around with code you can get from github. Patches, improvements and suggestion are welcome.

git clone git://github.com/tuupola/php_google_maps.git

wget http://github.com/tuupola/php_google_maps/zipball/master

Mika Tuupola

HTML5 Drag and Drop Multiple File Upload

PHP Upload Script

Setup Drop Target

Upload Files With Native JavaScript

More Reading

Drag and Drop File Upload With Google Gears

PHP for Receiving the Files

Initialize Gears and Setup Drop Target

Upload Files With JavaScript

Other Demo Code

Known Issues

Estonian Internet Awards 2009

The Winners

What Was Missing?

More Catchy Name?

About Categories

Thanks to All!

Verify Estonian Isikukood With JavaScript

Logging API for Frog Dashboard

Priority and Ident

Download

Bloggers Meeting at Altex

Money Money Money

Conference

State of Kuldmuna Digital and What To Do About It?

Kuldmuna Is an Advertising Competition

Judging With Advertising Mindset

What To Do About It?

Starting Steps

How Do You Feel About It?

Lazy Load Inside Container

Search jQuery API Docs from Spotlight

Why Is It cool?

Install From Zip

Install From Git

Acknowledgements

Mephisto Style Asset Management for Frog CMS

Send Emails from Frog CMS

Ultrafast Frog With Funky Cache

How About the Speed?

File Cache Plugin

Enter Funky Cache Plugin

Yes Gimme Some!

Introduction to Marker Clustering With Google Maps

Square Based Clustering

Distance Based Clustering

Distance Between Two Coordinates on Earth

Cluster Markers Together

Real Life Usage?

Simple Zoom and Pan Controls With Static Maps

Create Map and Some Markers

Add Zoom and Pan Controls

Add Infowindows / Bubbles

Where’s the Source?

Simple Static Maps With PHP

Create a Map Object

Add some markers

Automatically Calculate Zoom and Center

Show Marker Bounds

Show Map Bounds at Chosen Zoom Level

Where’s the Source?