Web Species blog

One month later... Azure+ is dead

2011-11-17T17:27:00-08:00

What a ride this was… Just a bit more than a month ago I posted an article on the project we were secretly working on - Azure+, the PHP cloud platform built on top of Windows Azure infrastructure. From then functionality was improved, amazing additions were planned, I travelled tens of thousands of miles and showed it to hundreds of people. And now I can announce that it’s dead. Here is what happened.

If you want to know the “Why?” part you can skip the first few sections all together and just go straight to the sad part. However, this case is a bit complicated… Well ok, it’s very complicated. And I can’t talk about the main reason at all, so the only thing you will find is an “abstraction” of what happened. Something that I have no control of.

The launch

The blog post about the project has to be the most popular post of this blog, it received well over 10’000 unique visitors in a day. I think this is a quite big number, which was mainly fueled by social networks as everyone wanted to see what can be done with Windows and PHP. That’s why it had the “Wait… what?” at the end of the title - I assumed a lot of people would question the value of this kind of project.

And oh my god there was a lot of people who did. Microsoft guys were internet-high-five’ing us all the way, but the majority of everyone else was either still questioning the point of this or plainly calling it stupid. They had their reasons though - Windows and PHP are not really a welcomed topic, mainly because of the prejudice because it works just fine, even if PaaS shouldn’t “expose” the OS it is running on (as much as that’s possible). Also similar projects like Orchestra.io or PhpFog had taken off.

Ignoring the arguments online of whether it was a good project, the prototype that we launched worked flawlessly. One of the biggest demos of it was me doing a demo in my PHP in the Cloud keynote at the PHP Barcelona conference. Deploying an app on the stage with 500 people watching it… crazy, but it worked. Not once did it failed, even after showing it hundreds of times to all sort of different people I had a chance to show it to.

What people are looking for?

You might not realize this, but PHP ecosystem is a bit different from any other language (as they are different from each other). PHP developers usually start their careers by just hacking on some code locally, because it only takes minutes to setup a PHP environment. And if something doesn’t work - fix it and refresh. No need to recompile or deal with DLL hells. This obviously brings some disadvantages, but this post is not about how good or bad PHP is.

Because of this upbringing, the tooling for PHP developers should follow this idea. And that’s what we were trying to achieve and push the industry to do the same. One of the key elements of the platform was that you could always push code directly. As awesome as pulling from Git sounds, you can do way more than that by just executing one terminal command. And if it doesn’t work - push again and refresh.

A lot of Microsoft folks asked us about all sorts of different enterprise behaviours and how we are planning to support them. We are not, because PHP (and even non PHP) projects do not need them. Need a custom PHP setup - build your own servers, you are obviously qualified enough. Same applies to all sorts of custom or niche functionality and technologies. But most PHP projects do not need that. And that’s why we didn’t overcomplicated our solution, we solved the problem for majority of folks and for anyone else who needs some flexibility which we don’t support - platform as a service is not right for you.

Future architecture

Azure+ wasn’t about being an abstraction on top of Windows Azure, even if initially it looked like one. The reason why apps would take 15 minutes to create was because we were spinning up new instances for every new app, this takes some time. Nonetheless the goal of Azure+ was to deploy apps to a “shared server” similarly to how Heroku does it (in quotes because it was not trying to be a shared hosting platform.)

Imagine that each server has N number of slots; each slot can host an app in a fully isolated and secured environment. If it happens that your app needs more resources than the slot can provide (thus not killing other apps), you get more slots - on the same server or other servers. Apps can scale beautifully, any extra modifications are not required and price of the service can match any other PaaS and beat it. If that’s too crazy for you - you would still be able to choose dedicated boxes.

Once we would’ve had that and MySQL setup done, I’d say it would have been ready for the first public beta. We weren’t that far actually - it would only have required some hardening of the web server and PHP, a bunch of reverse proxies and a DNS server. This setup works beautifully because applications can be scaled very dynamically by us and there is a cluster supporting the infrastructure so it’s highly reliable. But then some things happened…

Good bye

I was put in a position to kill it. Not forced, but continuing it anymore would make no sense. That’s how much I’m going to say - I was advised to keep my mouth shut by some clever people in suits. Of course I’m sad to let it go after so much effort and time we have put into this, but sometimes ideas just don’t work out.

P.S.
Name Azure+ was never an issue; it was chosen as a codename and would have been changed as the project progresses.

We built a cloud platform for PHP. Wait… what?

2011-10-03T15:27:00-07:00

We built a cloud platform for PHP. Yep, you heard it correctly. We see a huge opportunity in the market and are willing to work hard to make deploying PHP projects very easy. However this is a different one and here is the story behind it and what it can do for you.

We call it Azure+. Similarly to Notepad++ relation to Notepad, Azure+ is Azure done right and usable. This is a code name though, which might change once this goes to production. As will the design, which currently works as a good basis and is based on the great Twitter Bootstrap framework.

Why Azure?

Current workflow with Azure, original from XKCD

There is nothing specific about Azure that we wanted to leverage, but because so many existing PaaS providers are built on Amazon cloud it just made sense to try something else. Furthermore, I have a lot of experience with Windows and PHP so it all felt like a good plan. I think we are awesome enough to make Azure rock for PHP, because…

Azure is just impossible to use for PHP today. This is a fact. Doesn’t matter which way you look at it, it just su.. isn’t particularly good. The amount of steps you need to make, the knowledge you need to have and the fact that you can only deploy from Windows host are some of the things which make it a very painful experience. I had enough of this pain.

What is most important, I find Microsoft’s approach and tooling lacking in so many areas, that the only way I knew how to fix this was to build a service on top, rather than release Azure+ as a product or open source project. There was and still is no way I can change the 15-20 min. deploy time (try debugging a non-working app having to wait half hour before every retry), so we built something which overcomes it.

Oh God no, Windows?!

It’s not a big surprise that Azure is running on top of Windows, it’s a Microsoft cloud at the end of a day. I know a lot of PHP developers feel very negative about Microsoft and Windows specifically. Well, Internet Explorer 6 specifically, but Windows is not better either. But that is something what you would care if this was an infrastructure service.

Azure+ is Platform as a Service or PaaS in short. What that means is that you deploy apps to a cloud black box and the infrastructure it is running is completely irrelevant to you. There is more work to be completed to making it truly PaaS, but our goal is to make deploying to this service completely headache-free and to just make everything work*.

Important fact to note, this is not developed under any collaboration or affiliation with Microsoft and thus it’s our own decisions on where we’ll take it from here. I think PHP support on Windows is as good as on any other OS and all the PHP apps I tried (Zend Framework, Symfony2, Lithium) worked pretty much out of the box.

Features

First of all, PHP developers start by writing PHP code, because to start learning PHP you only need a Apache installed and that’s it. Hack on some code, click refresh and you see the result. That’s what PHP is. That’s why at least 15 minutes of wait is just something PHP developer wouldn’t want to do. We made it faster. How about 5 sec. or less deployment time?

Furthermore, in core we have mechanisms which allows us to support and change PHP configuration and version in the same short time. So you can try different PHP versions in a matter of one mouse click or switch off display_errors when your app is ready to live. Currently you can only choose from two PHP versions and error reporting mode, but there is more to come.

Speed of deployments and configuration freedom is a good building base to start with. But there is more baked in, like an API which allows pushing code directly and a service which will pull from a specified Git repository automatically. Right now we are working on adding MySQL support, so you can port pretty much any existing app. It’s a great core platform which allows adding new functionality very very easily.

Reception

It was an unbelievable journey so far and we learned insane amount of things about Azure itself and how to make PHP deployments blazing fast. Some things required hours to tackle, but in the end we made sure that our users are never going to have to deal with them. And believe me, there are a lot of things you can shoot yourself with when working with Windows.

This is a project which needs feedback and especially from people who know PHP, cloud stack etc. really well. I was running demos and giving access to some people I know and, I think, they were really impressed with the stack. Also because it relies heavily on Microsoft stack, I had spent past two weeks demoing it to a selected group of Microsoft friends and so far reception was amazing. To quote one:

I think you could single highhandedly revolutionize Azure

I think this is a great achievement for PHP community too, because a lot of the functionality we support is not available in some of the leading services so this should kick their asses a bit. We want to stay competitive and keep pushing the PHP ecosystem further, but when it comes to standards, we’ll adopt any upcoming specifications for PHP platforms.

Conclusion

Currently a group of 15 or so people is actively testing this and is sending us valuable feedback. Nevertheless it’s quite close to production-quality service and you’ll hear more about it very soon. If you feel like you’d like to test this (completely free of charge) and would be able to provide some good thoughts, feel welcome to write to me. You can find more details about Azure+ here.

Dependencies management in PHP projects

2011-09-09T13:27:00-07:00

Rarely a project lives by itself, especially in the days of frameworks. Furthermore, there are a lot of great open source libraries you might want to use to save time. But all of this raises a new problem - how could we manage all those dependencies. Here are some thoughts on this problem and how you might want to solve it; without shooting yourself in a foot. Which is commonly known as DLL Hell.

Usually SVN or Git integrated external references management tools are used for this. But… Version control systems are not made for managing dependencies. Period. They can be made to do so, but sooner or later they are going to fail at doing that. This is a fact and there is no way avoiding it, if you don’t trust me on this here are some proofs why.

Version control systems

The most popular one couple years ago was svn:externals for SVN, which is quite similar to git submodule for GIT. The first obvious problem is that they both only support referencing repositories of the same type, that is you can’t include a Git dependency in a SVN project. Which today is a very problematic thing because you might still be using SVN, although not sure why you would be doing so, but a lot of the open source projects have moved on to GitHub.

If you are fine with the above, I think you should be quite quickly annoyed by the fact that those sub-folders you are automatically populating are in fact full checkouts by themselves, thus not read-only. Which potentially is a very risky design characteristic, because most of the time you aren’t supposed to commit from those checkouts, even if you have changed something there.

Git users might be disappointed to know that submodules do not support partial checkouts, that is you can only checkout full repositories. This works fine most of the times, but quite often you’d like to checkout a sub-folder of the repository (for example only library folder from Zend Framework). There is a solution for that called subtree merge, but I find it way too complicated for my liking and I only have used it a handful of times.

How far you want to go

The most obvious use of external dependencies is to get a copy of the framework you are using. This is quite a simple task because it can even be solved by just downloading a copy of the framework and sticking it in the project folder. Easy enough to manage, although not ideal. If you have less than say 5 of such dependencies then any way you choose to manage them is going to be fine. As long as they don’t have dependencies themselves…

Dependencies actually are much more complicated than that. If you are using truly componentized libraries, those by themselves are going to have some dependencies. This introduces the transitive dependencies problem which you can’t easily solve. This is not such a big deal for PHP projects, because the biggest place where such libraries exist is PEAR.net and the tools there will help you with that. Anyhow, keep this in mind.

As you can see depending on what sort of external code you are trying pull in there are different problems attached to it. From my experience simple management of the dependencies is enough, because I’m yet to see a big number of libraries having clearly defined dependencies. So unless this changes soon, I just use the simplest tools available.

Tools made for this

One of the best known tools is Apache Maven, especially if you have Java experience. It does everything you’d want from a dependency manager and probably more, but having used it for couple projects I think it’s overcomplicating what I would need for our projects. Maybe because I haven’t worked on projects complicated enough, but more likely because I just don’t find tools like this attractive and valuable.

You might also want to use PEAR for dependencies management, although it requires external libraries to be stored in PEAR repositories. Similarly there is the composer project which tries to solve a lot of dependencies problems and can resolve them from various different sources, but it still seems to be in development and I haven’t played with it enough. I think composer might be the one to watch.

Symfony2 has an interesting approach of just having a deps file which is used to define where all the dependencies are and where to place them. Think of it as a very light build recipe. Following a similar approach I have extended it and added support for different repository types and sub-repository checkouts. One script ./bin/vendors install to run looks great to me.

Things you shouldn’t forget

With the growing popularity of Symfony2, there are more and more bundles floating around on GitHub. However I just recently had a case where a new developer checked out a project and it wasn’t working completely. Apparently in couple weeks of time one of the bundles was refactored resulting in all previous integrations completely failing.

It was my stupidity of not locking in to a specific revision of the bundle (which you can do using the deps.lock file), but it is likely that you will repeat this mistake too. The fact you need to understand is that most of the time you will be pulling 3rd party dependencies which you have no control of and if you depend on them heavily, which you probably are, you need to know a specific version you want to use. Point it to a stable version if they have one, because it’s extremely bad to just point your reference to a master branch.

Furthermore, a library can go away completely. If it’s hosted in a obscure small website you just found, month down the road there could be no trace of it anymore. Even GitHub doesn’t protect from this - repository can be deleted and will be gone forever. So you need to make a choice - how much do you trust the authors - and preferably backup the source code locally (or setup a mirror repository).

Conclusion

One way or another you will have to solve this problem. I’d say the easiest way to start with and have some room to grow is to integrate this with your build scripts, even as a simple bash script. Don’t try to reinvent the wheel if you need something more sophisticated - there exists working tools already, so just give some of them a go. Just make sure you know what version you want to depend on and have some safety nets against disappearing sources.

Never trust your sources

2011-08-25T13:27:00-07:00

Data validation sounds like an obvious thing and it appears that everyone is doing it, but here are some ideas on where you might be doing it wrong. It’s not a practical examples article though, I’d assume they are pretty easy to figure out; this is more about implications and causes of various different validation errors. All of them are where we had suffered before, so make sure not to repeat the same mistakes.

This post is not about security, although security is probably one of the most important users of validation. Here I’d like to talk about other uses cases of validation, mainly being how to make sense of data you are receiving and make sure it’s not breaking your applications.

Obvious rules

If you expect an integer, check if it’s an integer. If you expect a date, check if it’s date. It doesn’t matter if it is an admin interface and

“only ourselves will be using it, so we will be always entering valid data”

This is not phpMyAdmin you are building (even that is actually validating what you have entered before storing in a database), making sure there is no way to mess up the database from any app will save you time. And grey hair.

More than once have I seen the cases of applications not checking what they are accepting as a price of a product and then failing to render any successive screens because math operations on it are invalid. It’s especially bad when users can’t fix it themselves and need to contact you, the developer, to handle that from the other end. If I enter 1’000’000’000 to stock quantity field make sure the whole app doesn’t explode trying to insert so many rows in a database.

Make assumptions

Programming, I believe, is about logic. So don’t be an idiot and use some of it. Ask questions about the input you are receiving, be it user entered data or auto-imports from external sources, and lock down the expectations. Here are some basic rules, just examples of course, sounding so natural, but still I rarely see them in practice:

Product price larger than 0, smaller than 1000
Person’s age is between 0 and 150
Stock quantity is between 0 and 50
Order cannot have date in the future
Etc.

Obviously it depends on the application you are building and these might change, but most likely they won’t and allowing arbitrary data to be entered can create huge problems. This is better than the Blacklist approach which is not really pratical, as it requires specifying what your data can’t be rather what it can only be.

If you are importing data from an external source, does it make sense for the result to be empty? If it’s a list of events it cannot be empty, unless the whole thing got cancelled. So discard such an import and log that you got 0 events, and you expected at least a dozen… do not delete all events from your database. That is - do not trust the source 100%, use your, or code’s, head too.

Structure

This one is so easy to check it’s not even funny when your data imports go wrong. It can be an Excel spread sheet, some custom format or XML serialized data. All of those have structure, which you should be able to rely on. Personally, if data format changes, I make sure that my code would just stop processing it immediately, because it doesn’t know any more what any of it means.

For tabular data it’s very easy to check tables’ headers - the amount of them and their labels. The order can change, you can figure out how to handle this, but if some fields are missing it is indicating that possibly the actual data can be mixed up to. XML might be trickier to check as it has nested structures, but one could use validation against a DTD. If additional price element is added the code might still work, but the code doesn’t know if it’s using the right one anymore.

There are cases when you might not know all possible data formats, like what I noticed recently when importing some data from Amazon reports. Everything seemed fine when we were testing, but once we launched some products were reporting with wrong quantities. The type field, which we stupidly ignored because it always showed ‘Sellable’, apparently can also be different and when it’s different you should act differently. Obviously because we ignored it the data we imported didn’t make sense - what we should have done is validate the data and have our assumption about type field in place, this could have notified us about that unseen format.

Encoding

We had this issue when working with the Edinburgh Festival API while migrating some import sources to a new location. Everything seemed fine and data was successfully imported, but after some time users pointed out that some of the characters we are returning are invalid Unicode sequences. After some investigation we found out that the conversion of ISO-8859-1 to UTF-8 was in fact incorrect and Windows-1252 was supposed to have been used.

Obviously it’s very unlikely that encoding of the data might change, but, you know, sometimes things happen when you least expect them. Annoyingly encoding issues are easiest to spot by humans, because any single-byte encoding can be applied and it will work, kind of, but only an actual person would notice that the text it’s showing doesn’t make sense. Luckily computers now can guess too, by for example running this:

<?php
$str = 'áéóú'; // ISO-8859-1
mb_detect_encoding($str);

However results of this function are pretty unpredictable sometimes, although you can still use it to detect if data is a valid Unicode string, for example:

<?php
$str = 'áéóú'; // ISO-8859-1
mb_detect_encoding($str, 'UTF-8'); // 'UTF-8'
mb_detect_encoding($str, 'UTF-8', true); // false

And if you are converting from a specific encoding to another one, say Unicode, test if result doesn’t contain invisible characters or any other impossible sequences in human readable text.

Conclusion

I can’t stress enough how import data validation really is. There are so many attack vectors exploiting incomplete or faulty validation you can never be 100% sure all cases are covered. But rather than building a blacklist, go with whitelist approach, because most likely it’s going to be better and if conditions change you can always fix it later.

Web Scraping is actually pretty easy

2011-07-27T16:15:00-07:00

For some of our clients we worked on extracting or submitting data automatically from websites which didn’t have an API we could use. This and more is called web scraping. Since our announcement of SellerScout, which relies heavily on this, I received a list of questions how we actually do this. So here are some thoughts on how to get started in the interesting web scraping world.

This article talks about some basics, which will work fine for most cases. This is probably not even remotely close to how Google does this or how we do it at SellerScout. The reason being both of those systems work in much larger scale and use cases are different. For example relying on machine learning, text analysis and semantic search algorithms etc. are all the things you might be doing if you want to build something big.

Downloading the web

It's all just scraping

Spiders are the small applications you are going to be writing. Usually they are self-contained and CLI-friendly scripts, which have some internal logic how to extract information from a specific website or websites. As an example, the script might go to website’s homepage, download all the category pages, download products list for each of them and extract a list of products in the store.

If you are a Python guy, you might want to look at Twisted or Scrapy, later being very easy to use. If it’s PHP you are using, combination of cURL and libxml will allow doing the same; I’m not aware of any PHP frameworks for this. For any other language, you should give a look at Google.

Depending on your task you will need to support different functionality. If the website is for logged in users only, you should configure cURL to use cookies jar and initialize the scraping with a request to login page. If you need to extract thousands of documents, have some logic to pause and resume the script, so if it crashes it can start from the last completed document rather than from the start. In any case, try to replicate the natural user behaviour on the site.

Is it legal? Depends. There is no strict answer and it varies on what data you are trying to extract. Some data can be copyrighted, for example original texts, so if you are scraping them and showing in your website - you are being a bad person. Stop! Ideally you should discuss this with your lawyer, which we did, and get some thoughts on how to proceed.

Getting blocked

One of the decisions you will need to make is how you are going to identify the spider - you can either replicate normal browser’s headers or introduce the spider by its name (eg. ’googlebot’). First one will allow you to stay undetected, probably, while later one is considered to be the correct way. From my experience, for anything small Firefox headers will work just fine.

Websites might still decide to block you though, and it’s something you might want to be prepared for. If you are identifying the spider by a name, you should respect robots.txt and stop crawling if you are being denied by that file. However the most likely blocking mechanism is to block your IP address, which is going to happen if you are being stupid. Really stupid.

You see, when people are browsing the web they request 1 page each 3 or 4 seconds, hence if you have a list of 1000 urls to download and you just start iterating over them and issuing requests… Well, you are easy to catch by just looking at the access logs. Don’t do this. Rather have a queue of urls to download and issue requests with a random delay from a range of 1 to 5 seconds. It’s going to take longer, but it will help to avoid problems.

This doesn’t scale though, you might say. And in fact you are right, because 5 seconds delay between each request limits the amount of content you can download per day. Luckily for you, I have a tip here too - use proxy servers. It’s going to require writing a requests scheduler, but if you need to download the same 1000 urls you might as well distribute them over a list of proxies each with their own delay times. The more proxies you have the amount of content you can download increases linearly.

Extracting the data

Once you have the HTML you want to process (to extract links to follow or to extract actual data), you might wonder how to actually do it. There couple of ways and libraries for this, however if you want to keep it simple using XPath or CSS-like queries is going work just fine. If you feel like it, and believe me sometimes there is no other way, you might go with using regular expressions for this, but that’s got problems I’m going to talk about just in a second.

I tend to go with XPath because it’s very easy to write and to debug. Furthermore there are various extensions you get for your browser which will allow creating those queries and test them on the actual website. I have worked on spiders for over a 100 different websites and XPath worked fine all the time, as long as…

The problem you will need to solve is how to process invalid HTML or XHTML markup. And from my experience, I’m yet to see a website with all pages being 100% valid. The more invalid it is, the harder it’s to fix those problems. There are libraries though, most famously BeautifulSoap, which will try to process invalid markup. They do have performance implications, but keep them in mind because you won’t be able to issue XPath queries on invalid syntax.

Now let’s get back to regular expressions. Theoretically they might look awesome, because they can extract data even if the HTML markup is invalid, however the problem is that soon they get complicated and very easy to break. XPath allows you to work on a DOM tree, hence if the website structure change they just stop working completely. Regexps on the other hand might still work, but produce very unpredictable results.

Conclusion

We, as a company, have a lot of experience web scraping the data and it’s actually very very easy. As long as you follow the logical rules and don’t try to over-complicate the data extraction, you could easily extract all news items, products or blog posts in 30 or so code lines of spider. I can talk on this for hours or days, so I might write more on this soon, because this is just a top of an iceberg.

Building the Edinburgh Festival API

2011-07-04T17:10:00-07:00

Couple months ago we started working on a very exciting project - building data access API for world’s largest cultural event, the Edinburgh Festival. It was a very exciting journey and here I’m sharing how we built it and what’s the stack used. The goal of this is to show how we solved specific problems and how you might apply it for your applications.

The problem

We started from having a specification from Festivals Lab of what this API is supposed to be doing and that was a really trivial task - outputting data of 7 different sub-festivals in one format. And this is just about right what this project was about. We took the data from various sources, processed into formats we could understand, did some filtering, cleaning up and validation, and pushed that data through the API.

Of course this is just a read-only API, so that changed the design a lot. There was no need to connect to an actual database from the API server, but rather to a search server, which would be faster and more reliable for data lookups. And reliability was one of the requirements, because as with all APIs they should always work, in theory. Having a database and search server decoupled allowed increasing fault-tolerance and scalability.

What we tried to achieve is that even if all the servers would die, we can spin up a new one, run some scripts and API is back live. Thus data imports were supposed to be deterministic and fast. Once we got that, we were certain that even if things go horribly wrong we can recover very quickly and for smaller cases database’s and search server’s redundancy is protecting us.

The stack

Processing

Interestingly this is the part where we spent most of the time, because chasing for data has proven to be very challenging. However once we got all data in sensible formats, we wrote a collection of self-contained scripts processing the data into various our formats.

Key lesson here was that Excel is used very commonly and there are problems with most of the Excel file parsers out there. Especially with Unicode characters, which worked fine in most cases, but sometimes would just fail to unrecognizable chars. Exporting to CSV first and then reading that from the scripts was how we solved this.

Database server

Obviously data coming from data sources needs to be stored somewhere. Overall the structure was very trivial, only consisting of events containing a list of performances. So MySQL could have worked here easily, however the problem was that data structure was different between different festivals and also it was constantly evolving during development process.

Hence we went with CouchDB because of its reliability and the fact that we can just store events as nested objects.

Search server

One of the key elements of this API is to be able to filter data efficiently. ElasticSearch was chosen because it integrates with CouchDB almost out of the box, so it was as hard as executing one query and we had a (almost) real-time representation of data stored in a database, searchable through the API.

ElasticSearch also allows to kick-start in 5 minutes, without a need to define document’s structure allowing to add those later to optimize the performance. It supports anything Lucene supports too, so it wasn’t like we were throwing out possible features.

API

As I explained in the previous post, we use Python for all APIs. This wasn’t an exception. However, the actual API is just couple hundred lines of code - very thin layer on top of the search server, which is using HTTP interface anyway so there was no need to use any libraries too.

Nginx was used as a proxy for API servers, each monitored by supervisor. Deployments are obviously made using Ant and any task is one-click operation here, so any updates can be rolled out in a matter of minutes.

The outcome

Well, it works. In our internal tests we tried to make it go down, but it sustained them all quite well. We can’t share performance characteristics of the application, but it’s pretty damn fast and the servers have no problems with the load.

There is logging for all requests coming in, storing information about which API user accessed what sort of information. This should be producing some interesting results we might be able to share later. The API is going to be used in some mobile apps too and we haven’t worked with those before, so we will learn how use cases are different from mobile applications compared to websites or desktop applications.

Festival only runs for less than two months, so this project is quite short-lived, but there are big plans for next year to open the data even more. If it was us to decide, we would make it fully public and do not require any license agreements, but this is not the case, so we are going to push hard for this for next year.

Conclusion

Clients seem to be very happy about the outcome and we are happy because we had a chance to work on a very challenging and important impact-wise project. I made all those decisions, of course after discussions with the clients, and they seem to work so far. Any tips for future projects?

RESTful web services with Python. The easy way

2011-06-15T17:40:14-07:00

More and more projects are exposing their functionality via REST APIs. We think APIs are awesome and it’s great what they’ve done for the web overall, but we also see a lot of bad APIs examples, like Twitter API. It might be the case that if you don’t have the right tools, it becomes hard to implement them correctly and quick. Lately we have been working on a couple APIs and I decided to share our experiences and why we went with Python in the end.

What APIs should do?

As little as possible ideally. In most cases it’s just a layer on top of a database or search server, providing a RESTful way to access data and get it back in some fashion understandable by a client. The less code there is, the lighter it’s and the easier it’s to make changes the better. And with a rise of popularity to expose data and functionality using APIs it should behave following REST and HTTP standards, so it can be adopted in no time.

Depending on how you want to do it, you can also go full RESTful or just pretend that what you are doing is a REST API. Supporting Hypermedia for example is something you are supposed to do, in theory. But at least the API should handle HTTP Accept headers correctly, use native HTTP authentication like Basic Auth and have meaningful resources’ URLs. Just that will make it somewhat much better than most APIs out there.

Importantly, web services should be as fast as possible and support huge amounts of requests per second. In most cases APIs are called from other applications and the more time it takes for your API to respond the slower that application becomes. You should aim at no more than 10ms to fulfill the request. And if application developer decides to retrieve some resources in a loop, your API server shouldn’t crash either.

Python or not

Although we might seem as a PHP company, we are not really - we use PHP for websites, where it works best, and nothing else. The reason why we tend to go with Python is simply because it’s just perfect for what I described above. There are libraries for pretty much anything when it comes to reading and writing data from any storage and it’s super lightweight compared to a lot of other languages.

Don’t get me wrong - there is nothing wrong with any other languages; it’s just that Python worked really great for us. However if you for example want to stay with PHP, Frapi might be a good option for APIs. Although you can’t really achieve a lot of things as easily as with Python and the language is just much more concise. Performance is a questionable topic, but from our experience Python wins any day. “It scales” that is.

From functionality perspective decorators allow achieving a lot of things without destroying application flow with endless listeners and callbacks. When I need to provide authentication for the API, I just wrap the application with @auth or when data from some API call needs to be cached it just gets wrapped with @cache. Makes workflow really clear and doesn’t require nested if structures and duplicated logic. It’s used heavily in most of the Python web frameworks.

API in a Bottle

Bottle is one-file web framework based on WSGI, thus it works just as any other Python framework. It’s not really made for APIs exactly, but it works great for them. API looks a lot like Sinatra - it just maps routes to actions (functions). What is more, I find it to be allowing very rapid developing - in most cases I can write whole API in less than a day.

Compared to other Python frameworks it doesn’t do anything that special, but where it shines is that a lot of the things can be either configured or swapped for different ones. It’s just a box of building blocks with some default behaviour, but from there you can really make it work in any way you want. If you need real-time web services allowing you to push data to clients, Tornado might be a better choice though.

Here is an example of the simple API, with first method returning plain string and second returning Python dictionary which will be automatically converted to a JSON string:

import bottle
from bottle import route, run

@route('/', method='GET')
def homepage():
    return 'Hello world!'
    
@route('/events/:id', method='GET')
def get_event(id):
    return dict(name = 'Event ' + str(id))
   
bottle.debug(True) 
run()

A lot of functionality can be tweaked using plugins, so rather than allowing Bottle to automatically convert data structures to JSON, you might want to use plugin like this to return data strictly by the type client accepts:

class FormatPlugin(object):
    name = 'format'

    def apply(self, callback, context):
        def wrapper(*a, **ka):
            # Check if return data format is supported
            accept = request.environ.get('HTTP_ACCEPT')
            if not accept in ['application/json', 'application/atom+xml']:
                return HTTPError(500, "Unsupported data format")
            
            # Execute the action    
            rv = callback(*a, **ka)
            
            # Write out results
            response.content_type = accept
            if accept == 'application/json':
                return json.dumps(rv)
            elif accept == 'application/atom+xml':
                return render_xml(rv)
            return rv
        return wrapper

When it comes to performance we haven’t maxed it out yet. Usually we tunnel multiple Bottle applications through Nginx pool using uWSGI as a web server, which we chose because of the benchmarks done by Nicholas Piël where you’ll see that uWSGI demonstrates amazing performance. For this article I did some benchmarks on a VirtualBox VM with one core and I was getting at least 2000 req/s from an API talking to Mysql and MongoDB servers to fetch different data.

Conclusion

If you haven’t tried Bottle before I suggest you do - installation is as easy as easy_install bottle and you can be up and running in just a few minutes. It worked well for us as it allowed creating web services quickly and customizing certain behaviours, but even with defaults it was suitable for most of the use cases. If you need help getting up and running, you can always get in touch with us.

Symfony2 - the best framework today?

2011-06-07T15:55:14-07:00

I used to use Zend Framework extensively and still believe it’s the best framework for anything what doesn’t support PHP 5.3. However a couple months ago I started using Symfony2 for internal tools at Web Species and have stayed there since. It has its problems and flaws, but let me give you some thoughts why I think it’s the framework which is going to go big. Very big.

Frameworks are big creatures and naming interesting features can take thousands of words, so this is just a short glimpse of the few things I find interesting, to me. Obviously there is much more to it. Hopefully you won’t find yourself feeling like this guy:

I have nothing against Symfony2, I’ve been using it and it’s great. But this blog post is nothing but a gushing verbal sex exposition with Symfony as the subject.

What I like about it

First of all, once you grasp how it works, it starts to produce great results. I was really sceptical at first, because it seemed very complicated and over-engineered, however after few days of work I started liking how it works. And that’s the thing about Symfony2 - after some time it starts to feel great to work with it, it even makes working with PHP interesting again. Once you know how to configure it properly it starts to play along.

Not a big secret, but I’m quite a Doctrine user and the fact that Symfony2 integrates with Doctrine2 out-of-the-box just makes things easier. For example working with forms is way simpler now because it can create forms out of entities’ definitions and also populate data to entities. And integration for Doctrine CLI is obviously there, so the only thing you need to do is to specify connection properties and everything just works, in theory.

But that’s all code which is replicable to other frameworks; the most impressive bit is bundles. Bundles are small self-contained plugins allowing sharing some functionality between different projects. And GitHub is just flourishing with all sorts of different ones, showing how much people are interested in this framework and are actively using it. Using bundles makes Symfony2 core smaller, but also gives more flexibility with how you bootstrap a new project.

Business reasons

As much as I’m still involved in writing code, I need to make business decisions at the end about tools, frameworks and languages we use. And the tools I tend to go with are the tools I believe in can stay around for years to come and are, obviously, high quality and popular. Without any doubt Symfony2 is the fastest growing framework and quite quickly should become the most popular out of all.

There are a lot of not as well-known frameworks out there which are quite good and work somewhat well, but the reason I’m not using them is because they are not popular enough, which makes me question if they are actually that good. It’s hard to find developers with experience, it’s hard to find blog posts and discussions online and I’m unsure how long they are going to stay with their limited contributors list. Symfony2 makes me feel safe so far.

It’s hard to answer why, but for me Symfony2 looks the most professional and/or professionally-developed framework. Party because I know a lot of companies and people working on it personally and I know they can deliver, but also because code quality they produce is amazing. And with the amount of developers constantly contributing I can see it being actively developed and becoming even more solid.

The bad parts

A lot of the things in this framework still have to be proven to be the right decisions. I know a lot of people, who love Symfony2, but at the same time there is a big group who just hate it, and they have different reasons. Maybe because it has a very steep learning curve, at least now, but also because of some of the design ideas.

I’ve mentioned this briefly in my blog post about frameworks, which is the fact that I see a lot of Java patterns and ideas being used in PHP. This is a big topic and I might blog about it sometime, although I have nothing against Java (he he), but even things like Dependency injection containers just don’t feel right (when used in PHP).

DiC does work fine when you can load objects’ graph into memory and provide dependencies when needed, but nature of PHP is very different from Java and that graph needs to be built on each and every request. Of course the performance problem or impact, to be exact, can be solved in some ways, but the difference is still there. So as much as I like this idea, I’m not yet fully confident that it will work well.

This brings us to another problem with Symfony2 - the amount of configuration involved. When I was starting with Symfony2 it took me quite some time to figure out how to achieve even simple things. Maybe the framework went too far with removal of magic, right now it feels quite demanding and asking for everything to be configured explicitly. And once an exception is thrown somewhere deep in core, good luck figuring out why is it happening.

Conclusion

I think we made a right decision choosing this as a base for our web apps and are starting to receive more and more inquires about Symfony training courses and consulting we do, which just shows that popularity is growing and growing. It does have some issues and sometimes just feels clunky, but overall allows producing high quality projects and being sure about the results. I feel confident investing in Symfony2.

Lazy evaluation with PHP

2011-05-31T15:07:14-07:00

Recently I needed to process a huge array of data and because of PHP’s somewhat inefficient variables and especially arrays that was resulting in “out of memory” errors. However, I couldn’t use any other tools than PHP so was forced to come up with a solution implementation in it. Here is how I solved it using principles from functional languages.

In programming language theory, lazy evaluation or call-by-need is an evaluation strategy which delays the evaluation of an expression until its value is actually required (non-strict evaluation) and also avoid repeated evaluations (sharing). The sharing can reduce the running time of certain functions by an exponential factor over other non-strict evaluation strategies, such as call-by-name.

What you are about to read is not necessary about gaining extra performance, most of the times it’s going to stay the same, goal of this is to minimize memory usage as much as possible. So if you have a million of rows to process each weighting 10kb, the memory usage always stays at around 10kb. Theoretically, PHP is not that perfect at cleaning up memory.

The problem

Let’s image that you need to process those million database records, how are you going to do it? Well, obviously the first step is to fetch them from a database. Stop! I can’t even remember how many times I’ve seen people making mistakes here… Why? Eager evaluation, lookup this term. Wait, I’m going to make sure you know what I’m talking about so read it here in Wikipedia.

All comes down to using fetchAll() method - the script was fetching all orders in a specified time range from a database. What fetchAll() does is returns an array with all results matching the query, but this requires quite some memory if amount of results is in thousands. Later the script was doing some calculations and creating a second results array now with processed data. At the end of the main loop memory usage was row count * size per row * 2, a very big number.

It’s very rarely beneficial to do it like this - most of the time data can be processed per row, removing the need to store everything in memory. Sometimes SQL queries won’t allow it, but even those can be changed to make it possible to abandon pre-computation and work with data as streams. Once you have a data stream, you can start working with it using pipelines.

Functional languages

Before looking at PHP solutions for this problem, let’s analyse how it can be done in a functional language. Image a function returning all Fibonacci numbers, here is an implementation in Haskell (from Haskell documentation):

magic :: Int -> Int -> [Int]
magic 0 _ = []
magic m n = m: (magic n (m+n))

By calling it with magic 1 1 it will return a list [1,1,2,3,5,8,…]. But the important part is that this list or the return value is infinite. There is no boundary like “return 100 numbers”, it will actually return all possible numbers. You might say that’s impossible and you are kind-of right, especially if you haven’t worked with similar functional languages before.

Because Haskell uses lazy evaluation (with strict being an option), calling this method doesn’t actually compute anything. Instead it creates a generator-like resource from which you can read as many numbers as you want, and as long as you are reading them it’s computing next one. So with a function like:

getIt :: [Int] -> Int -> Int
getIt [] _ = 0
getIt (x:xs) 1 = x
getIt (x:xs) n = getIt xs (n-1)

We can get Xth number of Fibonacci - call it like getIt (magic 1 1) 5 and output should be 5, because the 5th number of the sequence is 5. Important part here is that even though I’m passing a result of function magic to the getIt function, as mentioned above, magic doesn’t need to compute anything to return. getIt reads 5 numbers from that infinite list and terminates returning the last number.

PHP way

Sadly, you can’t really do anything like that easily in PHP, because it doesn’t support lazy evaluation or generators. However it’s possible to improvise and have a working solution. And the solution is… Iterators. One of the most underused functionality in PHP.

Any class which extends an Iterator can be used in foreach as a data source. Let me give you a short example of an iterator generating all numbers from 0 to infinity:

<?php
class Numbers implements Iterator
{
    private $position = 0;
    
    function rewind() {
        $this->position = 0;
    }
    function current() {
        return $this->position;
    }
    function key() {
        return $this->position;
    }
    function next() {
        ++$this->position;
    }
    function valid() {
        return true;
    }
}

$n = new Numbers;

foreach($n as $value) {
    print $value . PHP_EOL;
}

Running this script will yield a never-ending list of numbers. But you can only do 10 iterations and get 10th number all without ever needing to store whole sequence in memory. Of course this is going to be slower than just calculating 10, but using similar approach you can process data one atomic unit per cycle without ever storing it in memory.

If you need to render data in a view again remember the memory - do not compute the result in a model (or god forbid controller) and then pass it to a view. Creating an iterator in model and return it for consumption in a view. For view it is the same thing - array and iterator are both iteratable, but you are saving a lot of otherwise wasted memory.

Database side

Aggregation should always happen in database side, when possible. So if you need a total of all items sold for each order - calculate that using a SUM() function rather than doing it when iterating over results. Because to do it you need to look back or forward in results set, and that breaks lazy evaluation immediately.

First of all, fetch data with a normal fetch() method in a while loop, rather that iterating over fetchAll() result. Nothing will break, hopefully, but instead of building array of all results and then processing them, the script will process each row and release it from memory. MySQL returns a cursor to results’ set and PHP driver will use that to get each row after row.

You might wonder how to render pages like a list of orders with items. It’s quite easy, just join orders’ and items’ tables and order results by order id. The data you will be getting back will look something like this

Order ID   Item ID   Item name
1          156       Milk
1          897       Bread
2          156       Milk

To render this just iterate over results and as soon as order id changes from previous one, start a new HTML table for an order. Using this approach you can render whole history of all orders with information about items too without ever using more memory that one row size. The only limiting factor is then response HTML size.

Conclusion

Obviously PHP wasn’t really created for anything like this, but situations when you need to make it work happen. Make sure you are not building up any data or fetching it eagerly and release resources as soon as you are done. This solved all the problems for me and all of the sudden same script was processing all the data without any memory leaks.

HTML5 History API - dynamic websites like never before

2011-05-26T13:47:13-07:00

I have talked about this before, but JavaScript should not dictate content or website structure. It should only improve the UI, but even with JavaScript disabled website should work. Using the new HTML5 History API allows to do that one step further - making dynamic websites behave like normal ones.

What is History API?

History API is quite a simple concept - a JavaScript API you can use to control history state. If user clicks on an image and you show a lightbox with enlarged version, clicking Back sends user back to previous page, rather than closing lightbox popup. It does this, because there is no state information browser can use to know how to close that popup window.

With using History API you can add an entry to history stack once some dynamic content is loaded. When user clicks Back browser would go back by one element in history stack and fire off an event which then you can handle by closing the popup window. Same thing happens when user clicks Forward - event happens and it’s up to your script to handle this gracefully.

All this behavior is very well explained in “Dive into HTML5” book, but in short when you load some content using Ajax or you move user to a place in a page which you want to have linkable - use History API. Of course this involves handling the links in server side too, but this is quite trivial - build the website first then make use of History API to improve the interface. In my tests this improved user experienced a lot and allowed to achieve very rich user interfaces without destroying website structure.

Of course like with most of the new HTML5 functionality, this is not supported in all browsers. Most importantly this is not supported at all in any IE versions, not even IE9 which was released only couple months ago. But you can handle this by having improved UI for some users and falling back to normal non-js behavior for IE, for example this is what Github does (click on folders).

How to use it?

There is only one main method:

history.pushState(state, title, link);

and one event:

window.addEventListener("popstate", function(e) {
    alert("location: " + document.location + ", state: " + JSON.stringify(event.state));
}

State example from Mozzila documentation:

history.pushState({page: 1}, "title 1", "?page=1");
history.pushState({page: 2}, "title 2", "?page=2");
history.replaceState({page: 3}, "title 3", "?page=3");
history.back(); // alerts "location: http://example.com/example.html?page=1, state: {"page":1}"
history.back(); // alerts "location: http://example.com/example.html, state: null
history.go(2);  // alerts "location: http://example.com/example.html?page=3, state: {"page":3}

However I would recommend using some abstraction for this, mainly because you need to do quite some manual work, which I hate doing. A great tool exists called History.js which does exactly that - abstracts History API and also has fall-back support for lame browsers like IE. I’ve used it extensively and it works great and it even has adapters for jQuery et al so you can use the same interface.

Most impressive part of it is optional fall-back for older browsers. What it does is uses a similar to hashbang url, like http://webspecies.co.uk/#/about which it handles inside and all methods for getting current url still return /about. And if you go to this url from a modern browser it detects the support for proper History API and redirects to correct url. All combined, makes everything work nicely and future-proof, here is an example of full script for Ajax page.

How I used it

When building our Web Species website designer had an idea to have all content in one scrollable page. If you go to the homepage and start scrolling you’d notice that immediately - menu stays in place but content slides scroll as usual. Now even though this is very nice from user perspective, it creates a problem from content side. Very big problem.

Main problem is that there is no way to link people to specific slide and that it’s hard to get good rankings on Google. Now even though the first problem can be easily fixed by using anchor urls (like http://webspecies.co.uk/#about), but fixing SEO is, I believe, impossible. Impossible because one page contains a lot of content and you won’t get good rankings for any keywords.

If you look at view source of http://webspecies.co.uk/about you’d see correct title and only “about” content, but as soon as you load this in browser AJAX loads all slides and replaces existing slide with them. So from user perspective there still exists one-page effect, but actual pages can be called directly (in server side I have all separate pages like about, training, clients etc. and one with all slides).

To fix linking I employed History API and it worked out beautifully. If you click a menu item on left side of the website, AJAX event fires off resulting in two actions - pushState() and scroll to correct slide. Same thing happens on Back or forward history actions - content is scrolled to where it belongs. If you go to news section clicking on news items result in same behaviour too.

Conclusion

As with all HTML5 functionally History API is still quite rarely used, but if you feel like building something awesome - I’d say give it a go. It works beautifully in modern browsers and there are tools which can make it happen for older browsers too. From my point of view, it allows to achieve crazy UI ideas and still have semantically correct websites, which is always my goal.

The new era of PHP frameworks

2011-05-23T17:51:13-07:00

I have worked on a lot of different systems and projects in my years and most of that was spent doing PHP. However just recently I have noticed a new major point in time - a new era of PHP frameworks. Seems like everything is changing these days. I want to discuss what I think the current state is, what’s wrong with it and how the new gang of frameworks is going to change it.

On May 21st I have delivered a talk at Dutch PHP conference (DPC in short) about this topic and I had very interesting discussions to follow it. To start with here are the slides of the talk, of course keep in mind that they do not work that well without me talking. This article is a brief version of what I talked about.

Frameworks are born

I use PHP frameworks

6 years ago CakePHP, one of the first PHP frameworks, was released and from then we’ve seen a plethora of PHP frameworks. Currently there is about… a million of them, all with their different MVC, DBAL and templating implementations. I like them, even if they have their weirdness’s, but still their adoption is not massive.

If you’d look at numbers of PHP open-source projects based on frameworks, you’d see that there is only a few. Which is sad. Partly the reason is that a lot of those projects were released before any PHP frameworks even existed, but also because doing some work with a PHP framework required quite some learning. Thus if a project is to be based on a framework it would increase the learning-curve, at least in most cases.

Nonetheless they have changed how we do PHP. A lot of developers claim they know OOP, but when frameworks came they were forced to actually prove it (before that you can hack in any way you want). And frameworks have thought millions of PHP developers what is real OOP and how it works. Ask someone to use mysql_query nowadays and you might get punched in a face. Twice. Because they would also need to use mysql_real_escape_string.

How it was done?

No one really knew what PHP frameworks should be. Nor what features they might have. So how did people managed to make them happen? Well, they either followed existing frameworks in other languages (like RoR) or came up with their own ideas. Because experience was not there, most of the frameworks up to today have legacy designs which everyone knows are bad, but are impossible to fix.

Pragmatic approach of PHP developers here helped a lot - similarly as how PHP as a language evolved, PHP frameworks also changed and grew driven by feedback and requests. In a couple years most people were happy with what we had. But if you’d look closely in 2007 we had Zend Framework 1.0, which had, compared to 1.11, a very limited feature set. So even today frameworks are evolving rapidly to meet features’ needs.

PHP 4 was (and surprisingly still is by some of them) supported by all the frameworks. This led to a lot of code which is now very out-dated, especially the OOP paradigms. Trying to support this has led to complicated process of implementing new features and fixing bugs. Furthermore, less and less developers want to work on such old code anymore.

What’s broken?

Magic kills

First of all, back then it was very popular to use PHP’s magic functions (__get, __call etc.). There is nothing wrong with them from a first look, but they are actually really dangerous. They make APIs unclear, auto-completion impossible and most importantly they are slow. The use case for them was to hack PHP to do things which it didn’t want to. And it worked. But made bad things happen.

SCOP - Static class oriented programming, is a term I invented to describe most of the PHP code. Static methods are bad for a lot of reasons I’m not going to dive into today, but most importantly if a class is a collection of static methods, it’s nowhere near OOP. It’s just using a class as a container for functions. There are even full frameworks just doing this.

Zend Framework for a long time was my favourite PHP framework (and still is for PHP 5.2), but my main issue with it is that it’s trying too hard to be a component library. Other frameworks followed the same path - rather than using existing libraries, they wrote their own. This has created so many libraries inside PHP which are kind-of standalone, but still require downloading whole framework. Fat frameworks are really annoying.

The new era in 2011

To improve this situation people have chosen to do a couple of things. But mainly is to rewrite frameworks from scratch on top of PHP 5.3. This allows to establish new standards, agree on interfaces between all frameworks and throw away all (most) legacy problems. Sounds easy, but only by doing these things we can enter the new era of frameworks.

I haven’t used any frameworks before CakePHP was born, so I’m going to use it as a landmark. Actually I even doubt there existed anything before CakePHP, of course if you don’t call Drupal a framework. From CakePHP years till today 6 years have passed which I mark as first era. 2011 marks second and completely new things will finally happen, mainly in a form of releases and announcements.

Interestingly, in 2011 PHP is no longer PHP. Or no longer just PHP. LAMP stack is boring and less and less used with new tools like Nginx and CouchDB available. Today integration and interoperability are crucial elements. Same is for language - PHP 5.3 is a completely new beast which makes possible for amazing functionality, and if you do PHP 5.3 there is no real backwards compatibility support to do.

Let’s fix it then, shall we?

Use git

Moving to GIT allowed a lot of frameworks to make it easier to contribute. I’m mostly impressed with Symfony results, because they have managed to attract huge number of contributors (see the graph here). Current pace is much faster and compared to few years ago frameworks are improving at much higher rates.

A lot of trimming has been done. First of all, all that magic I was mentioning earlier is now gone and explicit definitions are used all over. Furthermore, there is more thinking towards having a small core and additional features coming through extensions and libraries. This is a great way to make it easier to work with frameworks and reduce memory footprint.

Performance was a major issue for PHP frameworks and most of them had it in their plans for new releases. Front-end received a lot of attention too with frameworks like Symfony helping with assets (JavaScript and CSS) management and proper headers for static content. PHP side profited from removed magic and code clean-up, and PHP 5.3 added a huge performance boost.

New features

Obviously all the new language features are incorporated. Like namespaces for example. Namespaces support lead to needing to research and agree on autoloading approach which would work for most of the frameworks. PSR-0 was born earlier, but is now well integrated into frameworks. And then anonymous functions are finding their way in.

Dependency injection containers and Annotations are two I’d like to mention here; both change how you code. I like using them and Symfony makes great use of them, but other frameworks are catching up and should start to incorporate them soon. Combination of those and new PHP features allows to create very clean micro-frameworks, look at Silex.

I’m not entirely sure that I like growing list of features directly ported from Java environment. Java works differently (and requires 1GB of RAM) so even DiC is tricky in PHP. We’ll see where it gets us, but so far I’m a bit worried because I know that PHP likes light systems rather than complicate objects’ graphs. As much as cool those patterns sound, they can create more problems then solve.

So when?

Symfony2 release

Zend Framework 2.0 is on its way, but will take some time. Because ZF has a massive code base first thing they did was to convert it all to namespaced code. Once it was, it was time to start refactoring functionality and implement new one. Currently the work is being done on MVC part, but I’d hope late this year a final release might happen.

Lithium is about to be there, it’s in dev mode, but seems pretty close to finished. It’s a completely different framework from those regular ones, so it would be worth to check it out. I’m mostly impressed with their AOP implementation and would like to see adopted in more frameworks. Obviously they are PHP 5.3 only, but they do support CouchDB and MongoDB quite nicely.

Symfony2, in my opinion, is leading the pack. They are currently in beta2 and final release is a matter of months it seems. List of features is hard to grasp, so it’s worthwhile checking them out in their website, but to name one - Bundles. Bundles are a way to have extensible application structure with a collection of components from outside. Think plugins.

Conclusion

I’m super excited about the things currently happening in PHP industry and I believe they will lead to great achievements. Finally we have a time when we can throw away all (most) our legacy and implement fresh ideas. Fast forward 5 years from now we should be as excited as we are today.

Virtual Machines for Web Developers

2011-05-16T13:30:13-07:00

There are millions of articles on how to setup LAMP setup on your own machine to allow developing websites locally. I think this is a wrong approach as running server programs in one’s computer creates a lot of potential problems. Better approach for this would be to use Virtual Machines as they allow bigger flexibility and fewer headaches when something goes wrong.

Of course you probably don’t want to do it if you are only working on one project or you are just starting your carrier as especially at first it will feel kind of weird. However for people like me who work on a dozen different projects throughout the week it’s just so much better. Hopefully after reading this you will rethink your setup.

Performance

Having a VM for web server first of all makes your computer boot much faster. I was getting really frustrated at some point because I had so many different server components installed in my main laptop that it was just unusable. Web server with a dozen of websites, database servers, monitoring tools etc. all of this was just too much. Once all of this is moved to a VM this problem is solved immediately.

Processors and other hardware nowadays have really good support for virtualization (you can read more about it in Wikipedia) so there is no real performance drawback. Just make sure you have enough RAM to run multiple OSes and as much as possible cores in your processor for the load to be spread out. Not surprisingly only install systems without graphical UI, for example install Ubuntu Server without ubuntu-desktop package, this will save a lot of resources.

Advantages

I especially like that, it makes it easier to reinstall machines and migrate them. Not only you can move VMs between different machines (in most cases this works) so you can have same setup for testing on a laptop and on a main computer, but in case you need to reinstall anything you are safer not to screw up the whole system. Tools like VirtualBox also support snapshots which allow reverting back to some known state if things go wrong.

Obviously decoupling of testing environments from an actual machine also makes it possible to have different environments to test different things. For example I have one VM with Windows server and one with Ubuntu, both of them acting as web servers. This allows testing same applications with different configurations very easily. Furthermore, VMs can also be used for browsers’ support testing.

What is more you can test load balancing, database replication and other things which require multiple servers really easy. And with a help of VM software and tools available in OS one can also simulate network latency, dropped packages and bandwidth. Some of those things you can’t even test on real hardware (like network performance) and VMs just work great for that, it’s only a matter of using right tools.

VirtualBox

For VMs I use VirtualBox because it’s free, works really well and supports all the features I might need. My setup is quite trivial but has worked really well so far - all VMs are in a Host-Only network with static IPs. So I can access them from a computer or from any other VM by IP like 192.168.56.104, but they are not visible from other computers in a network while still having internet connection from a second network connection in NAT mode.

Sharing files with VirtualBox is really easy as it supports shared folders. Shared folders are exactly what they stand for - any folder in computer becomes a visible partition (Windows) or device (Linux) which then you can map to wherever you want. To support this you need to install guest additions in each VM’s as you need a special kernel module for shared folders. I haven’t yet figured out how to install it without full additions, so let me know if you have any ideas.

Better setup

However I don’t use shared folders that often as I have build scripts for deployment. I believe this is just a better as I don’t want to map my projects’ folders to be accessed by web server as they might create some cache files or modify them in any other way which would be really bad. So I just execute ant testing and it deploys application to an assigned VM.

To make this work well I have my public key deployed to all of the VMs so I can connect to them without using a password. Also I have mapped the IP addresses to specific hosts like doctrine.dev, which allow to connect to a VM as easily as ssh doctrine.dev or to access a website hosted in that server http://doctrine.dev/.

Once you have done this, you completely forget the fact that it’s VM you are using. Because I use Ubuntu I have all VMs open in a separate workplace which I don’t use for anything else. And I “talk” to them like with any other server - using SSH. Yet another decoupling point - all the build scripts you write can be used for staging/production server later as you would be accessing them in a same way.

Conclusion

I can’t be happier for this switch as it finally allowed using my computer not only for development, because if VMs are not started there is nothing waiting in system processes. The amount of flexibility and features is just great and much bigger than initial hassle to get the setup working.

Jekyll - back to the basics of Web

2011-05-09T16:35:13-07:00

When considering which platform to use for our website and for this blog I had a couple choices: use Wordpress or any other CMS, build our own custom solution or have a completely static website. I went with none of those and chose a small tool with great powers - Jekyll. This post is a walkthrough and an introduction to it and when it makes sense to use it.

Jekyll is a static site generator and that’s exactly what it does - turning a collection of templates into HTML files. You can then deploy them to any web server and all of a sudden website is live. Content can be written using your usual editor, so you can write your blog with VIM if you feel like it. Absolutely no need for a database, PHP or any other external dependency, which brings us to Jekyll advantages.

Advantages

Well, it generates HTML pages, so it makes a secure website (you can’t really hack a HTML page, now can you?). In comparison, Wordpress is known to have security holes, as any software you say, but blogs are often under heavy attacks by spam bots, so the more secure the better. Unless you have misconfigured your server, I doubt it’s even theoretically possible to get hacked.

HTML pages bring another advantage of being crazy fast. PHP for example is fast enough for most of the stuff, but still falls behind static pages, for obvious reasons. I like fast responding websites and the only limit I want to have is how fast my internet connection is. This blog is being served by Nginx, an awesome tool by itself, and load times are pretty much instant.

I have started my career by creating HTML websites years ago and still can do it very quickly. Once I have received a HTML code from my designer I don’t need to wrap it around Joomla or Wordpress specific template structure - with Jekyll it can be used as it is. Makes it ideal for creating websites fast and because there aren’t billion templates, views and partials to juggle with, fixing bugs in markup is child’s play.

Disadvantages

Static pages are static, thus requiring some workarounds for your usual dynamic parts. Contact forms cannot be implemented as there is no handler for handling the POST data. Luckily there are services like Wufoo which work for most of the tasks, but require more fiddling that just creating a PHP handler. Same is for comments, ratings etc. anything requiring to store data.

There is no online interface for editing so it won’t work well for clients’ websites. I can imagine hacks which would make it somewhat possible, but let’s leave them out. This is not an issue for me as there is no need to edit anything on this blog once it’s deployed (I can redeploy in a matter of seconds anyway), however clients will need this feature. Thus rendering Jekyll usable only for your own projects or limited-editing websites.

That’s pretty much it. Well, at least for me. If none of that sound like a big show stopper, I suggest you try it out.

How do I start?

Project is hosted on Github with full installation instructions, but in most systems it can be installed as easy as using:

gem install jekyll

Then you obviously need a sample project. Create index.html and _config.yml in some folder and run this command from it:

jekyll --server --auto

This will launch a web server and --auto command makes it monitor directory for changes, so as soon as you change some files it will regenerate the site. Guide your browser to http://localhost:4000/ and you should see your website. You can go from there and start adding more pages, using layouts etc., all this information is in the manual. One last thing - generated website is stored _site folder and this is the folder you can deploy to anywhere.

I use Markdown for most of the content, but it supports a bunch more and obviously regular HTML files. Templates can be written using Liquid language which is quite nice and more complicated tasks can be written as plugins (categories and tags pages on this blog are generated with them). There is a gallery of websites based on Jekyll here.

Conclusion

Next time you are considering building a not complicated website or a blog give a look at Jekyll. It worked great for us and I’m very happy with flexibility and lightness of it. It’s also very easy to start with and potentially Liquid templates can be later migrated to other frameworks (Liquid is for example very close to Twig).

P.S.
Source code of this blog is actually opensourced too and available here, so feel free to hack with it. You can do this:

git clone git@github.com:webspecies/blog.webspecies.co.uk.git
cd blog.webspecies.co.uk/
jekyll --server

JavaScript is not your boss

2011-05-05T17:45:13-07:00

If you’d have met me personally you would probably know that I don’t like JavaScript. But in a different way. I don’t like JavaScript because most of the time, I believe, it’s implemented in a way which just doesn’t make sense. It still makes sense for the end-user, which is awesome, but from what-web-was-supposed-to-be balcony it’s far far away.

From my perspective, JavaScript is a layer of logic (or code) sitting on top of HTML. So you have a HTML page with links, images and text, and some JavaScript which makes nice animations, adds Google ads and maybe something else. Is it always like this? Well, you guessed it – no. Let’s dive in to see how we can make things better, without using Flash of course.

Content

JavaScript!

Let’s go back a bit and see what JavaScript offers and how it changed websites. First thing which comes to my mind is Ajax – a way to asynchronously retrieve some data after the page has finished loading. This post is going to be mainly about it.

You might have a product page with a few tabs like “Description”, “Specifications”, “Reviews” etc. and each of those is loaded dynamically when that tab is clicked on. Makes it possible to fit more data in one page and have use chose which one to show. I love it when websites do that.

But this functionality should not break the website structure from JavaScript-disabled browser’s point of view. Nowadays one might ignore the users who have chosen to disable JavaScript, but what you can’t ignore is Google. And the reason why you can’t ignore Google is because it makes you money; at least it does for most of the websites I had a chance to work on.

Missing content

Now if you look at the product page example I have mentioned above, if it is implemented like I have described, with all tabs being loaded once requested, search engines’ spiders cannot see their content. And this is bad. All of the sudden you have valuable, relevant and semantically related information not being in that page, which makes that page rank low for a lot of valuable keywords. I’ve seen this happen a lot.

There are cases when it is allowed to ignore all of this, for example if you are Grooveshark. But this is more like an exception from a rule, in most cases it’s recommended to have HTML fall-back version of the website. Of course this is going to require more work than just creating a JavaScript-only version; however benefits of doing so are going to be larger.

Browser side

Ajax also allows to POST data back to the server without reloading the page. So contact form can show a nice thank you message without redirect user to any other page. Again, pretty cool feature. But when you start doing big forms (think order page) like that… You should be smart. Even if you do validation with JavaScript to make it faster for user to order something, you still need to do it server side also. Maintaining two sets of rules is tricky and you might get hacked. I was, once.

If you are a frontend developer you know that some things just can’t be achieved without some JavaScript sauce. Of course it’s quite crazy to expect a website to look/behave exactly the same when JavaScript is not available, but it should work.

“The right way”

Don’t code for JavaScript, code for content. Content is the boss, not JavaScript.

When I can I start by defining website structure, implementing all the underlying logic and pretty much finishing the whole website, before I start working on JavaScript. Obviously this is a generalization, but you get the idea. Yet again we can look at the product page example – once ready, add JavaScript to hide some parts of the HTML, thus only showing the current Tab. From user perspective UX is exactly the same.

If client/designer/kitten asks for some fancy navigation, have it mind, but don’t create a website based on that idea. Not because the requirements might (will) change, but because you are coupling presentation to data. If with disabled JavaScript a website doesn’t work at all or it’s impossible to navigate, you have coupling. And coupling is bad. And will make things bad, sooner or later.

Even simple things like lightbox effect (a popup-like window with enlarged image) should, ideally, work just fine without JavaScript. When it’s done like this:

<a href="big_image.jpg" class="lightbox"><img src="small_image.jpg" alt="Image" /></a>

It’s clean and clear. When like this:

<a href="javascript:onclick(openImage(‘big_image.jpg’))"><img src="small_image.jpg" alt="Image" /></a>

You are asking for problems.

Conclusion

This is not in any sense an argument against using JavaScript. It can help achieve amazing things, especially together with HTML5 (allowing to make RIA apps). But make sure website makes sense from JavaScript-disabled access points’ point of view, for example Google should be able to index all the data without any tricks.

P.S. Hashbang is omitted from this post to make it less rantty. This doesn’t mean hashbangs are good. They are not.

Sharing the knowledge

2011-05-04T17:05:03-07:00

As you might know from our website we work with a lot of different technologies, tools and projects. For example when building the website we used all the best from HTML5 and CSS3. It only makes sense for us to share our experiences to help others build things faster and better. This is the reason why we have created this blog - sharing our knowledge.

It’s my promise to you that you won’t find any company-related content or advertisement of our products in this blog. As you might have seen, we have a news section for this in our website and that’s where all that is going to be talked about. Blog on the other hand is mainly for developers and designers, so the content here is going to be very technical and thus different.

Our point of view is quite similar to the one of a company called 37signals, which is talked about in this talk:

We are not afraid to put out what we know best. Hey, most of the stack we use is already open-source so why not contribute back by blogging about it? Of course without revealing security mechanisms and unique algorithms, but if we solved HTML5 History API usage problems we are going to share it. In turn hopefully this will also bring us a couple new clients, that would be great.

This is just an introductory post, but the bare minimum we are going to do is one blog post per week, so make sure to subscribe our RSS feed and leave comments. I’ll see you soon!