Het bijstere spoor

Escaping in iCalendar and vCard

2013-05-10T15:57:12Z

The #1 bug report in my vObject library (a library to parse and create iCalendar and vCard objects in PHP) is that it does a bad job escaping/un-escaping of values.

In particular, it double-escapes certain values, changing things like ; into \\; and in other cases it's a bit too liberal un-escaping.

It's gotten to a point where I got so frustrated about this bug, I've been working all week on a new version of the parser.

Determined to do things right this time, I wanted to make sure I complied with all the relevant standards, in particular:

vCard 2.1
vCard 3.0 (rfc2425, rfc2426)
vCard 4.0 (rfc6350)
iCalendar 2.0 (rfc5545)

When I first wrote the vObject I naively thought that these formats were more or less the same. On the surface it does indeed seem that way, everything does seem to follow this basic structure:

BEGIN:VCARD
VERSION:4.0
FN:Evert Pot
END:VCARD

The nuances and slight difference between the specifications are enough to drive a simple person to madness though.

Just on the topic of ecaping values (the part after the :) the specifications have the following to say:

vCard 2.1

vCard 2.1, as well as the other specs have a concept of 'compound' or multi-value properties. An example:

BEGIN:VCARD
VERSION:2.1
N:Pot;Evert;Middle;Dr.;M.D.
END:VCARD

As you can see, the N property has multiple values. Any of these values may also contain a ;, which must be escaped as \;. So we also cannot blindly encode a string and automatically add backslashes to any ; we see.

The semi-colons should only be escaped in the ADR, ORG and N fields, but we can assume that backslashed semi-colons may also appear in other values.

Any property may have a parameter, a parameter looks a bit like this:

BEGIN:VCARD
VERSION:2.1
NOTE;ENCODING=QUOTED-PRINTABLE:Handsome guy, for sure..
END:VCARD

A parameter in vCard starts with a ;, has a name and a value. Only the colon may be escaped in parameters, using \:.

If you somehow wanted to encode a real backslash though, there's no mention of escaping it as a double-backslash.

If you need newlines in any values, quoted-printable encoding must be used. Other specs all encode newlines as \n or \N.

vCard 3.0

rfc2425 says that backslashes (\\), newlines (\N or \n) and comma's (\,) must always be escaped, no exceptions.. Well except when the comma is used as a delimiter for multiple values.

rfc2426 add semi-colon (\;) to this list, except when it's used as a delimiter. Semi-colon is used as a delimiter in the N, ADR, GEO and ORG fields. NICKNAME and CATEGORIES use comma's.

vCard also says that individual parts of ADR, and N may also contain multiple values themselves, which are themselves split by a comma.

Quoted-printable is now deprecated, and should no longer be used.

Parameters have also changed. The new rule is that parameters must not contain ;, : or ", unless they are surrounded by double-quotes, in which case only " may not appear. Escaping of the colon character (\:) has disappeared.

vCard 4.0

vCard 4 changes the interpretation of 3.0 a bit, and now states that semi-colons may be escaped, depending on the property.

The implication is that we need to maintain lists of properties, if they support multiple- or compound-values and which delimiter they use (, or ;).

Semi-colons are now used by N, ADR, ORG and CLIENTPIDMAP. Comma's are used by NICKNAME, RELATED, CATEGORIES and PID.

Even though the spec does say that comma's must always be escaped, it does appear to violate this rule in it's own examples, specifically the example for GEO (which is no longer a compound float value, but a url).

iCalendar 2.0

iCalendar 2.0 largely follows the same rules as vCard 4.0, but commas and semi-colons must be esacped, unless they are used as a delimiter.

Semi-colons are used as a delimiter in REQUEST-STATUS, RRULE, GEO and EXRULE, every other property uses commas.

rfc6868

One major flaw in all the above specs was that it was not possible to encode just any value as a parameter. Newlines are not allowed, and in no case can you encode a double-quote.

rfc6868 updates both iCalendar 2.0 and vCard 4.0 to use caret (^) as an escape character. To write a double quote, use ^', to encode a newline use ^n and to encode a caret, use ^^.

The hard part

A simple generic parser is with this information simply out of the window, not only will my parser have to be aware which document it's parsing, it will also have to make individual decisions based on which property it's parsing.

Researching and listing these rules helped, and I hope it's also helpful for a future implementor.

It's important to be strict in generating these formats, but considering the complexity of these rules, it's extremely likely other software has bugs when generating these (and they do! a lot!) any parser needs to be able to handle these mistakes and attempt to make logical decisions based on what likely the intention was.

Found mistakes?

You can fork this post or easily make edits of this post on Github.

SabreDAV acquired by fruux

2013-05-02T15:58:42Z

Last week we have finished the talks to transfer ownership of the SabreDAV project to fruux.

I've been working at fruux as a lead-dev for some time now, and SabreDAV is really at the core of everything we do, so this was a logical and inevitable choice.

To me it's an important step. SabreDAV has been one of the very little things that have remained consistent in my life. This validates my work from the last 6 years a bit. In another way it feels a bit like I'm letting go something that's very important and personal.

However, I'll still be running the project so in reality the change for me personally is not that big :).

What's SabreDAV?

SabreDAV is an open-source project that I've started all the way 2007. It allows you to write WebDAV, CalDAV and CardDAV servers in PHP. These three technologies allow you to do file, calendar and addressbook syncing.

A few years ago steam really picked up, and it changed from the hobby project that it was back then, to something I could actually make a living off by doing consulting.

In 2012 I've took it one step further, and teamed up with fruux to build a commercial product around SabreDAV.

What will change?

Small stuff really. The GitHub urls have changed. The new project pages are:

A bigger change is that fruux will now be the official go-to place for commercial support. I feel this will strengthen the SabreDAV project, especially for the potential users that are into that sort of thing.

It also means that getting a new website is in the cards :). The current one has passed it's expiry date.

Does this mean SabreDAV will go closed-source?

No. I would never let this happen. SabreDAV will remain as open source as it has always been, and we also don't intend on making it crippleware.

The BSD license in place today is very liberal, and I feel that it's exactly this that has allowed many projects and businesses to adopt it. Taking that away would not only damage the project, it would also feel ethically wrong.

However, SabreDAV is primarily a library, suitable for integration into other systems. We're definitely looking into the possibility of offering a more complete package, such as a standalone server with solid management interfaces.

Thanks!

Thanks everyone for your continued support, pull requests and relying on SabreDAV for your projects :). It's the best reward, really.

If you're interested, also check out the announcement on the fruux blog

A new blog!

2013-04-29T20:53:11Z

I just launched a new version of my blog, and you're looking at it :). I've went through a number of platforms, most recently habari, which I still have a lot of love for.

But my installation was going rapidly out of date, was becoming very slow due to using Sqlite as a backend, and most of all, I wanted to use GitHub to store my posts so everyone can make edits.

So I settled on using Jekyll, hosted on GitHub pages. Biggest drawback: it's all 100% static, so I had to use Disqus to enable comments. Don't really like the idea of someone else owning my precious, precious data, but alas… I'm in the 'cloud services business' myself, and you gotta practice what you preach.

The first responses have been an overwhelming "weren't you able to find a better template anywhere?". I'm obviously not much a visual artist myself, so I had no problem with the concept. But it's hard to find solid minimalistic designs, and I really wanted a strong focus on actual content, and a great mobile experience, so I've had some trouble finding things that matched those objectives.

Regardless, I'm quite happy with the look of embedded code, I mean… I think this looks pretty sweet:

<?php

include __DIR__ . '/vendor/autoload.php';

use Sabre\XML;

$reader = new XML\Reader();
$reader->open('atom.xml');

$output = $reader->parse();
var_dump($output);

If you recognize the color scheme, indeed.. It's the old vim 'desert' scheme ported to css.

I also ditched the old domainname (rooftopsolutions.nl). I started 'Rooftop Solutions' when I was a 17-year old kid, over 10 years ago. This has been my moniker ever since, but recently I officially 'shut down' this business when I moved from Münster, Germany to London.

Figured I might as well start fresh and use my real name.

While we're on the topic of London. If you're reading this and happen to also live in London. If you're up for some armchair philosophy or programming discussions over a pint, I'm still pretty new here and in the market for friends, so drop me a line!

To conclude: I hope a new system will encourage me to do a better job at blogging. I've been pretty shit last year, and this is my first post of 2013.

I'm actually not sure how relevant blogs still are in 2013. There's definitely been a steep decline, but I enjoy the medium as a consumer and I reckon 10 years down the road older blog posts may still hold some relevancy, unlike compressed twitter discussions with little context.

Anyway, if you're reading this, thanks for sticking with me. If you have any criticisms or suggestions, I'd love to hear it!

Switching to PHP 5.4 on OS X

2012-12-16T16:53:22Z

I like using OS X's built-in packages where possible, but unfortunately Apple is way behind with their PHP package, having it locked on 5.3.15. In the past I've seen people use tools like Mamp, or Xampp to provide this for them, but frankly I'm not a big fan of these tools.

Homebrew provides a solution. Homebrew is OS X missing package manager, and it's an absolute great tool to work with. Getting started with it is a bit harder, as there's a few bigger dependencies you need, such as an up-to-date XCode installation. Once you've installed homebrew, it's a matter of running the following commands:

brew tap homebrew/dupes
brew tap josegonzalez/homebrew-php
brew install php54 --with-mysql 
brew install php54-memcached
brew install php54-xdebug

After that, open /etc/apache2/httpd.conf, and look for this line:

LoadModule php5_module libexec/apache2/libphp5.so

Comment that out (with #) and add the following line:

LoadModule php5_module /usr/local/Cellar/php54/5.4.12/libexec/apache2/libphp5.so

After that, restart apache and things should just work. For more detailed instructions, take a look at the documentation for homebrew-php

Edit: On one Mac I had to first run "brew unlink libiconv" for PHP compilation to complete.

SabreDAV 1.8 released (with namespaces!)

2012-11-09T12:50:59Z

Only a month after 1.7 I just released SabreDAV 1.8.

Not that much changes since 1.7, except that I migrated all old prefix-style code to PHP 5.3 namespaces.

This means that when you used to refer to a class as "Sabre_DAV_Server", this is now "Sabre\DAV\Server".

So for the most part it's a simple mapping, but there were a few exceptions. While "Sabre_CardDAV_Backend_Abstract" is a valid PHP class name, "Sabre\CardDAV\Backend\Abstract" is not, because the last bit (abstract) is a keyword.

I migrated all 343 classes and interfaces by hand(!). There are some tools out there that do the conversion automatically, but I didn't just want to apply a blanket patch, I wanted to make sure I 'use' the right classes and namespaces in a way that it made sense. Nothing beats the human touch.

Some links

Markdown output for PHPDocumentor2

2012-11-09T12:50:21Z

I'm moving some codebases to PHP namespaces, and as a result I could no longer use my crappy API documentation script.

So, I needed a replacement, and with the help of PHPDocumentor, I was able to create a simple package that outputs github-compatible markdown documentation, based on twig templates.

Here's an example.

It's not perfect yet, but hopefully useful for others in the same boat. You can find the project and setup instructions here:

https://github.com/evert/phpdoc-md.

SabreDAV 1.7 released

2012-10-06T12:47:38Z

I just released SabreDAV 1.7, after about 7 months of work. A lot of work has gone into optimizing operations, and a bunch of new features have been added. I also stopped supporting the PEAR packages, and fully migrated to composer. If you download the zip, the required composer files are actually shipped along.

SabreDAV's uptake and contributions has also been very encouraging. This is especially nice since it solves such a niche problem :).

Most relevant new stuff:

A bunch of REPORT queries got a lot faster. Especially some CalDAV-related requests can be exponentially faster. Some larger operations sped up from several minutes, to several seconds.
The VObject library has spun off into a separate project: sabre-vobject.
Experimental support for caldav-sharing and caldav-notifications.
Added a VCFExportPlugin, similar to the ICSExportPlugin.
Added a bunch of free-busy reports that now make it possible for iCal to use them.
Added support for a PATCH format for binary updates
Switched to using the Composer autoloader.
Stronger validation of both iCalendar and vCard files.
Support for Brief and Prefer: return-minimal headers.

Some links

A big thank you to everybody involved!

New open source project: Sabre VObject

2012-08-08T08:13:35Z

Over the last little while I've been working at fruux on a new open source project: an iCalendar/vCard parser for PHP.

Actually, the project has existed since 2010 as it's really a spin-off from SabreDAV. I felt like the library could appeal to a wider audience, and benefit from a separate release schedule.

The tool heavily makes use of PHP's magic object and array accessors, to provide an API quite similar to what simplexml is for XML. In addition it contains features for parsing dates, expanding recurrence rules and automatically creating FREEBUSY reports.

To start using it, simply check out the github project page. The readme contains a lot of documentation.

Some things I want to add in the future:

Conversion tools for vcalendar 1.0, vcard 2.1, 3.0, 4.0, etc.
A relaxed parsing mode, that's a bit more forgiven to broken formats.
Convenience API's for formatting and parsing property values.
Validation for all property values and components.

Currently no zip is provided. You should simply add it to your composer dependencies, as:

{
    "require" : {
      "sabre/vobject" : "2.0.*"
    }
}

I hope it's useful, and would love to hear your feedback!

OAuth 2.0 and the Road to Hell

2012-07-26T15:50:22Z

I just came across a a very interesting blog post by Eran Hammer about OAuth 2.0, its progress and its past.

If you're considering OAuth, it's worth a read. It also kind of reflects how I initially felt when I looked at the sea of OAuth 2-related documentation. My experience from tracking OpenID and DAV related standards is very similar. It is very, very hard for committees to create simple standards. I reckon the only way it can work, is with a trusted BDFL-type at the helm.

IE10 and DNT-header update

2012-06-07T16:30:11Z

A quick update, since my last post Microsoft has reverted it's decision to enable the DNT header by default. I sort of expected this, and I'm glad they did.

Ars Technica has the details.

IE10 will enable the DNT header by default, and why I think this is a bad thing.

2012-06-02T13:49:47Z

Microsoft recently announced that they will automatically enable the 'Do Not Track' header in Internet Explorer 10.

The DNT header allows a user to opt-out of being tracked by websites. This was introduced by Firefox, and now also has support from Opera and Safari (but not Chrome).

This header is a great feature. Before, the ad providers had to ask the user on their website to opt-out, and this actually was usually implemented using a Cookie.

Although not every ad publisher supports the DNT header yet, it's gaining traction. However, publishers are not forced to comply with this. They do have to provide an opt-out mechanism to comply with many countries' laws, but there's nowhere stated this has to be done using the DNT header.

Currently, browsers have the DNT setting disabled by default. If IE10 enables it, this means that IE10 users will have to explicitly opt-in to allow publishers to track them. This is very different from how Firefox and other browsers deal with this, as they've chosen for the explicit opt-out.

I strongly suspect that publishers will feel that their tracking ability will now be severely diminished by supporting the header. Since they are not forced to support the DNT header, there will be much less of an incentive to start supporting it. They can just provide their own inferior cookie mechanism. I can even see some publishers that already support the DNT header reverse this decision.

So while I commend Microsoft for putting the consumer first; If we want the DNT header to work, publishers must either want to support it, or must be forced to by law. Until the latter is the case, I hope Microsoft reverses this decision to ensure that the DNT header stays useful.

References

New HTTP status codes

2012-05-04T20:44:14Z

RFC 6585 has been published quite recently. This document describes 4 new HTTP status codes.

So in case you were wondering, yes.. HTTP is still evolving :), and these new statuses may be quite useful for developing your REST, or otherwise HTTP-based service. This post describes why they are important, and when you should use them.

428 Precondition Required

A precondition is something a client can send along with a HTTP request. This condition needs to be met in order for the request to complete.

A good example is the If-None-Match header, which is often used along with GET requests. If the If-None-Match is specified, the client can request to only receive the response if the ETag changed.

Another example of a precondition, is the similar 'If-Match' header. An If-Match header is usually sent along with PUT requests to indicate to only update the resource if it hasn't changed. This is useful if multiple clients are using a HTTP-based service, and they want to make sure they are not overwriting each others contents.

Using the 428 Precondition Required status, the server can now indicate that the client *must* send along one of those headers to perform the request. This is effectively a way for the server to force clients to prevent this 'lost update' problem.

429 Too Many Requests

This status code is useful in cases where you want to limit the amount of requests a client may want to do on your API (also known as rate limiting).

In the past different status codes have been used, such as '509 Bandwidth Limit Exceeded'. Twitter uses 420 for some stuff (which is an unused status code). Thus it was important enough to give it it's own code.

So if you limit the number of requests clients may do on your server, 429 Too Many Requests is the way to go. Include a 'Retry-After' response header to indicate to a client when they are allowed to make requests again.

431 Request Header Fields Too Large

I was surprised to see that this was a common enough usecase to warrant it's own status code, but here it is!

In a case where a client is sending a HTTP request header that's too big, the server can respond with 431 Request Header Fields Too Large to indicate exactly that.

I have no idea why they skipped over 430 though. I tried to search around, but couldn't quite find the reasoning. My best guess is that a lot of people may have mistyped '403 Forbidden' as '430 Forbidden', and they wanted to avoid complications. If you know, let me know!

511 Network Authentication Required

This status code is very interesting to me. You will not have to deal with this if you're writing a server, but it can be important if you're writing a (desktop) HTTP client.

If you move around with your laptop or smartphone a lot, you may have noticed that a lot of public wifi services now require you to accept their license agreement, or just log in before the web works.

This is generally done by intercepting the HTTP traffic and presenting a redirect and login when the user tries to access the web. Quite nasty, but that's the way it is.

Using these 'intercepting' clients can have some nasty side effects. There are two great examples mentioned in the RFC to illustrate this.

If you hit a website before logging in, the network device intercepts the first request. These devices also tend to have a 'favicon.ico' stored. After logging in, you'll notice that the favicon is now cached for the website you tried to visit, and it may follow you around for quite some time.
If a client uses HTTP requests to find documents, the 'network' may respond with a login page, instead of the json or other document you expected. Your client may (in error) assume it's a 'normal' response and use that instead. This can put clients in a broken or irrecoverable state. I've noticed this in real life a few times working on a CalDAV system as well.

So to fix this 511 Network Authentication Required is introduced.

So if you write an application that runs on an desktop or phone and use HTTP, you should ideally check for this HTTP response code. In a way, it simply means that a network is not yet available and you should pretty much ignore anything coming back until it is. You could even provide the user with the returned login page, like iOS and OS X 10.7 do.

New full-time gig at fruux

2012-03-21T13:44:50Z

I've been at this for a bit, but now it's official: I'm now based in Münster, and working at fruux as a co-founder / tech-lead.

The feeling of having this amount of control over a product is great. We've also recently got funded from High-Tech Gründerfonds and netSTART Venture and moved into a new office so things are indeed pretty serious. We have some flexibility to focus on making a great product, in what we feel is 'the right way' (whatever that may mean). It will be an exciting time :)

So if you were looking at SabreDAV, and needed something way easier to use, fruux may be a good option. Regardless, for SabreDAV the future is bright too, as there will be plenty of cross-pollination.

Also, if you're looking for a job, or know someone who's looking: drop me a line! We're in the market for good PHP and front-end developers.

Content Security Policy update

2012-02-06T22:57:04Z

A quick update about CSP. Browsers are well on their way to all adopt the specification.

An early draft was already adopted by Firefox 4, and I just found out that it's also working in Chrome, Safari and IE 10.

IE10 and FF are using the following header:

X-Content-Security-Policy: default-src 'self'

While Safari and Chrome use:

X-Webkit-CSP: default-src 'self'

When the specification is finalized, the X- will be dropped, and it will simply be 'Content-Security-Policy'.

A call for support

Hi Developers! Start implementing this feature! It's important for the future and security of the web. The web's biggest vulnerability, from what I understand, is still XSS, but if people start to properly implement CSP, XSS can effectively be a thing from the past.

So even if you don't want to risk using CSP on a production environment, at least consider adding the headers in your development environment. It will force you to write better code, by not embedding javascript directly into the HTML source. By considering this right now, you will also make it much easier if you do decide to adopt CSP at some point in the future.

I'm implementing CSP full-on in a new project, and one of the things I've noticed already is that some of the javascript we embed from 3rd parties use eval() and inline html events (onclick & friends). For the sake of security we will most likely decide to only use 3rd party code if they are indeed CSP-compatible.

Silex routing issues

2012-01-25T14:03:52Z

I've had a bit of an interest for micro frameworks, and for a new project I'm working on I decided on Silex.

I've started hitting some nasty walls though, the routing system seems inconsistent and intuitive. To better explain, some examples.

I have a default route that acts as a fallback. This default route simply includes a static template (if it exists). This route looks something like this:

<?php

$app->get('/{name}', function($name) { 
   // If the page exists, render it, otherwise throw 404
});

Simple enough. We also want to match the root of the application (the index, basically). This can be easily done with a default value:

<?php

$app->get('/{name}', function($name) { 
   // If the page exists, render it, otherwise throw 404
})->value('name','index');

The first problem arose here. The existing site used urls all ending with a slash (/contact/ for example). But these routes won't match that. These routes will match /contact, but not /contact/. This can be fixed though. If your routes end with a slash, silex will be able to match both:

<?php

$app->get('/{name}/', function($name) { 
   // If the page exists, render it, otherwise throw 404
})->value('name','index');

The preceding example will match /contact/. If the user went to /contact instead, it will redirect the user back to /contact/. I would have preferred the opposite in this case, but fair enough. This route will however no longer match the root of the website, so we need to refactor this a bit:

<?php

$staticHandler = function($name) {
  // If the page exists, render it, otherwise throw 404
}

$app->get('/{name}/', $staticHandler);
$app->get('/', function() use ($staticHandler) { 
  return $staticHandler('index'); 
} );

Note that the last example could have been structured a bit nicer, but you get the point.

The first real problem arose when trying to use 'ControllerProviders, or ControllerCollections'. These objects can be responsible for a sub-tree. An example of how we want to use this is have a single class be responsible for a /blog/ section of the site. This would be done with something like:

<?php

$app->mount('/blog/', new MySite\Controller\Blog());

RFC search provider

2012-01-21T18:10:08Z

In my work I have to look a lot at the IETF rfc documents, and I always have to jump through a bunch of hoops to find the document I want. Google often links me to the text version, and often I'm looking at obsoleted versions of documents when a new rfc has been published.

Well, so I created a simple OpenSearch provider to solve this. It does exactly what I need, and I only tested in Firefox. If it's useful to anyone else, you can add the provider from rfcsearch.phpfogapp.com.

It works with a simple hardcoded list of known keywords, and it will fall back to "Google's I'm feeling lucky" if it doesn't know the keyword. Feel free to fork it or laugh at my source code.

My PHP Advent article

2011-12-06T21:31:56Z

My PHP Advent article just got published. It's a list of best practices around dealing with dates and times in PHP. Have a read and tell me what you think. Also, be sure to follow @phpadvent or subscribe.

PHP Includes file generator

2011-11-29T15:55:58Z

While profiling SabreDAV, I noticed a few times more than half of the request time was spent in the autoloader.

So instead of autoloading, now I prefer to unconditionally include every file for each package (there are 5 packages). For a while I manually maintained these files manually, but a while back I automated this process.

This is how you run it:

phpincludes . includes.php

This will generate a file such as:

<?php

// Begin includes
include __DIR__ . '/Interface1.php';
include __DIR__ . '/Class1.php';
include __DIR__ . '/Class2.php';
include __DIR__ . '/Class3.php';
// End includes

You can edit everything before "// Begin includes" and after "// End includes". Subsequent edits will only replace the lines in between those comments.

The script will automatically expand classes and interface dependencies and load them in the correct order. It also has support for a PHP 5.2-compatible syntax (dirname(__FILE__) instead of __DIR__).

If you like it, head over to github, or install it with:

pear channel-discover pear.sabredav.org
pear install sabredav/PHPIncludes

Moved SabreDAV to Github

2011-10-10T15:13:50Z

I've finally taken the plunge and moved the SabreDAV source from Google code to GitHub.

The main reason is that I'm hoping that people are more likely to fork and contribute. This was possible as well on googlecode, but I think people had a bit more trouble getting used to the feature.

Mercurial is still the best source control system, but I feel switching to Git was worth it, solely for GitHub.

You can find the source at https://github.com/evert/SabreDAV.

Timezone database closed down

2011-10-07T11:53:39Z

Interesting news today. The Olson database, which is used in countless operating systems and other software (including PHP) has been shut down due to a Copyright claim.

Details, which I won't repeat here can be found on Stephen Colbourne's blog.