My worst mistakes in programming</h1> <article> <h1>My worst mistakes in programming</h1> <p>2012-01-13T00:00:00+01:00</p> <p>I'm in the middle of refactoring a big infrastructure piece in our product <a href="http://www.popscan.com">PopScan</a>. It's very early code, rarely touched since its inception in 2004, so I'm dealing mainly with my sins of the past.</p> <p>This time like no time before, I'm feeling the two biggest mistake I have ever made in designing a program, so I though I'd make this post here in order to help others not fall into the same trap.</p> <p>Remember this: Once you are no longer alone working on your project, the code you have written sets an example. Mistakes you have made are copied - either verbatim or in spirit. The design you have chosen lives on in the code that others write (rightfully so - you should strive to keep code consistent).</p> <p>This makes it even more important not to screw up.</p> <p>Back in 2004 I have failed badly at two places.</p> <ul> <li><p>I chose a completely wrong abstraction in class design, mixing two things that should be separate.</p></li> <li><p>I chose - in a foolhearted whish to save on CPU time to create a ton of internal state instead of fetching the data when it's needed (I could still cache then, but I missed that).</p></li> </ul> <p>So here's the story.</p> <h2>One is the architectural issue.</h2> <p>Let me tell you, dear reader, should you <em>ever</em> be in the position of having to do anything even remotely related to an ecommerce solution dealing with products and orders, so repeat with me:</p> <blockquote><p>Product lists are not the same thing as orders. Orders are not the same thing as baskets.</p></blockquote> <p>and even more importantly:</p> <blockquote><p>A product and a line item are two completely different things.</p></blockquote> <p>A line item describes how a specific product is placed in a list, so at best, a product is contained in a line item. A product doesn't have a quantity. A product doesn't have a total price.</p> <p>A line item does.</p> <p>And when we are at it: «quantity» is not a number. It is the entitiy that describes the amount of times the product is contained within the line item. As such a quantity usually consists of an amount and a unit. If you change the unit, you change the quantity. If you change the amount, you change the quantity.</p> <p>Anyways - sitting down and thinking of the entities in the feature that you are implementing is an essential part of the work that you do. Even it it seems "kinda right" at the time, even if it works "right" for years - once you make a mistake at a bad place, you are stuck with it.</p> <p>PopScan is about products and ordering them. Me missing the distinction between a product and a line item back in 2004 worked fine until now, but as this is a core component of PopScan, it has grown the most over the years, more and more intertwining product and line item functionality to the point of where it's too late to fix this now or at least it would require countless hours of work.</p> <p>Work that will have to be done sooner rather than later. Work that deeply affects a core component of the product. Work that will change the API greatly and as such can only be tested for correctness in integration tests. Unit tests become useless as the units that are being tested won't exist any more in the future.</p> <p>Painful work.</p> <p>If only I had more time and experience those 8 years ago.</p> <h2>The other issue is about state</h2> <p>Let's say you have a class <code>FooBar</code> with a property <code>Foo</code> that is exposed as part of the public API via a <code>getFoo</code> method.</p> <p>That <code>Foo</code> relies of some external data - let's call it <code>foodata</code>.</p> <p>Now you have two options of dealing with that <code>foodata</code>:</p> <ol> <li><p>You could read <code>foodata</code> into an internal <code>foo</code> field at construction time. Then, whenever your <code>getFoo()</code> is called, you return the value you stored in <code>foo</code>.</p></li> <li><p>Or you could read nothing until <code>getFoo()</code> is called and then read <code>foodata</code> and return that (optionally caching it for the next call to <code>getFoo()</code>)</p></li> </ol> <p>Chosing the first design for most of the models back in 2004 was the second biggest coding mistake I have ever made in my life.</p> <p>Aside of the fact that constructing one of these <code>FooBar</code> objects becomes more and more expensive the more stuff you preload (likely never to be used for the lifetime of the object), you have also contributed to a huge amount of internal state of the object.</p> <p>The temptation to write a <code>getBar()</code> method that has a side effect of also altering the internal foo field is just too big. And now you end up with a <code>getBar()</code> that suddenly also depends on the internal state of <code>foo</code> which suddenly is disconnected from the initial <code>foodata</code>.</p> <p>Worse, suddenly calling code will see different results depending on whether it calls <code>getBar()</code> before it's calling <code>getFoo()</code>. Which will of course lead to code depending on that fact, so fixing it becomes very hard (but at least caught by unit tests).</p> <p>Having the internal fields also leads to <code>FooBar</code>'s implementation preferring these fields over the public methods, which is totally fine, as long as <code>FooBar</code> stands alone.</p> <p>But the moment there's a <code>FooFooBar</code> which inherits from <code>FooBar</code>, you lose all the advantages of polymorphism. <code>FooBar</code>'s implementation will always only use its own private fields. It's impossible for <code>FooFooBar</code> to affect <code>FooBar</code>'s implementation, causing the need to override many more methods than what would have been needed if <code>FooBar</code> used its own public API.</p> <h2>Conclusion</h2> <p>These two mistakes cost us hours and hours of working around our inability to do what we want. It cost us hours of debugging and it causes new features to come out much more clunky than they need to be.</p> <p>I have done so many bad things in my professional life. A <code>shutdown -h</code> instead of -r on a remote server. A <code>mem=512</code> boot parameter (yes. That number is/was interpreted as bytes. And yes. Linux needs more than 512 bytes of RAM to boot), an <code>update</code> without <code>where</code> clause - I've screwed up so badly in my life.</p> <p>But all of this is <em>nothing</em> compared to these two mistakes.</p> <p>These are not just inconveniencing myself. These are inconveniencing my coworkers and our customers (because we need more time to implement features).</p> <p>Shutting down a server by accident means 30 minutes of downtime at worst (none since we heavily use VMWare). Screwing up a class design twice is the gift that keeps on giving.</p> <p>I'm so sorry for you guys having to put up with <code>OrderSet</code> of doom.</p> <p>Sorry guys.</p> </article> <article> <h1>Abusing LiveConnect for fun and profit</h1> <p>2011-12-22T00:00:00+01:00</p> <p>On december 20th I gave a talk at the JSZurich user group meeting in Zürich. The talk is about a decade old technology which can be abused to get full, unrestricted access to a client machine from JavaScript and HTML.</p> <p>I was showing how you would script a Java Applet (which is completely hidden from the user) to do the dirty work for you while you are creating a very nice user interface using JavaScript and HTML.</p> </iframe> <p>The slides are <a href="http://bit.ly/vUmkZH">available in PDF format</a> too.</p> <p>While it's a very cool tech demo, it's IMHO also a very bad security issue which browser vendors and Oracle need to have a look at. The user sees nothing but a dialog like this:</p> <p> and once they click OK, they are completely owned.</p> <p>Even worse, while this dialog is showing the case of a valid certificate, the dialog in case of an invalid (self-signed or expired) certificate isn't much different, so users can easily tricked into clicking allow.</p> <p>The source code of the demo application is on <a href="https://github.com/pilif/gravedigging">github</a> and I've already written about this on this blog <a href="/2009/04/javascript-and-applet-interaction/">here</a>, but back then I was mainly interested in getting it work.</p> <p>By now though, I'm really concerned about putting an end to this, or at least increasing the hurdle the end-user has to jump through before this goes off - maybe force them to click a visible Applet. Or just remove the <a href="http://en.wikipedia.org/wiki/LiveConnect">LiveConnect</a> feature all together from browsers, thus forcing applets to be visible.</p> <p>But aside of the security issues, I still think that this is a very interesting case of long forgotten technology. If you are interested, do have a look at the talk and travel back in time to when stuff like this was only half as scary as it is now.</p> </article> <article> <h1>updated sacy - now with external tools</h1> <p>2011-11-09T00:00:00+01:00</p> <p>I've just updated the <a href="https://github.com/pilif/sacy">sacy repository</a> again and tagged a v0.3-beta1 release.</p> <p>The main feature since yesterday is support for the official compilers and tools if you can provide them on the target machine.</p> <p>The drawback is that these things come with hefty dependencies at times (I don't think you'd find a shared hoster willing to install node.js or Ruby for you), but if you can provide the tools, you can get some really nice advantages over the PHP ports of the various compilers:</p> <ul> <li><p>the PHP port of sass has <a href="http://code.google.com/p/phamlp/issues/detail?id=116">an issue</a> that prevents @import from working. sacy's build script does patch that, but the way they were parsing the file names doesn't inspire confidence in the library. You might get a more robust solution by using the official tool.</p></li> <li><p>uglifier-js is a bit faster than JSMin, produces significantly smaller output and comes with a better license (JSMin isn't strictly free software as it has this "do no evil" clause)</p></li> <li><p>coffee script is under very heavy development, so I'd much rather use the upstream source than some experimental fun project. So far I haven't seen issues with coffeescript-php, but then I haven't been using it much yet.</p></li> </ul> <p>Absent from the list you'll find less and css minification:</p> <ul> <li><p>the PHP native <a href="http://code.google.com/p/cssmin/">CSSMin</a> is really good and there's no single official external tool out that demonstrably better (maybe the YUI compressor, but I'm not going to support something that requires me to deal with Java)</p></li> <li><p><a href="http://leafo.net/lessphp/">lessphp</a> is very lightweight and yet very full featured and very actively developed. It also has a nice advantage over the native solution in that the currently released native compiler does not support reading its input from STDIN, so if you want to use the official less, you have to go with the git HEAD.</p></li> </ul> <p>Feel free to try this out (and/or send me a patch)!</p> <p>Oh and by the way: If you want to use uglifier or the original coffee script and you need node but can't install it, have a look at the <a href="http://pilif.github.com/2011/11/node-to-go/">static binary</a> I created</p> </article> <article> <h1>updated sacy - now with more coffee</h1> <p>2011-11-08T00:00:00+01:00</p> <p>I've just updated the <a href="https://github.com/pilif/sacy">sacy repository</a> to now also provide support for compiling Coffee Script.</p> <div class="highlight"><pre><code class="xml">{asset_compile} <span class="nt"><script</span> <span class="na">type=</span><span class="s">"text/coffeescript"</span> <span class="na">src=</span><span class="s">"/file1.coffee"</span><span class="nt">></script></span> <span class="nt"><script</span> <span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"/file2.js"</span><span class="nt">></script></span> {/asset_compile} </code></pre> </div> <p>will now not compile file1.coffee into JS before creating and linking one big chunk of minified JavaScript.</p> <div class="highlight"><pre><code class="xml"><span class="nt"><script</span> <span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"/assetcache/file2-deadbeef1234.js"</span><span class="nt">></script></span> </code></pre> </div> <p>As always, the support is seamless - this is all you have to do.</p> <p>Again, in order to keep deployment simple, I decided to go with a pure PHP solution (<a href="https://github.com/alxlit/coffeescript-php">coffeescript-php</a>).</p> <p>I do see some advantages in the native solutions though (performance, better output), so I'm actively looking into a solution to detect the availability of native converters that I could shell out to without having to hit the file system on every request.</p> <p>Also, when adding the coffee support, I noticed that the architecture of sacy isn't perfect for doing this transformation stuff. Too much code had to be duplicated between CSS and JavaScript, so I will do a bit of refactoring there.</p> <p>Once both the support for external tools and the refactoring of the transformation is completed, I'm going to release v0.3, but if you want/need coffee support right now, go ahead and clone <a href="https://github.com/pilif/sacy">the repository</a>.</p> </article> <article> <h1>node to go</h1> <p>2011-11-07T00:00:00+01:00</p> <p>Having node.js around on your machine can be very useful - not just if you are <a href="/tags/tempalias/">building your new fun project</a>, but also for quite real world applications.</p> <p>For me it was <a href="http://jashkenas.github.com/coffee-script/">coffee script</a>.</p> <p>After reading some incredibly beautiful coffee code by <a href="https://twitter.com/brainlock">@brainlock</a> (work related, so I can't link the code), I decided that I wanted to use coffee in PopScan and as such I need coffee support in sacy which handles asset compilation for us.</p> <p>This means that I need node.js on the server (sacy is allowing us a very cool checkout-and-forget deployment without any build-scripts, so I'd like to keep this going on).</p> <p>On servers we manage, this isn't an issue, but some customers insist on hosting PopScan within their DMZ and provide a pre-configured Linux machine running OS versions that weren't quite current a decade ago.</p> <p>Have fun compiling node.js for these: There are so many dependencies to meet (a recent python for example) to build it - if you even manage to get it to compile on these ancient C compilers available for these ancient systems.</p> <p>But I really wanted coffee.</p> <p>So here you go: Here's a statically linked (this required a bit of trickery) binary of node.js v0.4.7 compiled for 32bit Linux. This runs even on an ancient RedHat Enterprise 3 installation, so I'm quite confident that it runs everywhere running at least Linux 2.2:</p> <p><a href="http://www.pilif.ch/node-x86-v0.4.7.bz2" checksum="sha256:142085682187a57f312d095499e7d8b2b7677815c783b3a6751a846f102ac7b9">node-x86-v0.4.7.bz2</a> (SHA256: 142085682187a57f312d095499e7d8b2b7677815c783b3a6751a846f102ac7b9)</p> <pre><code>pilif@miscweb ~ % file node-x86-v0.4.7 node-x86: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, statically linked, for GNU/Linux 2.2.5, not stripped </code></pre> <p>The binary can be placed wherever you want and executed from there - node doesn't require any external files (which is very cool).</p> <p>I'll update the file from time to time and provide an updated post. 0.4.7 is good enough to run coffee script though.</p> </article> <article> <h1>protecting siri</h1> <p>2011-10-31T00:00:00+01:00</p> <p>Over the last weekend, 9to5mac.com <a href="http://9to5mac.com/2011/10/29/siri-hacked-to-fully-run-on-the-iphone-4-and-ipod-touch-iphone-4s-vs-iphone-4-siri-showdown-video-interview/">posted about a hack</a> which shows that it's possible to run Siri on a iPhone 4 and an iPod Touch 4g and possibly even oder devices - considering how much of Siri is running on Apple's servers.</p> <p>We've always suspected that the decision to restrict Siri to the 4S is basically a marketing decision and I don't really care about this either. Nobody is forcing you to use Siri and thus nobody is forcing you to update to anything.</p> <p>Siri is Apple's product and so are the various iPhones. It's their decision whom they want to sell what to.</p> <p>What I find more interesting is that it was even possible to have a hacked Siri on a non 4S-phone talk to Apple's servers. If I were in Apple's shoes, I would have made that (practically) impossible.</p> <p>And here's how:</p> <p>Having a device that you put into users hands and trusting it is always a very hard, if impossible thing to do as the device can (more or less) easily be tampered with.</p> <p>So to solve this problem, we need some component that we know reasonably well to be safe from the user's tampering and we need to find a way for that component to prove to the server that indeed the component is available and healthy.</p> <p>I would do that using public key crypto and specialized hardware that works like a TPM. So that would be a chip that contains a private key embedded in hardware, likely not updatable. Also, that private key will never leave that device. There is no API to read it.</p> <p>The only API the chip provides is either a relatively high-level API to sign an arbitrary binary blob or, more likely, a lower level one to encrypt some small input (a SHA1 hash for example) with the private key.</p> <p>OK. Now we have that device (also, it's likely that the iPhone already has something like that for its secured boot process). What's next?</p> <p>Next you make sure that the initial handshake with your servers requires that device. Have the server post a challenge to the phone. Have the phone solve it and have the response signed by that crypto device.</p> <p>On your server, you will have the matching public key. If the signature checks out, you talk to the device. If not, you don't.</p> <p>Now, it is possible using very expensive hardware to extract that key from the hardware (by opening the chip's casing and using a microscope and a lot of skills). If you are really concerned about this, give each device a unique private key. If a key gets compromised, blacklist it.</p> <p>This greatly complicates the manufacturing process of course, so you might go ahead with just one private key per hardware type and hope that cracking the key will take longer than the lifetime of the hardware (which is very likely).</p> <p>This isn't at all specific to Siri of course. Whenever you have to trust a device that you put into consumers hands, this is the way to go and I'm sure we'll be seeing more of this in the future (imagine the uses for copy protection - let's hope we don't end up there).</p> <p>I'm not particularly happy that this is possible, but I'd rather talk about it than to hope that it's never going to happen - it will and <a href="/2011/09/asking-for-permission/">I'll be pissed</a>.</p> <p>For now I'm just wondering why Apple wasn't doing it to protect Siri.</p> </article> <article> <h1>A new fun project</h1> <p>2011-10-12T00:00:00+02:00</p> <p>Like <a href="http://blip.tv/jsconfeu/by-philip-hofstetter-node-js-in-production-use-tempalias-com-4258344">back in 2010</a> I went to JSConf.eu this year around.</p> <p>One of the many impressive facts about JSConf is the quality of their Wifi connection. It's not just free and stable, it's also fast. Not only that, this time around, they had a very cool feature: You authenticated via twitter.</p> <p>As most of the JS community seems to be having twitter accounts anyways, this was probably the most convenient solution for everyone: You didn't have to deal with creating an account or asking someone for a password and on the other hand, the organizers could make sure that, if abuse should happen, they'd know whom to notify.</p> <p>On a related note: This was in stark contrast to the WiFi I had in the hotel which was unstable, slow and cost a ton of money to use and it didn't use Twitter either :-)</p> <p>In fact, the twitter thing was so cool to see in practice, that I want to use it for myself too.</p> <p>Since the days of WEP-only Nintendo DS, I'm running two WiFi networks at home: One is WPA protected and for my own use, the other is open, but it runs over a different interface on <a href="/2006/07/computers-under-my-command-issue-1-shion/">shion</a> which has no access to any other machine in my network. This is even more important as <a href="/2005/05/lots-of-fun-with-openvpn/">I have a permanent OpenVPN connection</a> to my office and I definitely don't want to give the world access to that.</p> <p>So now the plan would be to change that open network so that it redirects to a captive portal until the user has authenticated with twitter (I might add other providers later on - LinkedIn would be <em>awesome</em> for the office for example).</p> <p>In order for me to actually get the thing going, I'm doing a tempalias on this one too and keep a diary of my work.</p> <p>So here we go. I really think that every year I should do some fun-project that's programming related, can be done on my own and is at least of some use. <a href="/tags/tempalias/">Last time it was tempalias</a>, this time, it'll be <em>Jocotoco</em> (more about the name in the next installment).</p> <p>But before we take off, let me give, again, huge thanks to the JSConf crew for the amazing conference they manage to organize year after year. If I could, I'd already preorder the tickets for next year :p</p> <p>Attending a JSConf feels like a two-day drug-trip that lasts for at least two weeks.</p> </article> <article> <h1>E_NOTICE stays off.</h1> <p>2011-10-06T00:00:00+02:00</p> <p>I'm sure you've used this idiom a lot when writing JavaScript code</p> <div class="highlight"><pre><code class="javascript"><span class="nx">options</span><span class="p">[</span><span class="s1">'a'</span><span class="p">]</span> <span class="o">=</span> <span class="nx">options</span><span class="p">[</span><span class="s1">'a'</span><span class="p">]</span> <span class="o">||</span> <span class="s1">'foobar'</span><span class="p">;</span> </code></pre> </div> <p>It's short, it's concise and it's clear what it does. In ruby, you can even be more concise:</p> <div class="highlight"><pre><code class="ruby"><span class="n">params</span><span class="o">[</span><span class="ss">:a</span><span class="o">]</span> <span class="o">||=</span> <span class="s1">'foobar'</span> </code></pre> </div> <p>So you can imagine that I was happy with PHP 5.3's new ?: operator:</p> <div class="highlight"><pre><code class="php"><span class="cp"><?</span> <span class="nv">$options</span><span class="p">[</span><span class="s1">'a'</span><span class="p">]</span> <span class="o">=</span> <span class="nv">$options</span><span class="p">[</span><span class="s1">'a'</span><span class="p">]</span> <span class="o">?:</span> <span class="s1">'foobar'</span><span class="p">;</span> </code></pre> </div> <p>In all three cases, the syntax is concise and readable, though arguably, the PHP one could read a bit better, but, ?: still is better than writing the full ternary expression, spelling out <code>$options['a']</code> three times.</p> <p><a href="http://www.popscan.com">PopScan</a>, since forever (forever being 2004) runs with E_NOTICE turned off. Back in the times, I felt it provided just baggage and I just wanted (had to) get things done quickly.</p> <p>This, of course, lead to people not taking enough care for the code and recently, I had one too many case of a bug caused by accessing a variable that was undefined in a specific code path.</p> <p>I decided that I'm willing to spend the effort in cleaning all of this up and making sure that there are no undeclared fields and variables in all of PopScans codebase.</p> <p>Which turned out to be quite a bit of work as a lot of code is apparently happily relying on the default <code>null</code> that you can read out of undefined variables. Those instances might be ugly, but they are by no means bugs.</p> <p>Cases where the <code>null</code> wouldn't be expected are the ones I care about, but I don't even what to go and discern the two - I'll just fix all of the instances (embarrassingly many, most of them, thankfully, not mine).</p> <p>Of course, if I put hours into a cleanup project like this, I want to be sure that nobody destroys my work again over time.</p> <p>Which is why I was looking into running PHP with <code>E_NOTICE</code> in development mode at least.</p> <p>Which brings us back to the introduction.</p> <div class="highlight"><pre><code class="php"><span class="cp"><?</span> <span class="nv">$options</span><span class="p">[</span><span class="s1">'a'</span><span class="p">]</span> <span class="o">=</span> <span class="nv">$options</span><span class="p">[</span><span class="s1">'a'</span><span class="p">]</span> <span class="o">?:</span> <span class="s1">'foobar'</span><span class="p">;</span> </code></pre> </div> <p>is wrong code. Any accessing of an undefined index of an array always raises a notice. It's not like Python where you can chose (accessing a dictionary using [] will throw a KeyError, but there's get() which just returns None). No. You don't get to chose. You only get to add boilerplate:</p> <div class="highlight"><pre><code class="php"><span class="cp"><?</span> <span class="nv">$options</span><span class="p">[</span><span class="s1">'a'</span><span class="p">]</span> <span class="o">=</span> <span class="nb">isset</span><span class="p">(</span><span class="nv">$options</span><span class="p">[</span><span class="s1">'a'</span><span class="p">])</span> <span class="o">?</span> <span class="nv">$options</span><span class="p">[</span><span class="s1">'a'</span><span class="p">]</span> <span class="o">:</span> <span class="s1">'foobar'</span><span class="p">;</span> </code></pre> </div> <p>See how I'm now spelling <code>$options['a']</code> three times again? <code>?:</code> just got a whole lot less useful.</p> <p>But not only that. Let's say you have code like this:</p> <div class="highlight"><pre><code class="php"><span class="cp"><?</span> <span class="k">list</span><span class="p">(</span><span class="nv">$host</span><span class="p">,</span> <span class="nv">$port</span><span class="p">)</span> <span class="o">=</span> <span class="nb">explode</span><span class="p">(</span><span class="s1">':'</span><span class="p">,</span> <span class="nb">trim</span><span class="p">(</span><span class="nv">$def</span><span class="p">))</span> <span class="nv">$port</span> <span class="o">=</span> <span class="nv">$port</span> <span class="o">?:</span> <span class="mi">11211</span><span class="p">;</span> </code></pre> </div> <p>IMHO very readable and clear what it does: It extracts a host and a port and sets the port to 11211 if there's none in the initial string.</p> <p>This of course won't work with E_NOTICE enabled. You either lose the very concise list() syntax, or you do - <em>ugh</em> - this:</p> <div class="highlight"><pre><code class="php"><span class="cp"><?</span> <span class="k">list</span><span class="p">(</span><span class="nv">$host</span><span class="p">,</span> <span class="nv">$port</span><span class="p">)</span> <span class="o">=</span> <span class="nb">explode</span><span class="p">(</span><span class="s1">':'</span><span class="p">,</span> <span class="nb">trim</span><span class="p">(</span><span class="nv">$def</span><span class="p">))</span> <span class="o">+</span> <span class="k">array</span><span class="p">(</span><span class="k">null</span><span class="p">,</span> <span class="k">null</span><span class="p">);</span> <span class="nv">$port</span> <span class="o">=</span> <span class="nv">$port</span> <span class="o">?:</span> <span class="mi">11211</span><span class="p">;</span> </code></pre> </div> <p>Which looks ugly as hell. And no, you can't write a wrapper to explode() which always returns an array big enough, because you don't know what's big enough. You would have to pass the amount of nulls you want into the call too. That would look nicer then above hack, but it still doesn't even come close in conciseness to the solution which throws a notice.</p> <p>So. In the end, I'm just complaining about syntax you might think? I though so too and I wanted to add the syntax I liked, so I did a bit of experimenting. Here's a little something I've come up with:</p> <script src="https://gist.github.com/1267568.js?file=e_notice_stays_off.php"></script> <p>The wrapped array solution looks really compelling syntax-wise and I could totally see myself using this and even forcing everybody else to go there. But of course, I didn't trust PHP's interpreter and thus benchmarked the thing.</p> <pre><code>pilif@tali ~ % php e_notice_stays_off.php Notices off. Array 100000 iterations took 0.118751s Notices off. Inline. Array 100000 iterations took 0.044247s Notices off. Var. Array 100000 iterations took 0.118603s Wrapped array. 100000 iterations took 0.962119s Parameter call. 100000 iterations took 0.406003s Undefined var. 100000 iterations took 0.194525s </code></pre> <p>So. Using nice syntactic sugar costs 7 times the performance. The second best solution? Still 4 times. Out of the question. Yes. It could be seen as a micro-optimization, but 100'000 iterations, while a lot is not <em>that many</em>. Waiting nearly a second instead of 0.1 second is crazy, especially for a common operation like this.</p> <p>Interestingly, the most bloated code (that checks with isset()) is twice as fast as the most readable (just assign). Likely, the notice gets fired regardless of error_reporting() and then just ignored later on.</p> <p>What really pisses me off about this is the fact that everywhere else PHP doesn't give a damn. '0' is equal to 0. Heck, even 'abc' is equal to 0. It even fails silently many times.</p> <p>But in a case like this, where there is even newly added nice and concise syntax, it has to be anal and bitchy. And there's no way to get to the needed solution but to either write too expensive wrappers or ugly boilerplate.</p> <p>Dynamic languages give us a very useful tool to be dynamic in the APIs we write. We can create functions that take a dictionary (an array in PHP) of options. We can extend our objects at runtime by just adding a property. And with PHP's (way too) lenient data conversion rules, we can even do math with user supplied string data.</p> <p>But can we read data from $_GET without boilerplate? No. Not in PHP. Can we use a dictionary of optional parameters? Not in PHP. PHP would require boilerplate.</p> <p>If a language basically mandates retyping the same expression three times, then, IMHO, something is broken. And if all the workarounds are either crappy to read or have very bad runtime properties, then something is terribly broken.</p> <p>So, I decided to just fix the problem (undefined variable access) but leave E_NOTICE where it is (off). There's always <code>git blame</code> and I'll make sure I will get a beer every time somebody lets another undefined variable slip in :-)</p> </article> <article> <h1>Asking for permission</h1> <p>2011-09-22T00:00:00+02:00</p> <p>Only just last year, I told <a href="https://twitter.com/brainlock">@brainlock</a> (in real life, so I can't link) that the coolest thing about our industry was that you don't have to ask for permission to do anything.</p> <p>Want to start the next big web project? Just start it. Want to write about your opinions? Just write about them. Want to get famous? It's still a lot of work and marketing, but nothing (aside of lack of talent) is stopping you.</p> <p>Whenever you have a good idea for a project, you start working on it, you see how it turns out and you decide whether to continue working on it or whether to scrap it. Aside of a bit of cash for hosting, you don't need anything else.</p> <p>This is very cool because is empowers "normal people". Heck, I probably wouldn't be where I currently am if it wasn't for this. Back in 1996 I had no money, I wasn't known, I had no past experience. What I had though was enthusiasm.</p> <p>Which is all that's needed.</p> <p>Only a year later though, I'm sad to see that we are at the verge of losing all of this. Piece by piece.</p> <p>First was apple with their iPhone. Even with all the enthusiasm of the world, you are not going to write an app that other people can run on the phone. No. First you will have to ask Apple for permission.</p> <p>Want to access some third-party hardware from that iPhone app? Sure. But now you have to not only ask Apple, but also the third party vendor for permission.</p> <p>The explanation we were given is that a malicious app could easily bring down the mobile network. Thus they needed to be careful what we could run on our phones.</p> <p>But then, we got the iPad with the exact same restrictions even though not all of them even have mobile network access.</p> <p>The explanation this time? Security.</p> <p>As nobody wants their machine to be insecure, everybody just accepts it.</p> <p>Next came Microsoft: In the Windows Mobile days before the release of 7, you didn't have to ask anybody for permission. You bought (or pirated if you didn't have money) Visual Studio, you wrote your app, you published it.</p> <p>All of this is lost now. Now you ask for permission. Now you hope for the powers that be to allow you to write your software.</p> <p>Finally, <a href="http://mjg59.dreamwidth.org/5552.html">you can't even do what you want with your PC</a> - all because of security.</p> <p>So there's still the web you think? I wish I could be positive about that, but as we are running out of IP-addresses and the adoption of IPv6 is slow as ever, I believe that public IP addresses are becoming a scarce good at which point, again, you will be asking for permission.</p> <p>In some countries, even today, it's not possible to just write a blog post because the government is afraid of "unrest" (read: losing even more credibility). That's not just countries we always perceived as "not free" - heck, <s>even in Italy you must register with the government if you want to have a blog</s> (it turns out that law didn't come to pass - let's hope no other country has the same bright idea). In Germany, if you read the law by the letter, you can't blog at all without getting every post approved - you could write something that a minor might see.</p> <p>«But permission will be granted anyways», you might say. Are you sure though? What if you are a minor wanting to create an application for your first client? Back in my days, I could just do it. Are you sure that whatever entity is going to have to give permission wan't to do business with minors? You <em>do</em> know that you can't have a Gmail account if you are younger than 13 years, do you? So age barriers exist.</p> <p>What if your project competes with whatever entity has to give permission? Remember the <a href="http://www.google.com/search?ie=UTF-8&q=google+voice+iphone+rejection">story about the Google Voice app</a>? Once we are out of IP addresses, the big provider and media companies who still have addresses might see you little startup web project as competition in some way. Are you sure you will still get permission?</p> <p>Back in 1996 when I started my company in High-School, all you needed to earn your living was enthusiasm and a PC (yes - I started doing web programming without having access to the internet)</p> <p>Now you need signed contracts, signed NDAs, lobbying, developer program memberships, cash - the barriers to entry are infinitely higher at this point.</p> <p>I'm afraid though, that this is just the beginning. If we don't stand up now, if we continue to let big companies and governments take away our freedom of expression piece by piece, if we give up more and more of our freedom because of the false promise of security, then, at one point, all of what we had will be lost.</p> <p>We won't be able to just start our projects. We won't be able to create - only to work on other peoples projects. We will lose all that makes our profession interesting.</p> <p>Let's not go there.</p> <p>Please.</p> <p><a href="https://news.ycombinator.com/item?id=3025245">Discussion on HackerNews</a></p> </article> <article> <h1>Lion Server authentication issues</h1> <p>2011-09-19T00:00:00+02:00</p> <p>Lately I was having an issue with a Lion Server that refused logins of users stored in OpenDirectory. A quick check of <code>/var/log/opendirectoryd.log</code> revealed an issue with the «Password Server»:</p> <pre><code>Module: AppleODClient - unable to send command to Password Server - sendmsg() on socket fd 16 failed: Broken pipe (5205) </code></pre> <p>As this message apparently doesn't appear on Google yet, there's my contribution to solving this.</p> <p>The fix was to kill -9 the kerberos authentication daemon:</p> <pre><code>sudo killall kpasswdd </code></pre> <p>which in fact didn't help (sometimes <a href="http://xkcd.com/149/">even sudo isn't enough</a>), so I had to be more persuasive to get rid of the apparently badly hanging process:</p> <pre><code>sudo killall -9 kpasswdd </code></pre> <p>This time the process was really killed and subsequently instantly restarted by launchd.</p> <p>After that, the problem went away.</p> </article> <article> <h1>serialize() output is binary data!</h1> <p>2011-09-15T00:00:00+02:00</p> <p>When you call <a href="http://www.php.net/serialize">serialize()</a> in PHP, to serialize a value into something that you store for later use with <a href="http://www.php.net/unserialize">unserialize()</a>, then be very careful what you are doing with that data.</p> <p>When you look at the output, you'd be tempted to assume that it's text data:</p> <pre><code>php > $a = array('foo' => 'bar'); php > echo serialize($a); a:1:{s:3:"foo";s:3:"bar";} php > </code></pre> <p>and as such, you'd be tempted to treat this as text data (i.e. store it in a TEXT column in your database).</p> <p>But what looks like text on first glance isn't text data at all. Assume that my terminal is in ISO-8859-1 encoding:</p> <pre><code>php > echo serialize(array('foo' => 'bär')); a:1:{s:3:"foo";s:3:"bär";} </code></pre> <p>and now assume it's in UTF-8 encoding:</p> <pre><code>php > echo serialize(array('foo' => 'bär')); a:1:{s:3:"foo";s:4:"bär";} </code></pre> <p>You will notice that the format encodes the strings length together with the string. And because PHP is inherently not unicode capable, it's not encoding the strings character length, but its <em>byte-length</em>.</p> <p>unserialize() checks whether the encoded length matches the actual delimited strings length. This means that if you treat the serialized output as text and your databases's encoding changes along the way, that the retrieved string can't be unserialized any more.</p> <p>I just learned that the hard way (even though it's obvious in hindsight) while migrating <a href="http://www.popscan.ch">PopScan</a> from ISO-8859-1 to UTF-8:</p> <p>The databases of existing systems now contain a lot of output from serialize() which was run over ISO strings but now that the client-encoding in the database client is set to utf-8, the data will be retrieved as UTF-8 and because the serialize() output was stored in a TEXT column, it happily gets UTF-8 encoded.</p> <p>If we remove the database from the picture and express the problem in code, this is what's going on:</p> <pre><code>unserialize(utf8encode(serialize('data with 8bit chàracters'))); </code></pre> <p>i.e the data gets altered after serializing and the way it gets altered is a way that unserialize can't deal with the data any more.</p> <p>So, for everybody else not yet in this dead end:</p> <p>The output of serialize() is <em>binary data</em>. It looks like textual data, bit it isn't. Treat it as binary. If you store it somewhere, make sure that the medium you store it to treats the data as binary. No transformation what so ever must ever be made on it.</p> <p>Of course, that leaves you with a problem later on if you switch character sets and you have to unserialize, but at least you get to unserialize then. I have to go great lengths now to salvage the old data.</p> </article> <article> <h1>Another platform change</h1> <p>2011-08-03T00:00:00+02:00</p> <p>If you can read this, then it has happened - this blog moved again.</p> <p>Eons ago in internet time, <a href="/2002/11/welcome/">this project has started</a> wich lasted for 4 years at which point I got spammed so badly that I had to move away.</p> <p>So, still ages ago, <a href="/2006/06/new-face-new-engine-new-everything/">I moved to Serendipity</a> which fixed the spam issue for me.</p> <p>This lastes only two years before <a href="/2008/03/another-new-look/">I moved again</a>to WordPress this time - the nicer admin tool and the richer theme selection pushed me over the edge.</p> <p>While I'm still happy with WordPress in general, over time I learned a few things:</p> <ul> <li><p>while running your own server is fun, having it compromised is not. Using any well-known blogging engine that relies on server-side generation of<br/> content is ultimately a way to get compromised unless you constantly patch security issues, taking up a lot of time in the process.</p></li> <li><p>While the old name of this blog (gnegg) was a cool pun for people who knew me, it didn't at all convey my identity on the internet. Me? I'm <a href="http://pilif.me">pilif</a>, so this should be conveyed at least by the URL</p></li> <li><p>Most of my posts very relying heavily on custom markup, making the WP WYSIWYG editor more annoying than useful.</p></li> </ul> <p>So when <a href="https://twitter.com/rmurphey">@rmurphey</a> <a href="http://rmurphey.com/blog/2011/07/25/switching-to-octopress/">blogged about octopress</a> I immediately recognized the huge opportunity <a href="http://octopress.org">Octopress</a> provides:</p> <p>I can host static files on a server I don't own and thus don't have to care about compromising, I can blog using my favorite tools (any text editor and git) and I still get an acceptable layout.</p> <p>So here we are - at the end of yet another conversion.</p> <p>While the old URLs already 301 redirect, pictures are still missing and I'll work on getting them back. The comments I'll try to port over too the moment I see how disqus handles my WordPress export.</p> <p>The gnegg branding is gone and has been replaced by something that doesn't look like a name but isn't while still being a fun pun for people who know me. The tagline, of course, stays the same.</p> <p>So.</p> <p>Welcome to my new home and let's hope this lasts as long as the previous instances!</p> </article> <article> <h1>AJAX, Architecture, Frameworks and Hacks</h1> <p>2011-04-13T00:00:00+02:00</p> <p>Today I was talking with <a href="http://twitter.com/brainlock">@brainlock</a> about JavaScript, AJAX and Frameworks and about two paradigms that are in use today:</p> <p>The first is the "traditional" paradigm where your JS code is just glorified view code. This is how AJAX worked in the early days and how people are still using it. Your JS-code intercepts a click somewhere, sends an AJAX request to the server and gets back either more JS code which just gets evaulated (thus giving the server kind of indirect access to the client DOM) or a HTML fragment which gets inserted at the appropriate spot.</p> <p>This means that<em> your JS code will be ugly</em> (especially the code coming from the server), but it has the advantage that all your view code is right there where all your controllers and your models are: on the server. You see this pattern in use on the 37signals pages or in the <a href="http://github.com">github</a> file browser for example.</p> <p>Keep the file browser in mind as I'm going to use that for an example later on.</p> <p>The other paradigm is to go the other way around an promote JS to a first-class language. Now you build a framework on the client end and transmit only data (XML or JSON, but mostly JSON these days) from the server to the client. The server just provides a REST API for the data plus serves static HTML files. All the view logic lives only on the client side.</p> <p>The advantages are that you can organize your client side code much better, for example using <a href="http://documentcloud.github.com/backbone/">backbone</a>, that there's no expensive view rendering on the server side and that you basically get your third party API for free because the API is the only thing the server provides.</p> <p>This paradigm is used for the new twitter webpage or in my very own <a href="http://tempalias.com">tempalias.com</a>.</p> <p>Now <a href="http://twitter.com/brainlock">@brainlock</a> is a heavy proponent of the second paradigm. After being enlightened by the great Crockford, we both love JS and we both worked on huge messes of client-side JS code which has grown over the years and lacks structure and feels like copy pasta sometimes. In our defense: Tons of that code was written in the pre-enlightened age (2004).</p> <p>I on the other hand see some justification for the first pattern aswell and I wouldn't throw it away so quickly.</p> <p>The main reason: It's more pragmatic, it's more DRY once you need graceful degradation and arguably, you can reach your goal a bit faster.</p> <p>Let me explain by looking at the github file browser:</p> <p>If you have a browser that supoports the HTML5 history API, then a click on a directory will reload the file list via AJAX and at the same time the URL will be updated using push state (so that the current view keeps its absolute URL which is valid even after you open it in a new browser).</p> <p>If a browser doesn't support pushState, it will gracefully degrade by just using the traditional link (and reloading the full page).</p> <p>Let's map this functionality to the two paradigms.</p> <p>First the hacky one:</p> <ol> <li>You render the full page with the file list using a server-side template</li> <li>You intercept clicks to the file list. If it's a folder:</li> <li>you request the new file list</li> <li>the server now renders the file list partial (in rails terms - basically just the file list part) without the rest of the site</li> <li>the client gets that HTML code and inserts it in place of the current file list</li> <li>You patch up the url using push state</li> </ol> <p>done. The view code is only on the server. Whether the file list is requested using the AJAX call or the traditional full page load doesn't matter. The code path is exactly the same. The only difference is that the rest of the page isn't rendered in case of an AJAX call. You get graceful degradation and no additional work.</p> <p>Now assuming you want to keep graceful degradation possible and you want to go the JS framework route:</p> <ol> <li>You render the full page with the file list using a server-side template</li> <li>You intercept the click to the folder in the file list</li> <li>You request the JSON representation of the target folder</li> <li>You use that JSON representation to fill a client-side template which is a copy of the server side partial</li> <li>You insert that HTML at the place where the file list is</li> <li>You patch up the URL using push state</li> </ol> <p>The amount of steps is the same, but the amount of work isn't: If you want graceful degradation, then you write the file list template twice: Once as a server-side template, once as a client-side template. Both are quite similar but usually you'll be forced to use slightly different syntax. If you update one, you have to update the other or the experience will be different whether you click on a link or you open the URL directly.</p> <p>Also you are duplicating the code which fills that template: On the server side, you use ActiveRecord or whatever other ORM. On the client side, you'd probably use Backbone to do the same thing but now your backend isn't the database but the JSON response. Now, Backbone is really cool and a huge timesaver, but it's still more work than not doing it at all.</p> <p>OK. Then let's skip graceful degradation and make this a JS only client app (<a href="http://www.google.com/search?ie=UTF-8&q=gawker+redesign">good luck trying to get away with that</a>). Now the view code on the server goes away and you are just left with the model on the server to retrieve the data, with the model on the client (Backbone helps a lot here, but there's still a substatial amount of code that needs to be written that otherwise wouldn't) and with the view code on the client.</p> <p>Now don't ge me wrong.</p> <p>I <strong>love</strong> the idea of promoting JS to a first class language. I <strong>love</strong> JS frameworks for big JS only applications. I <strong>love</strong> having a "free", dogfooded-by-design REST API. I <strong>love</strong> building cool architectures.</p> <p>I'm just thinking that at this point it's so much work doing it right, that the old ways do have their advantages and that we should not condemn them for being hacky. True. They are. But they are also <em>pragmatic</em>.</p> </article> <article> <h1>DNSSEC to fix the SSL mess?</h1> <p>2011-04-07T00:00:00+02:00</p> <p>After <a href="http://codebutler.com/firesheep">Firesheep</a> it has become clear that there's no way around SSL.</p> <p>But still many people (and I'm including myself) are unhappy with the fact that to roll out SSL, you basically have to pay a sometimes significant premium for the certificate. And that's not all: You have to pay the same fee every n years (and while you could say that the CA does some work the first time, every following year, it's plain sucking money from you) and you have to remember to actually do it unless you want <a href="http://forum.skype.com/index.php?showtopic=784971">embarrassing warnings</a> pop up to your users.</p> <p>The usual suggestion is to make browsers accept self-signed certificates without complaining, but that doesn't really work to prevent a Firesheep style attack and is arguably even worse as it would allow not only your session id, but also your password to leak from sites that use the traditional SSL-for-login-HTTP-afterwards mechanism.</p> <p>See <a href="http://news.ycombinator.com/item?id=2348836">my comment on HackerNews</a> for more details.</p> <p>To make matters worse, last week news about a CA being compromised and issuing fraudulent (but still trusted) certificates made the rounds, so now even with the current CA based security mechanism, we still can't completely trust the infrastructure.</p> <p>Thinking about this, I had an idea.</p> <p>Let's assume that one day, one glorious day, DNSSEC will actually be deployed.</p> <p>If that's the case, then if I was the owner of gnegg.ch, I could just publish the certificate (or its fingerprint or a link to the certificate over SSL) in the DNS as a TXT record. DNSSEC would ensure that it was the owner of the domain who created the TXT entry and that the domain is the real one and not a faked one.</p> <p>So if that entry says that gnegg.ch is supposed to serve a certificate with the fingerprint 0xdeadbeef, then a connecting browser would be sure that if the site is serving that certificate (and has the matching private key), then the connection would be secure and not man-in-the-middle'd.</p> <p>Even better: If I lose the private key of gnegg.ch, I would just update the TXT record, making the old key useless. No non-working CRL or OCSP. Just one additional DNS query.</p> <p>And you know what? It would put CAs out of business for signing of site certificates as a self-signed certificate would be as good as an official one (they would still be needed to sign your DNSSEC zone file of course, but that could be done by the TLD owners).</p> <p>Oh and by the way: I could create my certificate with an incredibly long (if ever) expiration time: If I want the certificate to be invalid, I remove or change the TXT record and I'm done. As simple as that. No more embarrassing warnings. No more fear of missing the deadline.</p> <p>Now, this feels so incredibly simple that there <strong>must</strong> be something I'm missing. What is it? Is it just that politics is preventing DNSSEC from ever being real? Is there an error in my thinking?</p> <p> </p> </article> <article> <h1>rails, PostgreSQL and the native uuid type</h1> <p>2011-03-07T00:00:00+01:00</p> <p>UUID have the very handy property that they are uniqe and there are quite many of them for you to use. Also they are difficult to guess and knowing the UUID of one object, it's very hard to guess a valid UUID of another object.</p> <p>This makes UUIDs perfect for identifying things in web applications:</p> <ul> <li>Even if you shard across multiple machines, each machine can independently generate primary keys without (realistic) fear of overlapping.</li> <li>You can generate them without using any kind of locks.</li> <li>Sometimes, you have to expose such keys to the user. If possible, you will of course do authorization checks, but it still makes sense not allowing users know about neighboring keysThis gets even more important when you are not able to do authorization keys because the resource you are referring to is public (like a <a href="http://tempalias.com">mail alias</a>) but it should still not possible to know other items if you know one.</li> </ul> <p>Knowing that <a href="http://www.codinghorror.com/blog/2007/03/primary-keys-ids-versus-guids.html">UUIDs are a good thing</a>, you might want to use them in your application (or you just have to in the last case above).</p> <p>There are multiple recipes out there that show how to do it in a rails application (<a href="http://stackoverflow.com/questions/2487837/uuids-in-rails3">this one for example</a>).</p> <p>All of these recipes store UUIDs as varchar's in your database. In general, that's fine and also the only thing you can do as most databases don't have a native data type for UUIDs.</p> <p><a href="http://www.postgresql.org">PostgreSQL</a> on the other hand indeed has a native 128 bit integer type to store UUID.</p> <p>This is more space efficient than storing the UUID in string form (288 bit) and it might be a tad bit faster when doing comparison operations on the database as integer operations (even if they are this big) require a constant amount of operations whereas comparing two string UUIDs is a string comparison which is dependent on the string size and size of the matching parts.</p> <p>So maybe for the (minuscule) speed increase or for the purpose of correct semantics or just for interoperability with other applications, you might want to use native PostgreSQL UUIDs from your Rails (or other, but without the abstraction of a "Migration", just using UUID is trivial) applications.</p> <p>This already works quite nicely if you generate the columns as strings in your migrations and then manually send an <code>alter table</code> (whenever you restore the schema from scratch).</p> <p>But if you want to create the column with the correct type directly from the migration and you want the column to be created correctly when using <code>rake db:schema:load</code>, then you need a bit of additional magic, especially if you want to still support other databases.</p> <p>In my case, I was using PostgreSQL in production (<a href="http://www.gnegg.ch/2004/06/all-time-favourite-tools/">what</a> <a href="http://www.gnegg.ch/2009/02/all-time-favourite-tools-update/">else</a>?), but on my local machine, for the purpose of getting started quickly, I wanted to still be able to use SQLite for development.</p> <p>In the end, everything boils down to monkey patching ActiveRecord::ConnectionAdapters::<em>Adapters and PostgreSQLColumn of the same module. So here's what I've addded to <code>config/initializers/uuuid_support.rb</code> (Rails 3.0.</em>):</p> <div class="highlight"><pre><code class="ruby"><span class="k">module</span> <span class="nn">ActiveRecord</span> <span class="k">module</span> <span class="nn">ConnectionAdapters</span> <span class="no">SQLiteAdapter</span><span class="o">.</span><span class="n">class_eval</span> <span class="k">do</span> <span class="k">def</span> <span class="nf">native_database_types_with_uuid_support</span> <span class="n">a</span> <span class="o">=</span> <span class="n">native_database_types_without_uuid_support</span> <span class="n">a</span><span class="o">[</span><span class="ss">:uuid</span><span class="o">]</span> <span class="o">=</span> <span class="p">{</span><span class="ss">:name</span> <span class="o">=></span> <span class="s1">'varchar'</span><span class="p">,</span> <span class="ss">:limit</span> <span class="o">=></span> <span class="mi">36</span><span class="p">}</span> <span class="k">return</span> <span class="n">a</span> <span class="k">end</span> <span class="n">alias_method_chain</span> <span class="ss">:native_database_types</span><span class="p">,</span> <span class="ss">:uuid_support</span> <span class="k">end</span> <span class="k">if</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">adapter_name</span> <span class="o">==</span> <span class="s1">'SQLite'</span> <span class="k">if</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">adapter_name</span> <span class="o">==</span> <span class="s1">'PostgreSQL'</span> <span class="no">PostgreSQLAdapter</span><span class="o">.</span><span class="n">class_eval</span> <span class="k">do</span> <span class="k">def</span> <span class="nf">native_database_types_with_uuid_support</span> <span class="n">a</span> <span class="o">=</span> <span class="n">native_database_types_without_uuid_support</span> <span class="n">a</span><span class="o">[</span><span class="ss">:uuid</span><span class="o">]</span> <span class="o">=</span> <span class="p">{</span><span class="ss">:name</span> <span class="o">=></span> <span class="s1">'uuid'</span><span class="p">}</span> <span class="k">return</span> <span class="n">a</span> <span class="k">end</span> <span class="n">alias_method_chain</span> <span class="ss">:native_database_types</span><span class="p">,</span> <span class="ss">:uuid_support</span> <span class="k">end</span> <span class="no">PostgreSQLColumn</span><span class="o">.</span><span class="n">class_eval</span> <span class="k">do</span> <span class="k">def</span> <span class="nf">simplified_type_with_uuid_support</span><span class="p">(</span><span class="n">field_type</span><span class="p">)</span> <span class="k">if</span> <span class="n">field_type</span> <span class="o">==</span> <span class="s1">'uuid'</span> <span class="ss">:uuid</span> <span class="k">else</span> <span class="n">simplified_type_without_uuid_support</span><span class="p">(</span><span class="n">field_type</span><span class="p">)</span> <span class="k">end</span> <span class="k">end</span> <span class="n">alias_method_chain</span> <span class="ss">:simplified_type</span><span class="p">,</span> <span class="ss">:uuid_support</span> <span class="k">end</span> <span class="k">end</span> <span class="k">end</span> <span class="k">end</span> </code></pre> </div> <p>In your migrations you can then use the :uuid type. In my sample case, this was it:</p> <div class="highlight"><pre><code class="ruby"><span class="k">class</span> <span class="nc">AddGuuidToSites</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Migration</span> <span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">up</span> <span class="n">add_column</span> <span class="ss">:sites</span><span class="p">,</span> <span class="ss">:guuid</span><span class="p">,</span> <span class="ss">:uuid</span> <span class="n">add_index</span> <span class="ss">:sites</span><span class="p">,</span> <span class="ss">:guuid</span> <span class="k">end</span> <span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">down</span> <span class="n">remove_column</span> <span class="ss">:sites</span><span class="p">,</span> <span class="ss">:guuid</span> <span class="k">end</span> <span class="k">end</span> </code></pre> </div> <p>Maybe with a bit better Ruby knowledge than I have, it should be possible to just monkey-patch the parent <code>AbstractAdaper</code> while still calling the method of the current subclass. This would not require a separate patch for all adapters in use.</p> <p>For my case which was just support for SQLite and PostgreSQL, the above initializer was fine though.</p> </article> <article> <h1>How I back up gmail</h1> <p>2011-02-28T00:00:00+01:00</p> <p>There was a <a href="http://news.ycombinator.com/item?id=2269346">discussion on HackerNews</a> about Gmail having lost the email in some accounts. One sentiment in the comments was clear:</p> <p>It's totally the users problem if they don't back up their cloud based email.</p> <p>Personally, I think I would have to agree:</p> <p>Google is a provider like every other ISP or basically any other service too. There's no reason to believe that your data is more save on Google than it is any where else. Now granted, they are not exactly known for losing data, but there's other things that can happen.</p> <p>Like your account being closed because whatever automated system believed your usage patterns were consistent with those of a spammer.</p> <p>So the question is: What would happen if your Google account wasn't reachable at some point in the future?</p> <p>For my company (using commercial Google Apps accounts), I would start up that IMAP server which serves all mail ever sent to and from Gmail. People would use the already existing webmail client or their traditional IMAP clients. They would lose some productivity, but no single byte of data.</p> <p>This was my condition for migrating email over to Google. I needed to have a back up copy of that data. Otherwise, I would not have agreed to switch to a cloud based provider.</p> <p>The process is completely automated too. There's not even a backup script running somewhere. Heck, <strong>not even the Google Account passwords have to be stored anywhere for this to work</strong>.</p> <p>So. How does it work then?</p> <p>Before you read on, here are the drawbacks of the solution:</p> <ul> <li>I'm a die-hard <a href="http://exim.org/">Exim</a> fan (long story. It served me very well once - up to saving-my-ass level of well), so the configuration I'm outlining here is for Exim as the mail relay.</li> <li>Also, this <strong>only works with paid Google accounts</strong>. You can get somewhere using the free ones, but you don't get the full solution (i.e. having a backup of all sent email)</li> <li>This requires you to have full control over the MX machine(s) of your domain.</li> </ul> <p>If you can live with this, here's how you do it:</p> <p>First, you set up your Google domain as normal. Add all the users you want and do everything else just as you would do it in a traditional set up.</p> <p>Next, we'll have to configure Google Maps for <a href="http://www.gnegg.ch/2010/06/google-apps-provisioning-two-legged-oauth/">two-legged OAuth access</a> to our accounts. I've written about this <a href="http://www.gnegg.ch/2010/06/google-apps-provisioning-two-legged-oauth/">before</a>. We are doing this so we don't need to know our users passwords. Also, we need to enable the provisioning API to get access to the list of users and groups.</p> <p>Next, our mail relay will have to know about what users (and groups) are listed in our Google account. Here's what I quickly hacked together in Python (my first Python script ever - be polite while flaming) using the GData library:</p> <div class="highlight"><pre><code class="python"><span class="kn">import</span> <span class="nn">gdata.apps.service</span> <span class="n">consumer_key</span> <span class="o">=</span> <span class="s">'yourdomain.com'</span> <span class="n">consumer_secret</span> <span class="o">=</span> <span class="s">'2-legged-consumer-secret'</span> <span class="c">#see above</span> <span class="n">sig_method</span> <span class="o">=</span> <span class="n">gdata</span><span class="o">.</span><span class="n">auth</span><span class="o">.</span><span class="n">OAuthSignatureMethod</span><span class="o">.</span><span class="n">HMAC_SHA1</span> <span class="n">service</span> <span class="o">=</span> <span class="n">gdata</span><span class="o">.</span><span class="n">apps</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">AppsService</span><span class="p">(</span><span class="n">domain</span><span class="o">=</span><span class="n">consumer_key</span><span class="p">)</span> <span class="n">service</span><span class="o">.</span><span class="n">SetOAuthInputParameters</span><span class="p">(</span><span class="n">sig_method</span><span class="p">,</span> <span class="n">consumer_key</span><span class="p">,</span>\ <span class="n">consumer_secret</span><span class="o">=</span><span class="n">consumer_secret</span><span class="p">,</span> <span class="n">two_legged_oauth</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="n">res</span> <span class="o">=</span> <span class="n">service</span><span class="o">.</span><span class="n">RetrieveAllUsers</span><span class="p">()</span> <span class="k">for</span> <span class="n">entry</span> <span class="ow">in</span> <span class="n">res</span><span class="o">.</span><span class="n">entry</span><span class="p">:</span> <span class="k">print</span> <span class="n">entry</span><span class="o">.</span><span class="n">login</span><span class="o">.</span><span class="n">user_name</span> <span class="kn">import</span> <span class="nn">gdata.apps.groups.service</span> <span class="n">service</span> <span class="o">=</span> <span class="n">gdata</span><span class="o">.</span><span class="n">apps</span><span class="o">.</span><span class="n">groups</span><span class="o">.</span><span class="n">service</span><span class="o">.</span><span class="n">GroupsService</span><span class="p">(</span><span class="n">domain</span><span class="o">=</span><span class="n">consumer_key</span><span class="p">)</span> <span class="n">service</span><span class="o">.</span><span class="n">SetOAuthInputParameters</span><span class="p">(</span><span class="n">sig_method</span><span class="p">,</span> <span class="n">consumer_key</span><span class="p">,</span>\ <span class="n">consumer_secret</span><span class="o">=</span><span class="n">consumer_secret</span><span class="p">,</span> <span class="n">two_legged_oauth</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="n">res</span> <span class="o">=</span> <span class="n">service</span><span class="o">.</span><span class="n">RetrieveAllGroups</span><span class="p">()</span> <span class="k">for</span> <span class="n">entry</span> <span class="ow">in</span> <span class="n">res</span><span class="p">:</span> <span class="k">print</span> <span class="n">entry</span><span class="p">[</span><span class="s">'groupName'</span><span class="p">]</span> </code></pre> </div> <p>Place this script somewhere on your mail relay and run it in a cron job. In my case, I'm having its output redirected to <code>/etc/exim4/gmail_accounts</code>. The script will emit one user (and group) name per line.</p> <p>Next, we'll deal with incoming email:</p> <p>In the Exim configuration of your mail relay, add the following routers:</p> <div class="highlight"><pre><code class="text">yourdomain_gmail_users: driver = accept domains = yourdomain.com local_parts = lsearch;/etc/exim4/gmail_accounts transport_home_directory = /var/mail/yourdomain/${lc:$local_part} router_home_directory = /var/mail/yourdomain/${lc:$local_part} transport = gmail_local_delivery unseen yourdomain_gmail_remote: driver = accept domains = yourdomain.com local_parts = lsearch;/etc/exim4/gmail_accounts transport = gmail_t </code></pre> </div> <p>yourdomain_gmail_users is what creates the local copy. It accepts all mail sent to yourdomain.com, if the local part (the stuff in front of the @) is listed in that gmail_accounts file. Then it sets up some paths for the local transport (see below) and marks the mail as unseen so the next router gets a chance too.</p> <p>Which is yourdomain_gmail_remote. This one is again checking domain and the local part and if they match, it's just delegating to the gmail_t remote transport (which will then send the email to Google).</p> <p>The transports look like this:</p> <div class="highlight"><pre><code class="text">gmail_t: driver = smtp hosts = aspmx.l.google.com:alt1.aspmx.l.google.com:\ alt2.aspmx.l.google.com:aspmx5.googlemail.com:\ aspmx2.googlemail.com:aspmx3.googlemail.com:\ aspmx4.googlemail.com gethostbyname gmail_local_delivery: driver = appendfile check_string = delivery_date_add envelope_to_add group=mail maildir_format directory = MAILDIR/yourdomain/${lc:$local_part} maildir_tag = ,S=$message_size message_prefix = message_suffix = return_path_add user = Debian-exim create_file = anywhere create_directory </code></pre> </div> <p>the gmail_t transport is simple. The local one you might have to patch up users and groups plus the location where you what to write the mail to.</p> <p>Now we are ready to reconfigure Google as this is all that's needed to get a copy of every inbound mail into a local maildir on the mail relay.</p> <p>Here's what you do:</p> <ul> <li>You change the MX of your domain to point to this relay of yours</li> </ul> <p>The next two steps are the reason you need a paid account: These controls are not available for the free accounts:</p> <ul> <li>In your Google Administration panel, you visit the Email settings and configure the outbound gateway. Set it to your relay.</li> <li>Then you configure your inbound gateway and set it to your relay too (and to your backup MX if you have one).</li> </ul> <p>This screenshot will help you:</p> <p><a href="http://www.gnegg.ch/wp-content/uploads/2011/02/gmail-config.png"></a>All email sent to your MX (over the gmail_t transport we have configured above) will now be accepted by gmail.</p> <p>Also, Gmail will now send all outgoing Email to your relay which needs to be configured to accept (and relay) email from Google. This pretty much depends on your otherwise existing Exim configuration, but here's what I added (which will work with the default ACL):</p> <div class="highlight"><pre><code class="text">hostlist google_relays = 216.239.32.0/19:64.233.160.0/19:66.249.80.0/20:\ 72.14.192.0/18:209.85.128.0/17:66.102.0.0/20:\ 74.125.0.0/16:64.18.0.0/20:207.126.144.0/20 hostlist relay_from_hosts = 127.0.0.1:+google_relays </code></pre> </div> <p>And lastly, the tricky part: Storing a copy of all mail that is being sent through Gmail (we are already correctly sending the mail. What we want is a copy):</p> <p>Here is the exim router we need:</p> <div class="highlight"><pre><code class="text">gmail_outgoing: driver = accept condition = "${if and{\ { eq{$sender_address_domain}{yourdomain.com} }\ {=={${lookup{$sender_address_local_part}lsearch{/etc/exim4/gmail_accounts}{1}}}{1}}} {1}{0}}" transport = store_outgoing_copy unseen </code></pre> </div> <p>(did I mention that I severely dislike RPN?)</p> <p>and here's the transport:</p> <div class="highlight"><pre><code class="text">store_outgoing_copy: driver = appendfile check_string = delivery_date_add envelope_to_add group=mail maildir_format directory = MAILDIR/yourdomain/${lc:$sender_address_local_part}/.Sent/ maildir_tag = ,S=$message_size message_prefix = message_suffix = return_path_add user = Debian-exim create_file = anywhere create_directory </code></pre> </div> <p>The maildir I've chosen is the correct one if the IMAP-server you want to use is Courier IMAPd. Other servers use different methods.</p> <p>One little thing: When you CC or BCC other people in your domain, Google will send out multiple copies of the same message. This will yield some message duplication in the sent directory (one per recipient), but as they say: Better backup too much than too little.</p> <p>Now if something happens to your google account, just start up an IMAP server and have it serve mail from these maildir directories.</p> <p>And remember to back them up too, but you can just use rsync or rsnapshot or whatever other technology you might have in use. They are just directories containing one file per email.</p> </article> <article> <h1>sacy 0.2 - now with less, sass and scss</h1> <p>2011-02-16T00:00:00+01:00</p> <p>To fresh up your memory (<a href="/2009/09/introducing-sacy-the-smarty-asset-compiler/">it has been a while</a>): <a href="http://github.com/pilif/sacy">sacy</a> is a <a href="http://www.smarty.net">Smarty</a> (both 2 and 3) plugin that turns</p> <div class="highlight"><pre><code class="xml">{asset_compile} <span class="nt"><link</span> <span class="na">type=</span><span class="s">"text/css"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/styles/file1.css"</span> <span class="nt">/></span> <span class="nt"><link</span> <span class="na">type=</span><span class="s">"text/css"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/styles/file2.css"</span> <span class="nt">/></span> <span class="nt"><link</span> <span class="na">type=</span><span class="s">"text/css"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/styles/file3.css"</span> <span class="nt">/></span> <span class="nt"><link</span> <span class="na">type=</span><span class="s">"text/css"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/styles/file4.css"</span> <span class="nt">/></span> <span class="nt"><script</span> <span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"/jslib/file1.js"</span><span class="nt">></script></span> <span class="nt"><script</span> <span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"/jslib/file2.js"</span><span class="nt">></script></span> <span class="nt"><script</span> <span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"/jslib/file3.js"</span><span class="nt">></script></span> {/asset_compile} </code></pre> </div> <p>into</p> <div class="highlight"><pre><code class="xml"><span class="nt"><link</span> <span class="na">type=</span><span class="s">"text/css"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/assets/files-1234abc.css"</span> <span class="nt">/></span> <span class="nt"><script</span> <span class="na">type=</span><span class="s">"text/javascript"</span> <span class="na">src=</span><span class="s">"/assets/files-abc123.js"</span><span class="nt">></script></span> </code></pre> </div> <p>It does this without you ever having to manually run a compiler, without serving all your assets through some script (thus saving RAM) and without worries about stale copies being served. In fact, you can serve all static files generated with sacy with cache headers telling browsers to never revisit them!</p> <p>All of this, using two lines of code (wrap as much content as you want in {asset_compile}...{/asset_compile})</p> <p>Sacy has been around for a bit more than a year now and has since been in production use in <a href="http://www.popscan.com">PopScan</a>. During this time, no single bug in Sacy has been found, so I would say that it's pretty usable.</p> <p>Coworkers have bugged me enough about how much better <a href="http://lesscss.org/">less</a> or <a href="http://sass-lang.com/">sass</a> would be compared to pure CSS so that I finally decided to update <a href="http://github.com/pilif/sacy">sacy</a> to allow us to use less in PopScan:</p> <p>Aside of consolidating and minimizing CSS and JavaScript, sacy can now also transform less and sass (or scss) files using the exact same method as before but just changing the mime-type:</p> <div class="highlight"><pre><code class="xml"><span class="nt"><link</span> <span class="na">type=</span><span class="s">"text/x-less"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/styles/file1.less"</span> <span class="nt">/></span> <span class="nt"><link</span> <span class="na">type=</span><span class="s">"text/x-sass"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/styles/file2.sass"</span> <span class="nt">/></span> <span class="nt"><link</span> <span class="na">type=</span><span class="s">"text/x-scss"</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"/styles/file3.scss"</span> <span class="nt">/></span> </code></pre> </div> <p>Like before, you don't concern yourself with manual compilation or anything. Just use the links as is and sacy will do the magic for you.</p> <p>Interested? Read the (by now huge) <a href="https://github.com/pilif/sacy/blob/v0.2/README.markdown">documentation</a> on <a href="http://github.com/pilif">my github page</a>!</p> </article> <article> <h1>Find relation sizes in PostgreSQL</h1> <p>2011-02-07T00:00:00+01:00</p> <p>Like so many times before, today I was yet again in the situation where I wanted to know which tables/indexes take the most disk space in a particular PostgreSQL database.</p> <p>My usual procedure in this case was to <code>\dt+</code> in psql and scan the sizes by eye (this being on my development machine, trying to find out the biggest tables I could clean out to make room).</p> <p>But once you've done that a few times and considering that <code>\dt+</code> does nothing but query some PostgreSQL internal tables, I thought that I want this solved in an easier way that also is less error prone. In the end I just wanted the output of \dt+ sorted by size.</p> <p>The lead to some digging in the source code of psql itself (<code>src/bin/psql</code>) where I quickly found the function that builds the query (<code>listTables</code> in <code>describe.c</code>), so from now on, this is what I'm using when I need to get an overview over all relation sizes ordered by size in descending order:</p> <div class="highlight"><pre><code class="sql"><span class="k">select</span> <span class="n">n</span><span class="p">.</span><span class="n">nspname</span> <span class="k">as</span> <span class="ss">"Schema"</span><span class="p">,</span> <span class="k">c</span><span class="p">.</span><span class="n">relname</span> <span class="k">as</span> <span class="ss">"Name"</span><span class="p">,</span> <span class="k">case</span> <span class="k">c</span><span class="p">.</span><span class="n">relkind</span> <span class="k">when</span> <span class="s1">'r'</span> <span class="k">then</span> <span class="s1">'table'</span> <span class="k">when</span> <span class="s1">'v'</span> <span class="k">then</span> <span class="s1">'view'</span> <span class="k">when</span> <span class="s1">'i'</span> <span class="k">then</span> <span class="s1">'index'</span> <span class="k">when</span> <span class="s1">'S'</span> <span class="k">then</span> <span class="s1">'sequence'</span> <span class="k">when</span> <span class="s1">'s'</span> <span class="k">then</span> <span class="s1">'special'</span> <span class="k">end</span> <span class="k">as</span> <span class="ss">"Type"</span><span class="p">,</span> <span class="n">pg_catalog</span><span class="p">.</span><span class="n">pg_get_userbyid</span><span class="p">(</span><span class="k">c</span><span class="p">.</span><span class="n">relowner</span><span class="p">)</span> <span class="k">as</span> <span class="ss">"Owner"</span><span class="p">,</span> <span class="n">pg_catalog</span><span class="p">.</span><span class="n">pg_size_pretty</span><span class="p">(</span><span class="n">pg_catalog</span><span class="p">.</span><span class="n">pg_relation_size</span><span class="p">(</span><span class="k">c</span><span class="p">.</span><span class="n">oid</span><span class="p">))</span> <span class="k">as</span> <span class="ss">"Size"</span> <span class="k">from</span> <span class="n">pg_catalog</span><span class="p">.</span><span class="n">pg_class</span> <span class="k">c</span> <span class="k">left</span> <span class="k">join</span> <span class="n">pg_catalog</span><span class="p">.</span><span class="n">pg_namespace</span> <span class="n">n</span> <span class="k">on</span> <span class="n">n</span><span class="p">.</span><span class="n">oid</span> <span class="o">=</span> <span class="k">c</span><span class="p">.</span><span class="n">relnamespace</span> <span class="k">where</span> <span class="k">c</span><span class="p">.</span><span class="n">relkind</span> <span class="k">IN</span> <span class="p">(</span><span class="s1">'r'</span><span class="p">,</span> <span class="s1">'v'</span><span class="p">,</span> <span class="s1">'i'</span><span class="p">)</span> <span class="k">order</span> <span class="k">by</span> <span class="n">pg_catalog</span><span class="p">.</span><span class="n">pg_relation_size</span><span class="p">(</span><span class="k">c</span><span class="p">.</span><span class="n">oid</span><span class="p">)</span> <span class="k">desc</span><span class="p">;</span> </code></pre> </div> <p>Of course I could have come up with this without source code digging, but honestly, I didn't know about relkind s, about pg_size_pretty and pg_relation_size (I would have thought that one to be stored in some system view), so figuring all of this out would have taken much more time than just reading the source code.</p> <p>Now it's here so I remember it next time I need it.</p> </article> <article> <h1>overpriced data roaming</h1> <p>2010-11-04T00:00:00+01:00</p> <p>You shouldn't complain if something gets cheaper. But if something just gets 7 times cheaper from one day to the next, then that leaves you thinking whether the price offered so far might have been a tad bit too high.</p> <p>I'm talking about Swisscom's data roaming charges.</p> <p>Up to now, you paid CHF 50 per 5 MB (CHF 10 per MB) when roaming in the EU. Yes. That's around $10 and EUR 6.60 per <strong>Megabyte</strong>. Yes. Megabyte. Not Gigabyte. And you people complain about getting limited to 5 GB for your $30.</p> <p>Just now I got a <a href="http://www.swisscom.ch/NR/exeres/FC2C644E-DFB7-49E4-8326-93C03D317BAF,frameless.htm?lang=de">press release</a> form Swisscom that they are changing their roaming charges to CHF 7 per 5 MB. That's CHF 1.40 per MB which is <strong>7 times cheaper.</strong></p> <p>If you can make a product of yours 7 times cheaper from one day to the other, the rates you charged before that were clearly way too high.</p> </article> <article> <h1>How to kill IE performance</h1> <p>2010-10-07T00:00:00+02:00</p> <p>While working on my day job, we are often dealing with huge data tables in HTML augmented with some JavaScript to do calculations with that data.</p> <p>Think huge shopping cart: You change the quantity of a line item and the line total as well as the order total will change.</p> <p>This leads to the same data (line items) having three representations:</p> <ol> <li>The model on the server</li> <li>The HTML UI that is shown to the user</li> <li>The model that's seen by JavaScript to do the calculations on the client side (and then updating the UI)</li> </ol> <p>You might think that the JavaScript running in the browser would somehow be able to work with the data from 2) so that the third model wouldn't be needed, but due to various localization issues (think number formatting) and data that's not displayed but affects the calculations, that's not possible.</p> <p>So the question is: Considering we have some HTML templating language to build 2), how do we get to 3).</p> <p>Back in 2004 when I initially designed that system (using AJAX before it was widely called AJAX even), I hadn't seen <a href="http://video.yahoo.com/watch/111593/1710507">Crockford's lecture</a>s yet, so I still lived in the "JS sucks" world, where I've done something like this</p> <div class="highlight"><pre><code class="xml"><span class="c"></span> <span class="nt"><tr></span> <span class="nt"><td></span>Column 1 <span class="nt"><script></span>addSet(1234 /*prodid*/, 1 /*quantity*/, 10 /*price*/, /* and, later, more, stuff, so, really, ugly */)<span class="nt"></script></td></span> <span class="nt"><td></span>Column 2<span class="nt"></td></span> <span class="nt"><td></span>Column 3<span class="nt"></td></span> <span class="nt"></tr></span> <span class="c"></span> </code></pre> </div> <p>(Yeah - as I said: 2004. No object literals, global functions. We had a lot to learn back then, but so did you, so don't be too angry at me - we improved)</p> <p>Obviously, this doesn't scale: As the line items got more complicated, that parameter list grew and grew. The HTML code got uglier and uglier and of course, cluttering the window object is a big no-no too. So we went ahead and built a beautiful design:</p> <div class="highlight"><pre><code class="xml"><span class="c"></span> <span class="nt"><tr</span> <span class="na">class=</span><span class="s">"lineitem"</span> <span class="na">data-ps-lineitem=</span><span class="s">'{"prodid": 1234, "quantity": 1, "price": 10, "foo": "bar", "blah": "blah"}'</span><span class="nt">></span> <span class="nt"><td></span>Column 1<span class="nt"></td></span> <span class="nt"><td></span>Column 2<span class="nt"></td></span> <span class="nt"><td></span>Column 3<span class="nt"></td></span> <span class="nt"></tr></span> <span class="c"></span> </code></pre> </div> <p>The first iteration was then parsing that JSON every time we needed to access any of the associated data (and serializing again whenever it changed). Of course this didn't go that well performance-wise, so we began caching and did something like this (using jQuery):</p> <div class="highlight"><pre><code class="javascript"><span class="nx">$</span><span class="p">(</span><span class="kd">function</span><span class="p">(){</span> <span class="nx">$</span><span class="p">(</span><span class="s1">'.lineitem'</span><span class="p">).</span><span class="nx">each</span><span class="p">(</span><span class="kd">function</span><span class="p">(){</span> <span class="k">this</span><span class="p">.</span><span class="nx">ps_data</span> <span class="o">=</span> <span class="nx">$</span><span class="p">.</span><span class="nx">parseJSON</span><span class="p">(</span><span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">).</span><span class="nx">attr</span><span class="p">(</span><span class="s1">'data-ps-lineitem'</span><span class="p">));</span> <span class="p">});</span> <span class="p">});</span> </code></pre> </div> <p>Now each DOM element representing one of these <tr>'s had a ps_data member which allowed for quick access. The JSON had to be parsed only once and then the data was available. If it changed, writing it back didn't require a re-serialization either - you just changed that property directly.</p> <p>This design is reasonably clean (still not as DRY as the initial attempt which had the data only in that JSON string) while still providing enough performance.</p> <p>Until you begin to amass datasets. That is.</p> <p>Well. Until you do so and expect this to work in IE.</p> <p>800 rows like this made IE lock up its UI thread for 40 seconds.</p> <p>So more optimization was in order.</p> <p>First,</p> <div class="highlight"><pre><code class="javascript"><span class="nx">$</span><span class="p">(</span><span class="s1">'.lineitem'</span><span class="p">)</span> </code></pre> </div> <p>will kill IE. Remember: IE (still) doesn't have getElementsByClassName, so in IE, jQuery has to iterate the whole DOM and check whether each elements class attribute contains "lineitem". Considering that IE's DOM isn't really fast to start with, this is a HUGE no-no.</p> <p>So.</p> <div class="highlight"><pre><code class="javascript"><span class="nx">$</span><span class="p">(</span><span class="s1">'tr.lineitem'</span><span class="p">)</span> </code></pre> </div> <p>Nope. Nearly as bad considering there are still at least 800 tr's to iterate over.</p> <div class="highlight"><pre><code class="javascript"><span class="nx">$</span><span class="p">(</span><span class="s1">'#whatever tr.lineitem'</span><span class="p">)</span> </code></pre> </div> <p>Would help if it weren't 800 tr's that match. Using <a href="http://ajax.dynatrace.com/pages/">dynaTrace AJAX</a> (highly recommended tool, by the way) we found out that just selecting the elements alone (without the iteration) took more than 10 seconds.</p> <p>So the general take-away is: Selecting lots of elements in IE is painfully slow. Don't do that.</p> <p>But back to our little problem here. Unserializing that JSON at DOM ready time is not feasible in IE, because no matter what we do to that selector, once there are enough elements to handle, it's just going to be slow.</p> <p>Now by chunking up the amount of work to do and using setTimeout() to launch various deserialization jobs we could fix the locking up, but the total run time before all data is deserialized will still be the same (or slightly worse).</p> <p>So what we have done in 2004, even though it was ugly, was way more feasible in IE.</p> <p>Which is why we went back to the initial design with some improvements:</p> <div class="highlight"><pre><code class="xml"><span class="c"></span> <span class="nt"><tr</span> <span class="na">class=</span><span class="s">"lineitem"</span><span class="nt">></span> <span class="nt"><td></span>Column 1 <span class="nt"><script></span>PopScan.LineItems.add({"prodid": 1234, "quantity": 1, "price": 10, "foo": "bar", "blah": "blah"});<span class="nt"></script></td></span> <span class="nt"><td></span>Column 2<span class="nt"></td></span> <span class="nt"><td></span>Column 3<span class="nt"></td></span> <span class="nt"></tr></span> <span class="c"></span> </code></pre> </div> <p><em>phew</em> crisis averted.</p> <p>Loading time went back to where it was in the 2004 design. It was still bad though. With those 800 rows, IE was still taking more than 10 seconds for the rendering task. dynaTrace revealed that this time, the time was apparently spent rendering.</p> <p>The initial feeling was that there's not much to do at that point.</p> <p>Until we began suspecting the script tags.</p> <p>Doing this:</p> <div class="highlight"><pre><code class="xml"><span class="c"></span> <span class="nt"><tr</span> <span class="na">class=</span><span class="s">"lineitem"</span><span class="nt">></span> <span class="nt"><td></span>Column 1<span class="nt"></td></span> <span class="nt"><td></span>Column 2<span class="nt"></td></span> <span class="nt"><td></span>Column 3<span class="nt"></td></span> <span class="nt"></tr></span> <span class="c"></span> </code></pre> </div> <p>The page loaded instantly.</p> <p>Doing this</p> <div class="highlight"><pre><code class="xml"><span class="c"></span> <span class="nt"><tr</span> <span class="na">class=</span><span class="s">"lineitem"</span><span class="nt">></span> <span class="nt"><td></span>Column 1 <span class="nt"><script></span>1===1;<span class="nt"></script></td></span> <span class="nt"><td></span>Column 2<span class="nt"></td></span> <span class="nt"><td></span>Column 3<span class="nt"></td></span> <span class="nt"></tr></span> <span class="c"></span> </code></pre> </div> <p>it took 10 seconds again.</p> <p>Considering that IE's JavaScript engine runs as a COM component, this isn't actually that surprising: Whenever IE hits a script tag, it stops whatever it's doing, sends that script over to the COM component (first doing all the marshaling of the data), waits for that to execute, marshals the result back (depending on where the DOM lives and whether the script accesses it, possibly crossing that COM boundary many, many times in between) and then finally resumes page loading.</p> <p>It has to wait for each script because, potentially, that JavaScript could call document.open() / document.write() at which point the document could completely change.</p> <p>So the final solution was to loop through the server-side model twice and do something like this:</p> <div class="highlight"><pre><code class="xml"><span class="c"></span> <span class="nt"><tr</span> <span class="na">class=</span><span class="s">"lineitem"</span><span class="nt">></span> <span class="nt"><td></span>Column 1 <span class="nt"></td></span> <span class="nt"><td></span>Column 2<span class="nt"></td></span> <span class="nt"><td></span>Column 3<span class="nt"></td></span> <span class="nt"></tr></span> <span class="c"></span> <span class="nt"></table></span> <span class="nt"><script></span> PopScan.LineItems.add({prodid: 1234, quantity: 1, price: 10, foo: "bar", blah: "blah"}); // 800 more of these <span class="nt"></script></span> </code></pre> </div> <p>Problem solved. Not too ugly design. Certainly no 2004 design any more.</p> <p>And in closing, let me give you a couple of things you can do if you want to bring the performance of IE down to its knees:</p> <ul> <li>Use broad jQuery selectors. <code>$('.someclass')</code> will cause jQuery to loop through <em>all</em> elements on the page.</li> <li>Even if you try not to be broad, you can still kill performance: <code>$('div.someclass')</code>. The most help jQuery can expect from IE is getElementsByTagName, so while it's better than iterating <em>all</em> elements, it's still going over all div's on your page. Once it's more than 200, the performance extremely quickly falls down (probably doing some O(n^2) thing somehwere).</li> <li>Use a lot of <script>-tags. Every one of these will force IE to marshal data to the scripting engine COM component and to wait for the result.</li> </ul> <p>Next time, we'll have a look at how to use jQuery's delegate() to handle common cases with huge selectors.</p> </article> </main></body></html>