<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" version="2.0"><channel><description>Silk lets you publish collections. This is our engineering blog.</description><title>Silk Engineering Blog</title><generator>Tumblr (3.0; @silkapp-engineering)</generator><link>http://engineering.silk.co/</link><item><title>Announcing rest - A Haskell REST framework</title><description>&lt;p&gt;We are excited to officially announce the open source release of our REST framework &lt;code&gt;rest&lt;/code&gt;!&lt;/p&gt;

&lt;p&gt;&lt;code&gt;rest&lt;/code&gt; is a set of packages used to write, document, and use RESTful applications. You write your API in Haskell using &lt;code&gt;rest&lt;/code&gt;&amp;rsquo;s DSL. This API can then be run in different web frameworks like happstack, snap, or wai. Additionally, you can automatically generate documentation from it, as well as client libraries for Haskell and Javascript. We have been using it in production for most of our services for a long time and we like it so much that we decided to share it with the public.&lt;/p&gt;

&lt;p&gt;If you want to start using &lt;code&gt;rest&lt;/code&gt;, check out &lt;a href="http://silkapp.github.io/rest/tutorial.html"&gt;the tutorial&lt;/a&gt; or the &lt;a href="https://github.com/silkapp/rest/tree/master/rest-example"&gt;example application&lt;/a&gt;. You can also come to the &lt;a href="https://skillsmatter.com/conferences/1907-haskell-exchange-2014"&gt;Haskell Exchange 2014&lt;/a&gt; where Erik will give a talk about &lt;code&gt;rest&lt;/code&gt;. We&amp;rsquo;d also be happy to answer any questions you have, &lt;a href="https://groups.google.com/a/silk.co/forum/#!forum/code"&gt;shoot us an e-mail&lt;/a&gt;!&lt;/p&gt;

&lt;p&gt;The most important packages are:&lt;/p&gt;

&lt;ul&gt;&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/rest-core"&gt;rest-core&lt;/a&gt;: A DSL for defining versioned and web server agnostic REST resources. This is the workhorse of the framework&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/rest-gen"&gt;rest-gen&lt;/a&gt;: Automatically generates documentation, Haskell, JavaScript, and Ruby clients from a rest API&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/rest-snap"&gt;rest-snap&lt;/a&gt;, &lt;a href="http://hackage.haskell.org/package/rest-happstack"&gt;rest-happstack&lt;/a&gt;, &lt;a href="http://hackage.haskell.org/package/rest-wai"&gt;rest-wai&lt;/a&gt;: Drivers for running resources using the web server of your choice&lt;/li&gt;
&lt;/ul&gt;&lt;p&gt;We have also released other packages that are either used by or can be used with &lt;code&gt;rest&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/rest-client"&gt;rest-client&lt;/a&gt;: Used by haskell clients generated by rest-gen&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/rest-types"&gt;rest-types&lt;/a&gt;: Types used by the other rest packages&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/json-schema"&gt;json-schema&lt;/a&gt;: Define and derive schemas for JSON serializations&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/generic-aeson"&gt;generic-aeson&lt;/a&gt;: Generically derives JSON serializations for data types minimalistically&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/regular-xmlpickler"&gt;regular-xmlpickler&lt;/a&gt;: Generically derives XML serializations for data types&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/aeson-utils"&gt;aeson-utils&lt;/a&gt;: Utilities for working with Aeson.&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/hxt-pickle-utils"&gt;hxt-pickle-utils&lt;/a&gt;: Utility functions for using HXT picklers&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/multipart"&gt;multipart&lt;/a&gt;: HTTP Multipart implementation forked from the cgi package&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/rest-stringmap"&gt;rest-stringmap&lt;/a&gt;: Maps with string-like keys with built-in serialization to XML and JSON (since JSON doesn&amp;rsquo;t allow arbitrary keys)&lt;/li&gt;
&lt;li&gt;&lt;a href="http://hackage.haskell.org/package/code-builder"&gt;code-builder&lt;/a&gt;: String manipulation library for code generation&lt;/li&gt;
&lt;/ul&gt;&lt;p&gt;We had a great time working on rest at ZuriHac (thanks for organizing, Better!) and we are happy to see that a lot of people were interested in our work. We got a lot done, here are some highlights:&lt;/p&gt;

&lt;ul&gt;&lt;li&gt;&lt;a href="https://github.com/hesselink"&gt;Erik&lt;/a&gt; wrote an &lt;a href="http://silkapp.github.io/rest/tutorial.html"&gt;introductory tutorial&lt;/a&gt; to &lt;code&gt;rest&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/hth313"&gt;Håkan Thörngren&lt;/a&gt; rewrote the rest-gen Haskell code generator to use haskell-src-exts, it was released in rest-gen-0.14&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/jcberentsen"&gt;Christian Berentsen&lt;/a&gt; did several things:

&lt;ul&gt;&lt;li&gt;Cleaned up the interface of json-schema, it was released in json-schema-0.6&lt;/li&gt;
&lt;li&gt;Added support for outputing Fay compatible json as a separate output type (we want to make it easier to extend rest with more output types so things like this can go in external packages)&lt;/li&gt;
&lt;li&gt;Implemented a generic API discovery resource that you can hook into your API with no configuration&lt;/li&gt;
&lt;/ul&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/tomlokhorst"&gt;Tom Lokhorst&lt;/a&gt; helped out with some always appreciated bug fixing&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/sebastiaanvisser"&gt;Sebas&lt;/a&gt; wrote the rest-wai driver to make sure everyone can use the web server they prefer together with rest&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/bergmark"&gt;Adam&lt;/a&gt; worked on the &lt;a href="https://github.com/silkapp/rest/tree/master/rest-example"&gt;rest-example application&lt;/a&gt; and rewrote parts of rest-gen to make the code generator and the library interface simpler&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/wiz"&gt;wiz&lt;/a&gt; did performance benchmarks and created a script to generate haddocks for rest itself&lt;/li&gt;
&lt;/ul&gt;&lt;p&gt;I hope I didn&amp;rsquo;t forget anyone. A big thanks to everyone who participated!&lt;/p&gt;

&lt;p&gt;All the mentioned projects are available on &lt;a href="http://hackage.haskell.org"&gt;hackage&lt;/a&gt; and in public repositories on &lt;a href="http://www.github.com/silkapp"&gt;github&lt;/a&gt;. We also created &lt;a href="https://groups.google.com/a/silk.co/forum/#!forum/code"&gt;a public mailing list for all our open source projects&lt;/a&gt;.&lt;/p&gt;

&lt;div class="hiring"&gt;We use these packages to write our Haskell API code for Silk.&lt;br/&gt;&lt;a href="http://jobs.silk.co/"&gt;Join our team if you enjoy this stuff too, we&amp;rsquo;re hiring!&lt;/a&gt;&lt;/div&gt;</description><link>http://engineering.silk.co/post/90354057868</link><guid>http://engineering.silk.co/post/90354057868</guid><pubDate>Mon, 30 Jun 2014 10:22:00 -0400</pubDate><category>haskell</category><category>web</category></item><item><title>Writing admin interfaces with Fay using fay-builder</title><description>&lt;p&gt;We are in the process of open sourcing a lot of the general purpose
libraries we&amp;rsquo;ve built at Silk under the BSD3 license. We&amp;rsquo;re planning
to write a series of blog posts introducing them so please stay tuned!&lt;/p&gt;

&lt;p&gt;First off is one of the smaller packages,
&lt;a href="http://hackage.haskell.org/package/fay-builder"&gt;fay-builder&lt;/a&gt;
(&lt;a href="http://github.com/silkapp/fay-builder"&gt;github&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;&lt;a href="http://www.fay-lang.org"&gt;Fay&lt;/a&gt; is a proper subset of Haskell that
compiles to JavaScript, as well as a compiler. I&amp;rsquo;ve been helping out
with the project since its first release in July 2012.&lt;/p&gt;

&lt;p&gt;We have several Haskell backend services with admin interfaces written
in HTML and JavaScript that communicate with the running server using
REST APIs. It seemed like a good idea to write their admin interfaces
in Haskell and we have been experimenting with using Fay for
this. Since Fay supports &lt;a href="http://www.haskell.org/cabal/"&gt;Cabal&lt;/a&gt;
packages we can use our usual tools and upload these packages to the
public &lt;a href="http://hackage.haskell.org"&gt;hackage&lt;/a&gt; or our private instance.&lt;/p&gt;

&lt;p&gt;fay-builder lets you store options needed to compile Fay code along
with an executable. This is done by using custom cabal flags that can
be read either by &lt;code&gt;Setup.hs&lt;/code&gt; (but see caveats below) or from within the
application at startup, on request, or at some other point.&lt;/p&gt;

&lt;p&gt;If you want to try it out you can install it by running &lt;code&gt;cabal install
fay-builder&lt;/code&gt;.&lt;/p&gt;

&lt;div class="hiring"&gt;This post demonstrates how we build Haskell code for our admin interfaces.&lt;br/&gt;&lt;a href="http://jobs.silk.co/"&gt;Join our team if you enjoy this stuff too, we&amp;rsquo;re hiring!&lt;/a&gt;&lt;/div&gt;

&lt;h1&gt;The Cabal file&lt;/h1&gt;

&lt;p&gt;Here&amp;rsquo;s how you can define a Cabal file that uses fay-builder for a project:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;name:                my-program
version:             0.1
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;Use a custom build type&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;build-type:          Custom
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;code&gt;extra-source-files&lt;/code&gt; specifies files what should be included with a
source distribution. The fay files are added here so they don&amp;rsquo;t get
lost when running &lt;code&gt;cabal sdist&lt;/code&gt;.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;extra-source-files: fay/*.hs
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;code&gt;data-files&lt;/code&gt; are also included, with the addition that Cabal will
generate a paths module that you can use to fetch the resources during
runtime. We probably want to precompile the fay files so the program
can just serve them.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;data-files: my-program.cabal, js/*.js
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;&lt;code&gt;x-foo&lt;/code&gt; specifies custom flags that Cabal itself ignores, but we can
use it for extra configuration options.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;x-fay-packages:      fay-jquery, fay-text
x-fay-root-modules:  Index, Misc
x-fay-include-paths: src
x-fay-output-dir:    data/js
x-fay-source-dir:    fay

executable my-program
  main-is:           Main.hs
  hs-source-dirs:    src
  default-language:  Haskell2010
  ghc-options:       -Wall
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;This is the haskell module Cabal generates to let us access data-files.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;other-modules:     Paths_my_program
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;To make sure we can build the Fay code we add all the dependencies it
has here along with the GHC libraries dependencies.&lt;/p&gt;

&lt;p&gt;One thing to note is that we can&amp;rsquo;t depend on base and fay-base at the
same time since they have clashing module names. But if we depend on
some other fay package we get a transitive dependency to fay-base and
we know it will be available.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;build-depends:     base, Cabal, fay, fay-builder, fay-jquery, fay-text
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;We have a few options for when to build the Fay code.&lt;/p&gt;

&lt;h1&gt;Building as a Cabal hook&lt;/h1&gt;

&lt;p&gt;With a custom Setup.hs we can add a postBuildHook that will compile
the Fay code after the executable has been compiled, but before an
sdist is created. This is nice because it will generate the needed .js
files before the tarball is created. The disadvantage is that Setup.hs
cannot have explicit dependencies so you won&amp;rsquo;t be able to sdist your
program unless you already have all dependencies installed.&lt;/p&gt;

&lt;p&gt;It is possible to do this, and fay-builder includes a &lt;code&gt;defaultFayHook&lt;/code&gt;
that you can use in Setup.hs like this:&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;import Fay.Builder (defaultFayHook)
main = defaultFayHook
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;It&amp;rsquo;s pretty cool, but maybe not very useful because of the dependency
issue. Just make sure to have fay-builder installed before running
&lt;code&gt;cabal configure&lt;/code&gt;.&lt;/p&gt;

&lt;h1&gt;Building inside the program&lt;/h1&gt;

&lt;p&gt;For one application I started out with the above Setup.hs approach,
but soon realized that this would not work out since sdisting wasn&amp;rsquo;t
possible in a clean package DB. I instead chose to add a
&lt;code&gt;--compile-fay&lt;/code&gt; flag that would compile all Fay sources when the
process starts when developing. The live setup won&amp;rsquo;t do this and
instead read the data-files.&lt;/p&gt;

&lt;p&gt;The trick here is that we added the .cabal file as a data-file to let
us access it from other modules.&lt;/p&gt;

&lt;pre&gt;&lt;code&gt;import qualified Paths_my_program as Paths
import qualified Fay.Builder as Builder

compileFay :: IO ()
compileFay = do
  pkgDb       &amp;lt;- getPackageDb
  packageDesc &amp;lt;- Paths.getDataFileName "my-program.cabal"
                 &amp;gt;&amp;gt;= Builder.readPackageDescription
  Builder.build packageDesc pkgDb
    where
      -- Some clever way to fetch the current package DB
      -- if you are using a sandbox.
      getPackageDb :: IO (Maybe FilePath)
&lt;/code&gt;&lt;/pre&gt;

&lt;p&gt;If we had added the settings elsewhere, such as in a custom
&lt;a href="http://hackage.haskell.org/package/configurator"&gt;configurator&lt;/a&gt; file
like &lt;a href="http://hackage.haskell.org/package/snaplet-fay"&gt;snaplet-fay&lt;/a&gt;
does, we wouldn&amp;rsquo;t be able to read it from within &lt;code&gt;Setup.hs&lt;/code&gt; so the cabal
file might be the only place to put this configuration if we want both
features at once.&lt;/p&gt;

&lt;p&gt;One of the biggest advantages in using fay-builder is that all
dependencies can be listed in one cabal file instead of having to
split the client and server code into multiple packages (which would
turn into three different packages if there&amp;rsquo;s shared code).&lt;/p&gt;</description><link>http://engineering.silk.co/post/82777010096</link><guid>http://engineering.silk.co/post/82777010096</guid><pubDate>Tue, 15 Apr 2014 05:38:00 -0400</pubDate></item><item><title>Reactive Programming in JavaScript</title><description>&lt;p&gt;Writing large JavaScript projects is hard. The language lacks both a type system and a canonical module system, it has a rather clunky syntax and has a threading model that is, to say the least, interesting. No wonder so many third party JavaScript frameworks exist. Luckily JavaScript is a flexible language, which makes extending it easy.&lt;/p&gt;
&lt;p&gt;As the Silk client keeps growing, keeping our JavaScript code base clean has proven to be a bit of a challenge as well. One of the techniques we use to keep the complexity manageable is &lt;strong&gt;Functional Reactive Programming&lt;/strong&gt; (FRP). Reactive programming allows us to declaratively set up bindings between data in different parts of our product that keep in sync automatically. Most dynamic parts of our user interface are reactively connected to our underlying data model. FRP allows us to abstract over information that changes over time with minimal effort.&lt;/p&gt;
&lt;p&gt;Currently we solely use reactive programming within our client application. Although not unthinkable (or even uncommon) we have no reactive connection from the client to the server (yet).&lt;/p&gt;
&lt;p&gt;At Silk we developed our own JavaScript library for reactive programming. The interface is heavily inspired by functional programming idioms from Haskell. Prototyping the library in a type safe manner in Haskell gave us great confidence about the expressiveness of the programming interface.&lt;/p&gt;
&lt;div class="hiring"&gt;This post demonstrates how we use reactive programming as part of the development process of Silk. If you think this stuff is interesting &lt;a href="http://jobs.silk.co/"&gt;you might consider joining us!&lt;/a&gt;&lt;/div&gt;
&lt;h3 id="reactive-variables"&gt;Reactive Variables&lt;/h3&gt;
&lt;p&gt;At the basis of our library we can find &lt;strong&gt;reactive variables&lt;/strong&gt;, mutable reference cells containing a value that might change over time. Value changes within a variable can easily be observed by other reactive variables, or by listeners that can perform arbitrary side effects.&lt;/p&gt;
&lt;p&gt;We illustrate the basic usage of reactive variables below. We create a variable containing a number and inspect the current value by using the snapshot method &lt;code&gt;get&lt;/code&gt;. We can write a new value to the variable with &lt;code&gt;set&lt;/code&gt;, after which we can observe a new value. Pretty straightforward so far.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// Creating a variable with initial value:&lt;/span&gt;
&lt;span class="kw"&gt;var&lt;/span&gt; v = &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="dv"&gt;20&lt;/span&gt;);

&lt;span class="co"&gt;// Snapshot the value using 'get':&lt;/span&gt;
&lt;span class="ot"&gt;console&lt;/span&gt;.&lt;span class="fu"&gt;log&lt;/span&gt;(&lt;span class="st"&gt;"v:"&lt;/span&gt;, &lt;span class="ot"&gt;v&lt;/span&gt;.&lt;span class="fu"&gt;get&lt;/span&gt;()); &lt;span class="co"&gt;// prints 20&lt;/span&gt;

&lt;span class="co"&gt;// Update the value using 'set':&lt;/span&gt;
&lt;span class="ot"&gt;v&lt;/span&gt;.&lt;span class="fu"&gt;set&lt;/span&gt;(&lt;span class="dv"&gt;10&lt;/span&gt;);

&lt;span class="ot"&gt;console&lt;/span&gt;.&lt;span class="fu"&gt;log&lt;/span&gt;(&lt;span class="st"&gt;"v:"&lt;/span&gt;, &lt;span class="ot"&gt;v&lt;/span&gt;.&lt;span class="fu"&gt;get&lt;/span&gt;()); &lt;span class="co"&gt;// now prints 10&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Reactive variables can be observed by other reactive variables by linking them using the &lt;code&gt;link&lt;/code&gt; method. After linking two variables their values will keep in sync. Links can be broken with the &lt;code&gt;unlink&lt;/code&gt; method. Once a value is linked, so it depends on another value, setting it manually makes no sense anymore.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// Create two variables and link one to another:&lt;/span&gt;
&lt;span class="kw"&gt;var&lt;/span&gt; a = &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="dv"&gt;20&lt;/span&gt;);
&lt;span class="kw"&gt;var&lt;/span&gt; b = &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="dv"&gt;10&lt;/span&gt;);
&lt;span class="ot"&gt;b&lt;/span&gt;.&lt;span class="fu"&gt;link&lt;/span&gt;(a); &lt;span class="co"&gt;// b.get()  =&amp;gt;  20&lt;/span&gt;

&lt;span class="co"&gt;// Setting variable 'a' will now update the value of 'b' as well:&lt;/span&gt;
&lt;span class="ot"&gt;a&lt;/span&gt;.&lt;span class="fu"&gt;set&lt;/span&gt;(&lt;span class="dv"&gt;4&lt;/span&gt;);  &lt;span class="co"&gt;// b.get()  =&amp;gt;  4&lt;/span&gt;

&lt;span class="co"&gt;// Unlinking cleans up the connection:&lt;/span&gt;
&lt;span class="ot"&gt;b&lt;/span&gt;.&lt;span class="fu"&gt;unlink&lt;/span&gt;();
&lt;span class="ot"&gt;a&lt;/span&gt;.&lt;span class="fu"&gt;set&lt;/span&gt;(&lt;span class="dv"&gt;3&lt;/span&gt;);  &lt;span class="co"&gt;// b.get()  =&amp;gt; still 4&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Linking variables one-to-one can be useful, but isn’t the most interesting of use cases. In the section about composition we’ll see some combinator functions that link variables in more interesting ways.&lt;/p&gt;
&lt;h3 id="side-effects"&gt;Side Effects&lt;/h3&gt;
&lt;p&gt;To be able to perform actions when the value of a reactive variables changes, we can attach listeners. We use the function &lt;code&gt;onchange&lt;/code&gt; to register a side effect.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="kw"&gt;var&lt;/span&gt; myTitle = &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="st"&gt;"&amp;lt;untitled&amp;gt;"&lt;/span&gt;);

&lt;span class="co"&gt;// Install the side effect (using ECMAScript 6 lambdas!)&lt;/span&gt;
&lt;span class="kw"&gt;var&lt;/span&gt; eff = &lt;span class="ot"&gt;myTitle&lt;/span&gt;.&lt;span class="fu"&gt;onchange&lt;/span&gt;((t) =&amp;gt; &lt;span class="ot"&gt;document&lt;/span&gt;.&lt;span class="fu"&gt;title&lt;/span&gt; = t);

&lt;span class="co"&gt;// Setting the reactive value will now automatically&lt;/span&gt;
&lt;span class="co"&gt;// update the document.title as well:&lt;/span&gt;
&lt;span class="ot"&gt;myTitle&lt;/span&gt;.&lt;span class="fu"&gt;set&lt;/span&gt;(&lt;span class="st"&gt;"MyDocument-1"&lt;/span&gt;);

&lt;span class="co"&gt;// ...after a while, when no longer needed:&lt;/span&gt;
&lt;span class="ot"&gt;eff&lt;/span&gt;.&lt;span class="fu"&gt;cleanup&lt;/span&gt;();&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Note that if you set the value of a reactive variable to the value it already contains, no value propagation will happen and no side effects will be fired. Side effects only fire when the value changes or during initialization. The &lt;code&gt;onchange&lt;/code&gt; method returns a handle that can be used to cleanup the effect when no longer needed.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;onchange&lt;/code&gt; method is really only meant to connect a value (or network of values) to some part of the outside world (like the document title), not to update other variables. Updating other variables is better done using &lt;code&gt;link&lt;/code&gt; or one of the composition combinators described below.&lt;/p&gt;
&lt;h3 id="reactive-jquery"&gt;Reactive jQuery&lt;/h3&gt;
&lt;p&gt;As part of our framework we’ve built a small plug-in for jQuery that allows us to connect reactive variables to the DOM. We extend jQuery with methods similar to existing jQuery methods, but prefixed with the letter &lt;code&gt;r&lt;/code&gt;, indicating their reactive counterparts.&lt;/p&gt;
&lt;p&gt;For example, &lt;code&gt;rText&lt;/code&gt; allows us to connect a reactive string to the text contents of a DOM element.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="kw"&gt;var&lt;/span&gt; header = &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="st"&gt;"initial string"&lt;/span&gt;);
&lt;span class="fu"&gt;$&lt;/span&gt;(&lt;span class="st"&gt;"h1"&lt;/span&gt;).&lt;span class="fu"&gt;first&lt;/span&gt;().&lt;span class="fu"&gt;rText&lt;/span&gt;(header);

&lt;span class="co"&gt;// Setting the reactive value now automatically updates&lt;/span&gt;
&lt;span class="co"&gt;// the text contents of our header element:&lt;/span&gt;
&lt;span class="ot"&gt;header&lt;/span&gt;.&lt;span class="fu"&gt;set&lt;/span&gt;(&lt;span class="st"&gt;"New Header Text!"&lt;/span&gt;);&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Besides &lt;code&gt;rText&lt;/code&gt;, we have &lt;code&gt;rHtml&lt;/code&gt;, &lt;code&gt;rAttr&lt;/code&gt;, &lt;code&gt;rAddClass&lt;/code&gt;, &lt;code&gt;rToggle&lt;/code&gt;, and several variants. Instead of taking normal strings, HTML fragments, or booleans as input, the reactive counterparts take reactive variables containing strings, HTML fragments, or booleans as input. When the values change the DOM will update automatically. Additionally (with the use of mutation observers) we were even able to build a connection the other way around: when the DOM updates the connected variables also update.&lt;/p&gt;
&lt;p&gt;Adding more functionality as jQuery plug-ins is easy and can be done with the use of the &lt;code&gt;onchange&lt;/code&gt; method. For example, to get the value of an &lt;code&gt;&amp;lt;input&amp;gt;&lt;/code&gt; element as a reactive value we could write:&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// Extend jQuery with reactive input field values:&lt;/span&gt;
&lt;span class="ot"&gt;$&lt;/span&gt;.&lt;span class="ot"&gt;fn&lt;/span&gt;.&lt;span class="fu"&gt;rVal&lt;/span&gt; =
  &lt;span class="kw"&gt;function&lt;/span&gt; (rv) &lt;span class="co"&gt;// reactive variable as input&lt;/span&gt;
  {
    &lt;span class="kw"&gt;var&lt;/span&gt; el = &lt;span class="kw"&gt;this&lt;/span&gt;;

    &lt;span class="co"&gt;// When the input text changes, update the variable:&lt;/span&gt;
    &lt;span class="ot"&gt;el&lt;/span&gt;.&lt;span class="fu"&gt;on&lt;/span&gt;(&lt;span class="st"&gt;"input"&lt;/span&gt;, () =&amp;gt; &lt;span class="ot"&gt;rv&lt;/span&gt;.&lt;span class="fu"&gt;set&lt;/span&gt;(&lt;span class="ot"&gt;el&lt;/span&gt;.&lt;span class="fu"&gt;val&lt;/span&gt;()));

    &lt;span class="co"&gt;// When the variable changes, update the element:&lt;/span&gt;
    &lt;span class="ot"&gt;rv&lt;/span&gt;.&lt;span class="fu"&gt;onchange&lt;/span&gt;((v) =&amp;gt; &lt;span class="ot"&gt;el&lt;/span&gt;.&lt;span class="fu"&gt;val&lt;/span&gt;(v));

    &lt;span class="kw"&gt;return&lt;/span&gt; el;
  };&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now instead of jQuery’s &lt;code&gt;val&lt;/code&gt; method we can use our own &lt;code&gt;rVal&lt;/code&gt; method.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="kw"&gt;var&lt;/span&gt; myValue = &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="st"&gt;""&lt;/span&gt;);
&lt;span class="fu"&gt;$&lt;/span&gt;(&lt;span class="st"&gt;"form input"&lt;/span&gt;).&lt;span class="fu"&gt;rVal&lt;/span&gt;(myValue);

&lt;span class="co"&gt;// We can use the variable to interface with the input value:&lt;/span&gt;
&lt;span class="ot"&gt;myValue&lt;/span&gt;.&lt;span class="fu"&gt;set&lt;/span&gt;(&lt;span class="st"&gt;"new value"&lt;/span&gt;);&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Why would we want to use reactive variables instead of using jQuery directly? Because we can easily observe changes and compose them with other reactive variables!&lt;/p&gt;
&lt;h3 id="composition-using-lift"&gt;Composition Using Lift&lt;/h3&gt;
&lt;p&gt;Reactive values become interesting when you start composing them, building up data flow networks that automatically propagate information. The easiest way of composition in our framework is by lifting regular JavaScript functions into reactive functions. Reactive functions are functions that can be applied to reactive values and will return a reactive variable.&lt;/p&gt;
&lt;p&gt;Let’s say we have a JavaScript function to compute the length of the diagonal of a rectangle, given the width and height. We can make a reactive function out of this by using &lt;code&gt;Reactive.lift&lt;/code&gt;.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="kw"&gt;var&lt;/span&gt; diagonal = &lt;span class="ot"&gt;Reactive&lt;/span&gt;.&lt;span class="fu"&gt;lift&lt;/span&gt;((w, h) =&amp;gt; &lt;span class="ot"&gt;Math&lt;/span&gt;.&lt;span class="fu"&gt;sqrt&lt;/span&gt;(w * w + h * h));&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now we can apply it to reactive numbers instead of normal JavaScript numbers and get back a reactive number again.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// Create a rectangle with reactive properties:&lt;/span&gt;
&lt;span class="kw"&gt;var&lt;/span&gt; myRect =
  { &lt;span class="dt"&gt;width  &lt;/span&gt;: &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="dv"&gt;20&lt;/span&gt;)
  , &lt;span class="dt"&gt;height &lt;/span&gt;: &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="dv"&gt;4&lt;/span&gt;)
  };

&lt;span class="co"&gt;// Put the diagonal on our rectangle as a reactive computation&lt;/span&gt;
&lt;span class="co"&gt;// over the dimensions:&lt;/span&gt;
&lt;span class="ot"&gt;myRect&lt;/span&gt;.&lt;span class="fu"&gt;diagonal&lt;/span&gt; = &lt;span class="fu"&gt;diagonal&lt;/span&gt;(&lt;span class="ot"&gt;myRect&lt;/span&gt;.&lt;span class="fu"&gt;width&lt;/span&gt;, &lt;span class="ot"&gt;myRect&lt;/span&gt;.&lt;span class="fu"&gt;height&lt;/span&gt;);

&lt;span class="co"&gt;// Link it to the text of some DOM element:&lt;/span&gt;
&lt;span class="fu"&gt;$&lt;/span&gt;(&lt;span class="st"&gt;"div.diagonal"&lt;/span&gt;).&lt;span class="fu"&gt;rText&lt;/span&gt;(&lt;span class="ot"&gt;myRect&lt;/span&gt;.&lt;span class="fu"&gt;diagonal&lt;/span&gt;);

&lt;span class="co"&gt;// The text on the page will now automatically update when&lt;/span&gt;
&lt;span class="co"&gt;// changing the dimensions of our rectangle!&lt;/span&gt;
&lt;span class="ot"&gt;myRect&lt;/span&gt;.&lt;span class="ot"&gt;width&lt;/span&gt;.&lt;span class="fu"&gt;set&lt;/span&gt;(&lt;span class="dv"&gt;200&lt;/span&gt;);
&lt;span class="ot"&gt;myRect&lt;/span&gt;.&lt;span class="ot"&gt;height&lt;/span&gt;.&lt;span class="fu"&gt;set&lt;/span&gt;(&lt;span class="dv"&gt;2&lt;/span&gt;);&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The lifted functions will be re-run every time one of the input variables update. Because it is unclear when the function will run and even how many times it will run it must be kept &lt;strong&gt;pure&lt;/strong&gt;. Side effects in lifted functions can cause strange and undefined behavior.&lt;/p&gt;
&lt;p&gt;In the simple case of lifting a single argument function, we can use the method &lt;code&gt;map&lt;/code&gt; available directly on reactive values.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="kw"&gt;var&lt;/span&gt; numbers = &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="st"&gt;"1.0 30e3 20.07 50"&lt;/span&gt;);

&lt;span class="co"&gt;// Create a reactive view on the string as a list of actual&lt;/span&gt;
&lt;span class="co"&gt;// numbers. Because we lift a one argument function we can&lt;/span&gt;
&lt;span class="co"&gt;// just use 'map'.&lt;/span&gt;
&lt;span class="kw"&gt;var&lt;/span&gt; asList = &lt;span class="ot"&gt;numbers&lt;/span&gt;.&lt;span class="fu"&gt;map&lt;/span&gt;((s) =&amp;gt; &lt;span class="ot"&gt;s&lt;/span&gt;.&lt;span class="fu"&gt;split&lt;/span&gt;(&lt;span class="ot"&gt;/&lt;/span&gt;&lt;span class="bn"&gt;\s&lt;/span&gt;&lt;span class="fl"&gt;+&lt;/span&gt;&lt;span class="ot"&gt;/&lt;/span&gt;).&lt;span class="fu"&gt;map&lt;/span&gt;(parseFloat));&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;lift&lt;/code&gt; function is probably the mostly used combinator in our framework, because it allows us to create views on data that will automatically update when needed. These views are reactive variables as well, which makes attaching side effects and further composition easy.&lt;/p&gt;
&lt;div class="sidenote haskell"&gt;
&lt;p&gt;Why is the &lt;code&gt;map&lt;/code&gt; function called &lt;code&gt;map&lt;/code&gt;?&lt;/p&gt;
&lt;p&gt;The functions is called &lt;code&gt;map&lt;/code&gt; because it corresponds directly to the Haskell function &lt;code&gt;fmap&lt;/code&gt;, part of the &lt;code&gt;Functor&lt;/code&gt; type class. Similarly, the function &lt;code&gt;lift&lt;/code&gt; corresponds somewhat to a multi-argument version of Haskell’s &lt;code&gt;liftA&lt;/code&gt; (and &lt;code&gt;liftA2&lt;/code&gt;, &lt;code&gt;liftA3&lt;/code&gt;, etc) for the &lt;code&gt;Applicative&lt;/code&gt; type class. For those with a Haskell background we show the exact correspondence below.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// JavaScript:              --  Haskell:&lt;/span&gt;

&lt;span class="fu"&gt;lift&lt;/span&gt;(f)(a, b, c)            liftA3 f a b c
&lt;span class="fu"&gt;lift&lt;/span&gt;(f)(a, b, c)            f &amp;lt;$&amp;gt; a &amp;lt;*&amp;gt; b &amp;lt;*&amp;gt; c

&lt;span class="co"&gt;// JavaScript:              --  Haskell:&lt;/span&gt;

&lt;span class="fu"&gt;lift&lt;/span&gt;(f)(a)                  f &amp;lt;$&amp;gt; a
&lt;span class="fu"&gt;lift&lt;/span&gt;(f)(a)                  fmap f a
&lt;span class="ot"&gt;a&lt;/span&gt;.&lt;span class="fu"&gt;map&lt;/span&gt;(f)                    fmap f a&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Note that a unary &lt;code&gt;lift&lt;/code&gt; is equivalent to &lt;code&gt;map&lt;/code&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;h3 id="nested-reactive-variables"&gt;Nested Reactive Variables&lt;/h3&gt;
&lt;p&gt;What happens when a function lifted with &lt;code&gt;lift&lt;/code&gt; itself returns a reactive variable? The result of lifting will now be a reactive variable containing as the value another reactive variable. This is most likely not what you want.&lt;/p&gt;
&lt;p&gt;To flatten a nested reactive variable we can use the &lt;code&gt;join&lt;/code&gt; or &lt;code&gt;bind&lt;/code&gt; methods.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// Say 'v.inner' is reactive, we need 'join' to flatten afterwards:&lt;/span&gt;
&lt;span class="kw"&gt;var&lt;/span&gt; flattened = &lt;span class="ot"&gt;outer&lt;/span&gt;.&lt;span class="fu"&gt;map&lt;/span&gt;((v) =&amp;gt; &lt;span class="ot"&gt;v&lt;/span&gt;.&lt;span class="fu"&gt;inner&lt;/span&gt;).&lt;span class="fu"&gt;join&lt;/span&gt;();

&lt;span class="co"&gt;// Bind is simply a shortcut for 'join' after 'map':&lt;/span&gt;
&lt;span class="kw"&gt;var&lt;/span&gt; flattened = &lt;span class="ot"&gt;outer&lt;/span&gt;.&lt;span class="fu"&gt;bind&lt;/span&gt;((v) =&amp;gt; &lt;span class="ot"&gt;v&lt;/span&gt;.&lt;span class="fu"&gt;inner&lt;/span&gt;);&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To illustrate how this is useful we take a small example directly from the Silk product. In our client we allow users to edit Silk pages, which are internally called &lt;em&gt;documents&lt;/em&gt;. Every document contains a list of properties, like the title, the main article node and some more state variables. Some of those properties (like the title) are reactive, because we want an easy way to observe their changes.&lt;/p&gt;
&lt;p&gt;The Silk client also has the concept of a &lt;em&gt;canvas&lt;/em&gt;, an object that can render Silk documents to the screen. The canvas contains a reactive variable &lt;code&gt;activeDocument&lt;/code&gt; which tracks the document currently being rendered. Updating this property will render a new document.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// Create a Silk document:&lt;/span&gt;
&lt;span class="kw"&gt;var&lt;/span&gt; myDoc = { &lt;span class="dt"&gt;title &lt;/span&gt;: &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="st"&gt;"[untitled]"&lt;/span&gt;)
            , &lt;span class="dt"&gt;uri   &lt;/span&gt;: &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="st"&gt;"/page/untitled-document-31"&lt;/span&gt;)
            , &lt;span class="dt"&gt;node  &lt;/span&gt;: &lt;span class="fu"&gt;$&lt;/span&gt;(&lt;span class="st"&gt;"&amp;lt;article /&amp;gt;"&lt;/span&gt;)
            , &lt;span class="dt"&gt;dirty &lt;/span&gt;: &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(&lt;span class="kw"&gt;false&lt;/span&gt;)
            }

&lt;span class="co"&gt;// Render the document on the canvas by&lt;/span&gt;
&lt;span class="co"&gt;// setting the 'activeDocument':&lt;/span&gt;
&lt;span class="ot"&gt;Silk&lt;/span&gt;.&lt;span class="ot"&gt;canvas&lt;/span&gt;.&lt;span class="ot"&gt;activeDocument&lt;/span&gt;.&lt;span class="fu"&gt;set&lt;/span&gt;(myDoc);&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Part of the responsibility of the canvas is to keep the text of the page’s first &lt;code&gt;&amp;lt;h1&amp;gt;&lt;/code&gt; element in sync with active document’s title. We can model this as a reactive connection that we set up when constructing the canvas. The value will be kept in sync during the entire lifetime of the element.&lt;/p&gt;
&lt;p&gt;Because both the active document and the title property are reactive, we need &lt;code&gt;bind&lt;/code&gt; to project the title out.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// The Canvas constructor:&lt;/span&gt;
&lt;span class="kw"&gt;function&lt;/span&gt; &lt;span class="fu"&gt;Canvas&lt;/span&gt; (initialDoc)
{
  &lt;span class="kw"&gt;this&lt;/span&gt;.&lt;span class="fu"&gt;activeDocument&lt;/span&gt; = &lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(initialDoc);
  &lt;span class="kw"&gt;this&lt;/span&gt;.&lt;span class="fu"&gt;node&lt;/span&gt;           = &lt;span class="fu"&gt;$&lt;/span&gt;(&lt;span class="st"&gt;"#canvas"&lt;/span&gt;);

  &lt;span class="co"&gt;// Find the header and reactively set the text to&lt;/span&gt;
  &lt;span class="co"&gt;// the active document's title:&lt;/span&gt;
  &lt;span class="kw"&gt;var&lt;/span&gt; title = &lt;span class="kw"&gt;this&lt;/span&gt;.&lt;span class="ot"&gt;activeDocument&lt;/span&gt;.&lt;span class="fu"&gt;bind&lt;/span&gt;(d =&amp;gt; &lt;span class="ot"&gt;d&lt;/span&gt;.&lt;span class="fu"&gt;title&lt;/span&gt;);
  &lt;span class="fu"&gt;$&lt;/span&gt;(&lt;span class="st"&gt;"h1"&lt;/span&gt;).&lt;span class="fu"&gt;first&lt;/span&gt;().&lt;span class="fu"&gt;rText&lt;/span&gt;(title);
}&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The &lt;code&gt;bind&lt;/code&gt; method proves to be very useful here. When we now change the active document’s title, the header and canvas elements will stay up to date automatically. Interestingly, when we set a new active document, the elements will also change and now track the new document. We install the data bindings only once, during canvas construction. Still, all changes from now on will be reflected without any additional code!&lt;/p&gt;
&lt;div class="sidenote haskell"&gt;
&lt;p&gt;So, why is &lt;code&gt;bind&lt;/code&gt; called &lt;code&gt;bind&lt;/code&gt;?&lt;/p&gt;
&lt;p&gt;Again, the function name finds its origin in Haskell. Bind is one of the primitive methods from Haskell’s &lt;code&gt;Monad&lt;/code&gt; type class. Reactive variables form a monad, which means we can flatten nested layers of reactivity into one.&lt;/p&gt;
&lt;p&gt;The correspondence between our JavaScript functions and the Haskell counterparts is shown below. Note that &lt;code&gt;join&lt;/code&gt; after &lt;code&gt;map&lt;/code&gt; is equivalent to &lt;code&gt;bind&lt;/code&gt;.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// JavaScript:              --  Haskell:&lt;/span&gt;

&lt;span class="ot"&gt;a&lt;/span&gt;.&lt;span class="fu"&gt;join&lt;/span&gt;()                    join a

&lt;span class="ot"&gt;a&lt;/span&gt;.&lt;span class="fu"&gt;bind&lt;/span&gt;(f)                   a &amp;gt;&amp;gt;= f
&lt;span class="ot"&gt;a&lt;/span&gt;.&lt;span class="fu"&gt;bind&lt;/span&gt;(f)                   &lt;span class="kw"&gt;do&lt;/span&gt; v &amp;lt;- a
                               g v
&lt;span class="ot"&gt;a&lt;/span&gt;.&lt;span class="fu"&gt;map&lt;/span&gt;(f).&lt;span class="fu"&gt;join&lt;/span&gt;()             &lt;span class="fu"&gt;join&lt;/span&gt; (fmap a f)&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We find &lt;code&gt;return&lt;/code&gt; (and &lt;code&gt;pure&lt;/code&gt;) as the constructor for reactive variables:&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// JavaScript:              --  Haskell:&lt;/span&gt;

&lt;span class="kw"&gt;new&lt;/span&gt; &lt;span class="fu"&gt;Reactive&lt;/span&gt;(a)             &lt;span class="kw"&gt;return&lt;/span&gt; a&lt;/code&gt;&lt;/pre&gt;
&lt;/div&gt;
&lt;h3 id="additional-reactive-combinators"&gt;Additional Reactive Combinators&lt;/h3&gt;
&lt;p&gt;Besides the basic composition operators (&lt;code&gt;lift&lt;/code&gt;, &lt;code&gt;map&lt;/code&gt;, &lt;code&gt;bind&lt;/code&gt;, and &lt;code&gt;join&lt;/code&gt;), we have some other combinators that make working with reactive values worthwhile. To give you an idea what kind of combinators are possible with a reactive programming library like ours, we list some interesting functions with an example or their usages.&lt;/p&gt;
&lt;ul&gt;&lt;li&gt;
&lt;p&gt;&lt;code&gt;wait&lt;/code&gt;: Wait for a condition on a reactive value to become true, run an action once and unlink immediately after.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// Alert once when the user signs out:&lt;/span&gt;
&lt;span class="ot"&gt;user&lt;/span&gt;.&lt;span class="fu"&gt;map&lt;/span&gt;(authenticated)
    .&lt;span class="fu"&gt;wait&lt;/span&gt;( (isAuth) =&amp;gt; !isAuth                    &lt;span class="co"&gt;// predicate&lt;/span&gt;
         , () =&amp;gt; &lt;span class="fu"&gt;alert&lt;/span&gt;(&lt;span class="st"&gt;"no longer logged in!"&lt;/span&gt;)    &lt;span class="co"&gt;// action&lt;/span&gt;
         );&lt;/code&gt;&lt;/pre&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;sample&lt;/code&gt;: A small combinator to sample changes over some period of time.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// Send a new query string to the server if the string&lt;/span&gt;
&lt;span class="co"&gt;// representation of our internal data model changes, but&lt;/span&gt;
&lt;span class="co"&gt;// at most once every 5 seconds:&lt;/span&gt;
&lt;span class="ot"&gt;queryData&lt;/span&gt;.&lt;span class="fu"&gt;map&lt;/span&gt;(computeQueryString)
         .&lt;span class="fu"&gt;sample&lt;/span&gt;(&lt;span class="dv"&gt;5000&lt;/span&gt;)
         .&lt;span class="fu"&gt;onchange&lt;/span&gt;(performQueryXHR);&lt;/code&gt;&lt;/pre&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;collapse&lt;/code&gt;: A bit like &lt;code&gt;sample&lt;/code&gt;, but runs once at the end of a JavaScript thread.&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// Render the username as the text contents of a &amp;lt;div&amp;gt;,&lt;/span&gt;
&lt;span class="co"&gt;// but render at most once per thread:&lt;/span&gt;
&lt;span class="fu"&gt;$&lt;/span&gt;(&lt;span class="st"&gt;"div.username"&lt;/span&gt;).&lt;span class="fu"&gt;rText&lt;/span&gt;(&lt;span class="ot"&gt;user&lt;/span&gt;.&lt;span class="fu"&gt;bind&lt;/span&gt;((u) =&amp;gt; &lt;span class="ot"&gt;u&lt;/span&gt;.&lt;span class="fu"&gt;name&lt;/span&gt;).&lt;span class="fu"&gt;collapse&lt;/span&gt;());&lt;/code&gt;&lt;/pre&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;code&gt;bimap&lt;/code&gt;: A bit like &lt;code&gt;map&lt;/code&gt;, but connects two values in both directions. We specify two functions, one for every direction. The invariant is that both functions need to form a proper isomorphism (or at least converge to one).&lt;/p&gt;
&lt;pre class="sourceCode javascript"&gt;&lt;code class="sourceCode javascript"&gt;&lt;span class="co"&gt;// Makes sure the 'halved' and 'doubled' variables&lt;/span&gt;
&lt;span class="co"&gt;// always keep there invariant:&lt;/span&gt;
&lt;span class="fu"&gt;bimap&lt;/span&gt;( halved
     , doubled
     , (a) =&amp;gt; a * &lt;span class="dv"&gt;2&lt;/span&gt;
     , (a) =&amp;gt; &lt;span class="ot"&gt;Math&lt;/span&gt;.&lt;span class="fu"&gt;round&lt;/span&gt;(a / &lt;span class="dv"&gt;2&lt;/span&gt;)
     );&lt;/code&gt;&lt;/pre&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;p&gt;Like you see here, those reactive combinators allow for a syntactically lightweight form of asynchronous programming.&lt;/p&gt;
&lt;h3 id="drawbacks"&gt;Drawbacks&lt;/h3&gt;
&lt;p&gt;Even with a few very simple JavaScript types and combinators we are able to build very powerful data abstractions. The Silk client uses this reactive paradigm a lot and we can safely say this benefits the quality of our code base a lot. Unfortunately there are also several drawbacks to using FRP like this:&lt;/p&gt;
&lt;ul&gt;&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Resource management can be tricky.&lt;/strong&gt; Every variable you link adds a node to the reactive graph, when the link isn’t need anymore you need to clean it up again. This requires some form of manual garbage collection. Cleaning up automatically is hard, because two linked values have a mutual references and JavaScript doesn’t have pointer finializers. We managed to build a system that can at least clean up intermediate unused parts of reactive networks, so minimal work of the library user is required. Also, it must be noted that cleaning up resources is equally hard in the case of normal event based systems, when using jQuery &lt;code&gt;on&lt;/code&gt; and &lt;code&gt;off&lt;/code&gt; for example.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Occasionally care is needed to prevent value oscillation.&lt;/strong&gt; When two values depend on eachother (with either a direct, or indirect cycle) values must converge, otherwise the network will echo value changes continuously. Non-converging networks isn’t a widely occurring problem, but when it does happen debugging can be painful.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Debugging can be surprising.&lt;/strong&gt; Reactive networks a very good at propagating data changes through every part of your application. In that sense they work a bit like events. Debugging problems with reactive values can be tricky because problems in one part of your application might be caused by seemingly unrelated pieces of code. This needs some getting used to. Like most problems in JavaScript, a static type system could greatly benefit us here. We’re are currently investigating using TypeScript for parts of our code base to improve reasoning about our connections between different parts of the product.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;&lt;h3 id="conclusion"&gt;Conclusion&lt;/h3&gt;
&lt;p&gt;Overall we’re very happy with our framework and would encourage other client side programmers to the explore the realm of FRP as well. Especially when building complicated user interfaces that need to keep up to date with frequent changes in the underlying data model, reactive data bindings can keep the amount of manual work down.&lt;/p&gt;
&lt;p&gt;Discuss on: &lt;a href="http://www.reddit.com/r/programming/comments/20ssoa/reactive_programming_in_javascript/"&gt;reddit&lt;/a&gt; or &lt;a href="https://news.ycombinator.com/item?id=7427706"&gt;hacker news&lt;/a&gt;&lt;/p&gt;</description><link>http://engineering.silk.co/post/80056130804</link><guid>http://engineering.silk.co/post/80056130804</guid><pubDate>Wed, 19 Mar 2014 06:06:00 -0400</pubDate><category>javascript</category><category>frp</category><category>reactive</category><category>haskell</category></item><item><title>Bump it up</title><description>&lt;p&gt;We just released a new version of your Cabal package tool &lt;strong&gt;bumper&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Bumper lets you transitively bump sets of packages that you are developing so you don&amp;rsquo;t have to manually edit version constraints.&lt;/p&gt;

&lt;p&gt;Silk consists of a lot of packages, modifying all the dependencies by hand turned into quite the chore so we needed a tool to automate the process of releasing new versions. The typical work flow is to make changes to a set of packages, run bumper, run bumper again with the same arguments along with &lt;code&gt;--dry-run&lt;/code&gt; piped to a script that uploads to our internal Hackage server, and finally push the changes to github (which notifies our build server).&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/silkapp/bumper"&gt;See the readme in the repository&lt;/a&gt; for more information on how &lt;strong&gt;bumper&lt;/strong&gt; works, or install it from &lt;a href="http://hackage.haskell.org/package/bumper"&gt;Hackage&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;a href="http://silk.co"&gt;Silk&lt;/a&gt; is hiring! Check out &lt;a href="http://jobs.silk.co"&gt;jobs.silk.co&lt;/a&gt; to see if there might be something for you or someone you know&lt;/em&gt;&lt;/p&gt;</description><link>http://engineering.silk.co/post/77905387928</link><guid>http://engineering.silk.co/post/77905387928</guid><pubDate>Wed, 26 Feb 2014 08:54:00 -0500</pubDate><category>haskell</category></item><item><title>Using DOM Mutation Events for node stack traces</title><description>&lt;p&gt;The Silk client is a single-page application with lots of moving parts. Being new to the codebase, I often find myself wondering when and where a certain page component is created. Basically: which function inserted this particular DOM element? There&amp;rsquo;s a simple trick to find out.&lt;/p&gt;

&lt;p&gt;Remember &lt;a href="https://developer.mozilla.org/en-US/docs/Web/Guide/Events/Mutation_events"&gt;DOM Mutation Events&lt;/a&gt;? Widely mocked for being &amp;ldquo;verbose, slow, and crashy,&amp;rdquo; they&amp;rsquo;re marked for deprecation in the DOM spec—but they still work, at least in Chrome and Firefox. And for this use case, their flaws are actually what make them useful: they let us hook into DOM node insertion with the call stack intact.&lt;/p&gt;

&lt;p&gt;This behavior is problematic for many use cases, and it incurs a performance penalty. So with the newer API, &lt;a href="https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver"&gt;Mutation Observers&lt;/a&gt;, handling is postponed and batched into entire subtree insertions, which is more efficient. But it also means useful info is lost; in particular, there&amp;rsquo;s no way to find out anything about the execution context of the code that inserted each node. With good old mutation events, it&amp;rsquo;s trivial!&lt;/p&gt;

&lt;p&gt;So the basic idea is just to attach a mutation event handler for DOM node insertion that captures the current stack trace and saves it for debugging purposes. I chose to save the stack trace as a data attribute on the node itself; this way I can see them easily in the browser&amp;rsquo;s DOM inspector. Here is the simplest way to do this (pardon the jQuery):&lt;/p&gt;

&lt;pre class="prettyprint lang-js"&gt;$(document).bind('DOMNodeInserted', function (e) {
  $(e.target).attr('data-trace', (new Error).stack);
});
&lt;/pre&gt;

&lt;p&gt;With this handler in place, every dynamically inserted DOM element gets a property with a stack trace from its point of insertion.&lt;/p&gt;

&lt;p&gt;With many such elements, the DOM inspector becomes cluttered with big stack traces. To clean it up a bit, we can do some substitutions on the stack trace. This is messy, since the format of the (non-standardized) Error#stack property differs between JavaScript engines. Here&amp;rsquo;s a quick hack that works in Chrome:&lt;/p&gt;

&lt;pre class="prettyprint lang-js"&gt;$(document).bind('DOMNodeInserted', function (e) {
  $(e.target).attr('data-trace',
    (new Error).stack
      .split("\n").splice(6).join("\n")
      .replace(/( +at )|\([^()]+\)/g,"")
      .replace(/\n/g,"/ "));
});
&lt;/pre&gt;

&lt;p&gt;This results in a one-line stack trace without source locations, skipping a few uninteresting lines related to the event handler itself. It&amp;rsquo;s useful if you only need a bit of context.&lt;/p&gt;

&lt;p&gt;To make this useful across browsers, one could use the &lt;a href="https://github.com/stacktracejs/stacktrace.js"&gt;Stacktrace.js&lt;/a&gt; library, which has a whole arsenal of clever regular expressions to get uniform stack traces. This is left as an exercise for the reader&amp;hellip;&lt;/p&gt;

&lt;p&gt;Another fun trick is to also add some styling to the dynamically-inserted nodes; jQuery UI&amp;rsquo;s &lt;a href="http://api.jqueryui.com/highlight-effect/"&gt;highlight effect&lt;/a&gt;, perhaps? A simple border is nice, too.&lt;/p&gt;

&lt;p&gt;Finally, this little debugging snippet (minus the jQuery) is also suitable as a bookmarklet, in case you want to debug a site without changing its source. &lt;a href="https://gist.github.com/mbrock/9048254"&gt;This is the bookmarklet I use for Chrome&lt;/a&gt;. Try it on your favorite site with dynamic content!&lt;/p&gt;

&lt;p&gt;Since DOM mutation events are thoroughly deprecated, this trick has a tragic destiny. It would be a nice feature to have built into the browser debugging tools. But until that day, let us celebrate the sunset of this star-crossed API.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;&lt;a href="http://silk.co"&gt;Silk&lt;/a&gt; is hiring! Check out &lt;a href="http://jobs.silk.co"&gt;jobs.silk.co&lt;/a&gt; to see if that might be something for you or someone you know&lt;/em&gt;&lt;/p&gt;</description><link>http://engineering.silk.co/post/76945930473</link><guid>http://engineering.silk.co/post/76945930473</guid><pubDate>Mon, 17 Feb 2014 05:56:00 -0500</pubDate><category>javascript</category><category>debugging</category></item><item><title>We're hiring!</title><description>&lt;p&gt;&lt;a href="http://www.silk.co"&gt;Silk&lt;/a&gt; is growing and we are looking for talented people to join us. Silk makes it easy for people to create sites with content that is easy to query, visualize and share. We&amp;rsquo;re a startup with top-tier international investors, thousands of sites running on our platform, and a big vision to make the web more useful for people and machines.&lt;/p&gt;

&lt;p&gt;&lt;img src="http://images.silk.co/s/api/v1.12.2/site/current/file/id/9321824e-3282-4941-8c79-fc187be99640/" alt="Come work at Silk"/&gt;&lt;/p&gt;

&lt;p&gt;We&amp;rsquo;re looking for people who are self-motivated, make the right decisions on their own and can deal with the occasional startup chaos.&lt;/p&gt;

&lt;p&gt;Check out &lt;a href="http://jobs.silk.co"&gt;jobs.silk.co&lt;/a&gt; for more information and all open positions, or follow a link to one of the positions below. If you know someone who might be interested, we would love it if you could share this post.&lt;/p&gt;

&lt;iframe src="http://is.gd/MwRUMs" style="height:400px;width:600px;border:0;"&gt;&lt;/iframe&gt;

&lt;div style="margin-top:10px;font-size:12px;color:gray;"&gt;Data from &lt;a href="http://jobs.silk.co"&gt;jobs.silk.co&lt;/a&gt;&lt;/div&gt;</description><link>http://engineering.silk.co/post/66369046281</link><guid>http://engineering.silk.co/post/66369046281</guid><pubDate>Fri, 08 Nov 2013 08:59:16 -0500</pubDate><category>work at Silk</category><category>haskell engineer</category><category>frontend engineer</category><category>startup job</category></item><item><title>Silk raises $1.6MM in a seed round led by NEA</title><description>&lt;p&gt;Today we are very excited to announce the completion of a $1.6 million seed funding round led by New Enterprise Associates (NEA) with participation from existing Silk investor Atomico and three additional investors. The raised amount will be used to expand our team and to keep improving our product and infrastructure.&lt;/p&gt;
&lt;p&gt;We are thrilled to partner up with &lt;a href="http://www.nea.com/"&gt;NEA&lt;/a&gt;, the biggest US venture capital firm with a committed capital of $13 billion, who happened to &lt;a href="http://www.nea.com/NewsAndEvents/PressReleases.aspx?y=2012"&gt;announce&lt;/a&gt; their new $2.6 billion fund last week. Also, its a great confirmation that our existing investor &lt;a href="http://www.atomico.com/"&gt;Atomico&lt;/a&gt; (the fund co-founded by Skype co-founder Niklas Zennstrm) followed up on &lt;a href="http://about.silkapp.com/page/Silk%20completes%20%E2%82%AC320%2C000%20(%24475%2C000)%20funding%20round%20led%20by%20Atomico"&gt;their investment from last year&lt;/a&gt; by participating in this round as well.&lt;/p&gt;
&lt;p&gt;Additional backers in the round are Anil Hansjee, a well known angel and advisor to many companies, including Fon, who previously served as Google&amp;rsquo;s head of corporate development EMEA; Jens Christensen, best known for his role as CEO of Ellerdale (acquired by Flipboard); and Philippe Cases, an angel investor and entrepreneur, currently CEO of Spoke Software.&lt;/p&gt;
&lt;h3 id="whatareweworkingon"&gt;What are we working on?&lt;/h3&gt;
&lt;p&gt;For those who dont know us: Silk is a place to create, share and find structured information. Share those &lt;a href="http://madrid.silkapp.com/"&gt;special places in Madrid&lt;/a&gt;, the &lt;a href="http://mybacklog.silkapp.com/"&gt;video games you are playing&lt;/a&gt; or the &lt;a href="http://is.gd/JZtXJH"&gt;length of your children as they grow&lt;/a&gt;. Public transport companies can use Silk to find patterns in their delays. Human rights organizations can keep track of violations by governments. Investors use Silk to &lt;a href="http://nea.silkapp.com/"&gt;publish their investment portfolio&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Weve learned that Silk comes in handy for information that:&lt;/p&gt;
&lt;ul&gt;&lt;li&gt;has some structure,&lt;/li&gt;
&lt;li&gt;is valuable to share in real time, but&lt;/li&gt;
&lt;li&gt;should also remain available to be used later&lt;/li&gt;
&lt;/ul&gt;&lt;h3 id="example"&gt;Example&lt;/h3&gt;
&lt;p&gt;A current example would be a Silk site about Olympic medals. You may want to keep track of all Olympic medals as they are won, or perhaps filter on your country or on a sport you are interested in. The information should also remain available to generate overviews from, such as a map of the world indicating the number of medals each country has won or simply a list of medals won by The Netherlands.&lt;/p&gt;
&lt;p&gt;We are very excited to be working with both Atomico and NEA, as this will help us to further build on our vision. If you are interested in using Silk, &lt;a href="http://www.silkapp.com/"&gt;sign up here&lt;/a&gt;. Organizations who want to use Silk professionally, feel free to &lt;a href="http://www.silkapp.com/feedback"&gt;get in touch&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;And finally, yes, &lt;a href="http://about.silkapp.com/page/Jobs"&gt;we are hiring&lt;/a&gt;!&lt;/p&gt;</description><link>http://engineering.silk.co/post/32258858584</link><guid>http://engineering.silk.co/post/32258858584</guid><pubDate>Wed, 01 Aug 2012 09:42:00 -0400</pubDate><category>silk blog</category></item><item><title>Multiple IP addresses on Amazon EC2</title><description>&lt;p&gt;Last week, Amazon announced support for multiple IP’s for instances inside a VPC. This can be useful in many situations where a single IP for a machine is not enough. One example is SSL endpoints, which can only pass on certificates based on the IP listened on, and don’t work on hostname, for example. (&lt;a href="http://aws.typepad.com/aws/2012/07/multiple-ip-addresses-for-ec2-instances-in-a-virtual-private-cloud.html"&gt;Announcement here&lt;/a&gt;)&lt;/p&gt;
&lt;p&gt;Another reason to use multiple IP’s is if you want to access a service that is limited to a number of requests per IP. Having more IP’s will give you a higher limit, but creating new instances just for extra IP’s can get expensive quickly.&lt;/p&gt;
&lt;p&gt;While Amazon &lt;a href="http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-instance-addressing.html"&gt;has updated their instance IP documentation&lt;/a&gt;, using multiple IP addresses may still be a bit confusing for people. I hope this post will be able to explain the basics of Elastic IP’s and how to use multiple IP’s on a single instance.&lt;/p&gt;
&lt;h3&gt;Pricing&lt;/h3&gt;
&lt;p&gt;Amazon changed their IP pricing structure with this update. Prior to the update, used Elastic IP’s would cost nothing, and unused IP’s would cost around $10 / month. This monthly fee was introduced to discourage retaining unused IP’s – IP addresses get scarcer every year, and will be interesting to see what will do once they cannot acquire new IP blocks.&lt;/p&gt;
&lt;p&gt;With this update, Amazon has updated their pricing as follows: all IP’s cost $0.005 / hour (around $3.65 / month), unless the IP is assigned as primary IP of the first network interface of an instance, in which case it’s free.&lt;/p&gt;
&lt;h3&gt;Limits&lt;/h3&gt;
&lt;p&gt;When you start with your first VPC, you are limited to a maximum of 5 IP’s, probably because of the IP scarcity. However, it’s easy to request a higher limit from AWS, and it shouldn’t be a problem to raise your limit to for example 50 IP’s. You might want to do that before following the rest of this guide. Amazon published &lt;a href="http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/instance-types.html#AvailableIpPerENI"&gt;a list of interfaces and IP limits per instance&lt;/a&gt;.&lt;/p&gt;
&lt;h3&gt;How Elastic IPs work&lt;/h3&gt;
&lt;p&gt;By default, instance in the VPC will only get an IP from a private subnet within the VPC, usually somewhere in the 10.0.0.0/8 subnet. This means that while those instances can access other instances within that same subnet, they won’t be able to connect to the wider internet.&lt;/p&gt;
&lt;p&gt;One way to give every machine access is to setup a NAT machine which all machines use to route traffic through. We won’t go any deeper into this option.&lt;/p&gt;
&lt;p&gt;The alternative is to give every machine it’s own world-routable Elastic IP address. Amazon will route traffic to the elastic ip to your internal IP. This makes this setup simpler from a network topology perspective.&lt;/p&gt;
&lt;p&gt;Routing in the VPC works through routing tables which are set up for each subnet. You can view your routing table in the VPC tab of the AWS console. By default, requests within your subnet are routed locally, and requests outside of that range are routed through an ‘internet gateway’.&lt;/p&gt;
&lt;p&gt;For example, let’s say that you have an instance with IP 10.0.0.1. It has an Elastic IP associated with it, let’s say 1.2.3.4. Now whenever you access other internal machines on the 10.0.0.0/8 subnet, your request will originate from your private IP and nothing will happen to the traffic.&lt;/p&gt;
&lt;p&gt;When you access a machine outside of your subnet, the packets are oruted through the internet gateway, a VPC device with a name like igw-9d7534f2. The gateway (which isn’t an actual EC2 instance, just an opaque AWS system) accepts the request and then looks up the internal IP from which the request originates, and checks if this IP is associated with any Elastic IP’s. If so, the request is rewritten to appear as if the origin is the elastic IP, and then send out over the internet. When a package returns to the gateway with an elastic IP as destination, the gateway checks if the elastic IP is associated with an internal IP, and if so rewrites the package to that IP. The package is then forwarded into the private subnet.&lt;/p&gt;
&lt;p&gt;This is how the one-to-one NAT of elastic IP’s works. The advantage of this approach is that machines within the subnet need to have no knowledge of their elastic IP’s. This makes it easy to switch elastic IP’s of a running instance, since the only thing that needs to be updated is the mapping table on the internet gateway. The external IP can change many times; since the packages are rewritten to use internal IP’s before they arrive at the instance, the instance will never know this.&lt;/p&gt;
&lt;h3&gt;Multiple Network Interfaces&lt;/h3&gt;
&lt;p&gt;Before we dive into the new system Amazon introduced, let’s review what was available previously.&lt;/p&gt;
&lt;p&gt;In the past, Amazon added an option to add &lt;a href="http://aws.typepad.com/aws/2011/12/new-elastic-network-interfaces-in-the-virtual-private-cloud.html"&gt;multiple network interfaces to a VPC instance&lt;/a&gt;. These interfaces appear as separate ethernet cards on your machine, and will have a separate internal IP. Separate interfaces also means you need to set up your routing between these interfaces correctly: if you send out a packet over the wrong interface, the packet will simply be dropped.&lt;/p&gt;
&lt;p&gt;These network interfaces were introduced to give you the option of connecting different subnets together: you can have one interface in each subnet, and then route trafic between them as you like. Since each subnet has its own IP range, it is easy to route traffic to the right network interface.&lt;/p&gt;
&lt;p&gt;For each internal IP, it is possible to associate an external Elastic IP. While this is possible, you get into a bit of a tricky routing situation now. Both internal IP’s (on eth0 and eth1) have an associated external IP, and should be able to make requests to the wider internet, by having the internet gateway translate their internal IP to the Elastic IP. However, in order to do this correctly, the requests have to be send out over the correct interface. Amazon itself admits that this is not simple:&lt;/p&gt;
&lt;p&gt;I should note that attaching two public ENIs to the same instance is not the right way to create an EC2 instance with two public IP addresses. There’s no way to ensure that packets arriving via a particular ENI will leave through it without setting up some specialized routing. We are aware that a lot of people would like to have multiple IP addresses for a single EC2 instance and we plan to address this use case in 2012.&lt;/p&gt;
&lt;p&gt;By default, Linux has a routing table which routes all traffic outside of your local subnet to a single interface. You can look this up with route:&lt;/p&gt;
&lt;pre&gt;# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.0.0.1        0.0.0.0         UG    0      0        0 eth0
default         10.0.0.1        0.0.0.0         UG    100    0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 eth0
10.0.0.0        *               255.255.255.0   U     0      0        0 eth1
&lt;/pre&gt;
&lt;p&gt;Here, all internet-wide traffic will go out over interface eth0. Even packets with a source ip of the eth1 interface will go out over eth0, and then will be silently discarded instead of being NAT’d to the right elastic IP. While it’s possible to do source-based routing to fix this, this isn’t trivial to set up (note that you need to run dhclient after adding another interface to a machine):&lt;/p&gt;
&lt;pre&gt;# ifconfig | grep eth\\\|inet\ 
eth0      Link encap:Ethernet  HWaddr 02:86:10:77:7f:fe  
          inet addr:10.0.0.76  Bcast:10.0.0.255  Mask:255.255.255.0
eth1      Link encap:Ethernet  HWaddr 02:86:10:65:fd:31  
          inet addr:10.0.0.226  Bcast:10.0.0.255  Mask:255.255.255.0

# curl --interface 10.0.0.76 ifconfig.me
107.23.13.138
# curl --interface 10.0.0.226 ifconfig.me
&amp;lt;&amp;lt; TIMEOUT &amp;gt;&amp;gt;&amp;gt;

# ip rule add from 10.0.0.226 table out2
# ip route add default via 10.0.0.1 dev eth1 table 2
# ip route flush cache

# curl --interface 10.0.0.226 ifconfig.me
107.23.13.127
# curl --interface 10.0.0.76 ifconfig.me
107.23.13.138
&lt;/pre&gt;
&lt;p&gt;If you are OK with being a bit more limited in how many IP addresses you can use, it’s easier if you can assign multiple addresses to the same interface. The rest of this blog post assumes you will only use multiple IP’s on a single interface, but if you really need to, the above should be a good starter on how to add more interfaces.&lt;/p&gt;
&lt;h3&gt;Multiple IP Addresses&lt;/h3&gt;
&lt;p&gt;What Amazon announced last week is support for multiple internal IP’s on the same interface. This way we can avoid some of the routing nasties from above: all packets will be sent over eth0, regardless of which IP you have. Because of the one-to-one mapping of public and private addresses, you first need to add some new private addresses to one of your instances.&lt;/p&gt;
&lt;p&gt;Amazon’s API has been &lt;a href="http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/ApiReference-query-AssignPrivateIpAddresses.html"&gt;updated to support assigning secondary private IP’s&lt;/a&gt;, but for this example it’s easier to just go to the AWS Console. In the EC2 tab, go to Instances. Find the instance you want to update, right-click and choose “Manage Private IP Adresses”.&lt;/p&gt;
&lt;p&gt;&lt;img alt="" src="http://about.silkapp.com/s/api/v1.2.1/file/id/30f08b92-118d-4dbd-ba3a-72b531f5eb09"/&gt;&lt;/p&gt;
&lt;p&gt;&lt;img alt="" src="http://about.silkapp.com/s/api/v1.2.1/file/id/5f3123de-1318-4acd-bdd4-6913e54927e8"/&gt;&lt;/p&gt;
&lt;p&gt;You’ll now see that the instance has two private IP’s in your subnet (since an interface is locked to specific subnet, all IP’s on that interface have to fall within that same subnet). Nice! However, if you now check your instance:&lt;/p&gt;
&lt;pre&gt;# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 02:86:10:7b:e4:f5  
          inet addr:10.0.0.34  Bcast:10.0.0.255  Mask:255.255.255.0
&lt;/pre&gt;
&lt;p&gt;It won’t have the new IP yet. Let’s try retrieving it over DHCP!&lt;/p&gt;
&lt;pre&gt;root@ip-10-0-0-34:/# dhclient -d eth0
Listening on LPF/eth0/02:86:10:7b:e4:f5
Sending on   LPF/eth0/02:86:10:7b:e4:f5
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3
DHCPREQUEST of 10.0.0.34 on eth0 to 255.255.255.255 port 67
DHCPOFFER of 10.0.0.34 from 10.0.0.1
DHCPACK of 10.0.0.34 from 10.0.0.1
&lt;/pre&gt;
&lt;p&gt;Nope, we only get our primary private ip! With DHCP you can only get a single address. In the future, perhaps Amazon will add support for multiple IP’s using some client identifier or another mechanism, but for now you’ll have to add the address manually. Hopefully Ubuntu’s &lt;a href="https://help.ubuntu.com/community/CloudInit"&gt;cloud-init&lt;/a&gt; will add support for this in the future, so we don’t have to do it ourselves.&lt;/p&gt;
&lt;p&gt;Amazon &lt;a href="http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-instance-addressing.html#StepTwoConfigOSLinux"&gt;suggests creating a new virtual interface&lt;/a&gt; and then bringing that up. While that will work, it means you store your interface state on your hard drive, and any changes you make on AWS you will have to propagate to these files.&lt;/p&gt;
&lt;p&gt;For testing, we can do something simpler:&lt;/p&gt;
&lt;pre&gt;MAC_ADDR=$(ifconfig eth0 | sed -n 's/.*HWaddr \([a-f0-9:]*\).*/\1/p')
IP=($(curl &lt;a href="http://169.254.169.254/latest/meta-data/network/interfaces/macs/%24MAC_ADDR/local-ipv4s"&gt;http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC_ADDR/local-ipv4s&lt;/a&gt;))
for ip in ${IP[@]:1}; do
    echo "Adding IP: $ip"
    ip addr add dev eth0 $ip/24
done
&lt;/pre&gt;
&lt;p&gt;This will check the &lt;a href="http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html"&gt;EC2 instance meta-data&lt;/a&gt; for all internal IPs associated with eth0, and adds all the secondary IP’s.&lt;/p&gt;
&lt;p&gt;You can now asociate another elastic ip to the secondary IP in the AWS Console:&lt;/p&gt;
&lt;p&gt;&lt;img alt="" src="http://about.silkapp.com/s/api/v1.2.1/file/id/370801c1-10d3-4b1d-8608-d69745c8b9cc"/&gt;&lt;/p&gt;
&lt;p&gt;And now you have multiple external IP’s!&lt;/p&gt;
&lt;pre&gt;root@ip-10-0-0-34:/# curl --interface 10.0.0.58 ifconfig.me
107.23.13.127
root@ip-10-0-0-34:/# curl --interface 10.0.0.34 ifconfig.me
107.23.10.202
&lt;/pre&gt;
&lt;p&gt;While this works for testing, it’s not persistent across reboot. You can create an upstart or cloud-init script to do that for you. Even then, ip addresses aren’t automatically added when you add them through EC2. I’m not sure if there’s a good way to do that. Finally, this script also won’t remove any local addresses you may have removed in the meantime.&lt;/p&gt;
&lt;p&gt;However, it should be a good place to start!&lt;/p&gt;</description><link>http://engineering.silk.co/post/31923247961</link><guid>http://engineering.silk.co/post/31923247961</guid><pubDate>Mon, 09 Jul 2012 00:00:00 -0400</pubDate><category>engineering</category><category>amazon</category></item><item><title>Today we open up Silk</title><description>&lt;p&gt;Today is a good day: we just launched the Silk editor. Everyone can now create their own Silk site.&lt;/p&gt;
&lt;p&gt;Over the past few months Silk has been in private beta. Only the first 10,000 users could sign up and create a Silk site. It has been both exciting and insightful to see the first groups of users creating Silk sites. It teaches us what aspects of Silk are appreciated and where we can improve.&lt;/p&gt;
&lt;p&gt;We have seen interest from professional publishers and data journalists, but also from businesses and personal users. During the private beta, we allowed users to use the Silk web editor to structure existing content and to build entirely new websites. Many users tried our &lt;a href="http://about.silkapp.com/page/How%20do%20I%20import%20a%20CSV%20file"&gt;importer tools&lt;/a&gt; to turn existing data sets into structured Silk sites. If you are curious by now, you may &lt;a href="http://www.silkapp.com/"&gt;sign up here&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id="astructuredweb"&gt;A structured web&lt;/h3&gt;
&lt;p&gt;If you have been following us, you know that Silk sites are websites that can be understood by both humans and computers. This enables an entirely new kind of web search and new powerful ways to visualize information.&lt;/p&gt;
&lt;p&gt;The Silk sites that have been created so far demonstrate how Silk impacts the way information is consumed. For example, the &lt;a href="http://world.silkapp.com/"&gt;Countries of the world&lt;/a&gt; site contains information about the various countries in the world, just like Wikipedia does. The site contains text like any other website, but allows you to interact with the content more deeply. It is like a database that anyone can use. Want to know which countries have a life expectancy below 60, for example? It is easy to get the answer in a &lt;a href="http://is.gd/Fk3l5A"&gt;table&lt;/a&gt;, &lt;a href="http://is.gd/O3rFjj"&gt;chart&lt;/a&gt; or &lt;a href="http://is.gd/DmABvC"&gt;map&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Now that we have launched, more new Silk sites will be created each day. You can &lt;a href="http://www.silkapp.com/"&gt;create your own site here&lt;/a&gt;. Making information available through Silk will make the web a better place, one Silk site at a time.&lt;/p&gt;</description><link>http://engineering.silk.co/post/31923334292</link><guid>http://engineering.silk.co/post/31923334292</guid><pubDate>Thu, 10 May 2012 00:00:00 -0400</pubDate><category>engineering</category><category>silk blog</category></item><item><title>Erik's talk at Functional Programming eXchange 2012</title><description>&lt;p&gt;On the 16th of March, Skills Matter organized the Third Functional Programming eXchange, an annual Functional Programming conference. A day of talks, open-space discussions and brainstorming on Functional Programming. Third speaker on the agenda was our own Erik Hesselink.&lt;/p&gt;
&lt;p&gt;&lt;a href="http://about.silkapp.com/page/Erik%20Hesselink"&gt;Erik&lt;/a&gt; spoke about the uses of and experiences with functional programming at Silk. Our backend uses &lt;a href="http://about.silkapp.com/page/Haskell"&gt;Haskell&lt;/a&gt; to provide an &lt;a href="http://about.silkapp.com/page/API"&gt;API&lt;/a&gt;. It provides functionality from simple account creation to querying a custom graph database. Haskell&amp;rsquo;s types and abstraction power have enabled us to build complex functionality with very few errors. However, laziness and performance are tricky to understand and get under control. &lt;/p&gt;
&lt;p&gt;Silk&amp;rsquo;s frontend is written in &lt;a href="http://about.silkapp.com/page/Javascript"&gt;Javascript&lt;/a&gt;. With first-class functions, functional programming techniques can readily be used here. For example, we use reactive values to statically model the dependency between application state and the user interface. However, runtimes are not optimized for this, and performance in the browser can be limited.&lt;/p&gt;
&lt;p&gt;&lt;a href="http://skillsmatter.com/podcast/home/silkapp-a-case-study-in-creating-rich-internet-apps-in-haskell/js-3880"&gt;View Erik&amp;rsquo;s talk here&lt;/a&gt; (46 minutes) or read about the &lt;a href="http://skillsmatter.com/event/scala/functional-programming-exchange-2012"&gt;Functional Programming Exchange 2012&lt;/a&gt;. &lt;/p&gt;</description><link>http://engineering.silk.co/post/31922904552</link><guid>http://engineering.silk.co/post/31922904552</guid><pubDate>Mon, 02 Apr 2012 00:00:00 -0400</pubDate><category>haskell</category><category>engineering</category><category>javascript</category></item><item><title>A RESTful API with automatically generated bindings</title><description>&lt;p&gt;Silk is mainly built on two languages: Haskell and Javascript. We use Haskell for our back-end to get a stable, type-checked core and Javascript to bring the content to our users. While both these languages are high-level programming languages, the communication via the web forces us to write much lower-level HTTP requests and marshaling code. At Silk, we make an effort to avoid writing low-level code manually. A previous result of this are our generic XML/JSON picklers as described in &lt;a href="http://about.silkapp.com/page/Writing%20a%20generic%20XML%20pickler"&gt;this previous post&lt;/a&gt;. Since then we have not been idle and we took the code automation one step further: We now generate our &lt;a href="http://api.silkapp.com/"&gt;documentation&lt;/a&gt; and Javascript wrappers directly from our own Haskell web-EDSL.&lt;/p&gt;
&lt;p&gt;The benefits of this approach are obvious. Instead of having to update documentation and javascript wrappers on every web API change, these parts are always kept up to date and correct. This saves work and avoids errors.&lt;/p&gt;
&lt;p&gt;The heart of this system is our Haskell web-API-EDSL. This system allows the programmer to define API resources in a declarative way together with the associated functionality. The EDSL forces the resources to be constructed in a RESTful way. The EDSL constructs an abstract representation of web resources, which we can then interpret to generate the following functionality:&lt;/p&gt;
&lt;ul&gt;&lt;li&gt;We construct the actual web-API using Happstack. Because we kept the API abstract, we could easily switch out Happstack for another framework.&lt;/li&gt;
&lt;li&gt;We generate documentation which gives an overview of what entry points the API has and what parameters/methods entry points can be called with.&lt;/li&gt;
&lt;li&gt;We generate a javascript wrapper which simplifies access to the API resources.&lt;/li&gt;
&lt;/ul&gt;&lt;p&gt;All this is best illustrated with an example. We could for example define a User API resource in the following way:&lt;/p&gt;
&lt;pre&gt;user :: Resource Root WithUser User
user = mkResource 
  { identifier     = "user"
  , multiGetBy     = [ ("search",  search)  ]
  , singleGet      = [ ("current", current) ]
  , singleGetBy    = [ ("email",   byEmail) ]
  , singleCreate   = Just create
  , singleDelete   = Just delete
  }
&lt;/pre&gt;
&lt;p&gt;We can bundle such resources into an API:&lt;/p&gt;
&lt;pre&gt;myApi = 
  api ---/ user --/ permissions
                --/ subscriptions
      ---/ site
&lt;/pre&gt;
&lt;p&gt;This also shows that resources can be nested. We use a monad stack to allow child resources to access their parents&amp;rsquo; context information. We could now request the permissions of a certain user by requesting the following URL:&lt;/p&gt;
&lt;pre&gt;/user/email/e@mail.com/permissions
&lt;/pre&gt;
&lt;p&gt;This would list all permissions for the user with email adress e@mail.com. A sneak peek of what the documentation looks like for such URLs can be found &lt;a href="http://api.silkapp.com/doc/latest"&gt;here&lt;/a&gt;. All information on this page is automatically generated, including the examples and schemas for XML and JSON.&lt;/p&gt;
&lt;p&gt;We could also access this resource using our Javascript API as follows:&lt;/p&gt;
&lt;pre&gt;silkApi.User.byEmail("e@mail.com).Permissions.list()
&lt;/pre&gt;
&lt;p&gt;We are currently in the process of constructing similar wrappers for Ruby and Haskell. We are also working on finalizing the base EDSL and we plan to open source it in the near future.&lt;/p&gt;</description><link>http://engineering.silk.co/post/31922845283</link><guid>http://engineering.silk.co/post/31922845283</guid><pubDate>Thu, 09 Feb 2012 00:00:00 -0500</pubDate><category>haskell</category><category>engineering</category><category>javascript</category><category>ruby</category></item><item><title>Testing Javascript</title><description>&lt;p&gt;At typLAB, we primarily use two programming languages: Javascript and Haskell. Haskell has a compiler, strong static typing, and is pure (no state or side-effects) by default. Javascript, on the other hand, is interpreted, dynamically typed, and, when using objects, uses lots of state (since objects are an encapsulation of state). This makes testing Javascript more important: otherwise, a typo in a variable name might go undetected until someone uses our product.&lt;/p&gt;
&lt;p&gt;To test our code, we need a couple of things: we need to write tests, we need a framework to run our tests in, and we need to automatically run this framework regularly. The number of Javascript test frameworks is large these days, and we started by reviewing a few. We finally settled on &lt;a href="http://docs.jquery.com/QUnit"&gt;QUnit&lt;/a&gt;. It provides a few simple functions to write tests, and a clean and useful page to present the results.&lt;/p&gt;
&lt;p&gt;We then integrated QUnit into our work flow. We use a simple, home grown module system to write our Javascript. It uses keywords like &lt;code&gt;Module&lt;/code&gt;, &lt;code&gt;Class&lt;/code&gt; and &lt;code&gt;Static&lt;/code&gt; to easily build modular code. We defined a new annotation, &lt;code&gt;Test&lt;/code&gt;, to define a test in a module. All these tests are automatically collected. A separate test runner module imports the modules to test and calls a single function to run all tests in a module.&lt;/p&gt;
&lt;p&gt;This provided us with a simple way to write tests. But if this was all, people would forget to run the tests, the tests would start failing without anyone seeing it, and the effort to write the tests would be wasted. To prevent this from happening, we wanted to integrate the running of tests in our work flow. Luckily, &lt;a href="http://ejohn.org/"&gt;John Resig&lt;/a&gt; has written &lt;a href="http://testswarm.com/"&gt;TestSwarm&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Traditionally, integrating Javascript tests in a build process or as a commit hook has been difficult, since Javascript generally depends on the browser to run. Different browsers have different quirks that might make your Javascript behave differently, and your code might also depend on theDOM, which is only available in a browser. However, you don&amp;rsquo;t want your developer machine to start many different browsers on each build or commit, and your server might not even have aGUI.&lt;/p&gt;
&lt;p&gt;TestSwarm is a solution to this problem. It provides a central server, and new tests to be run can be pushed to this server. Clients willing to run tests can also connect to it, and are given tests to run. Results are reported back to the server. This allows TestSwarm to build reports, showing test results per browser for each test run. In our case, our server pushes a new test run to the server on each commit set that is pushed. We have a few browsers open pointing to our TestSwarm instance, which run these tests. This means that testing doesn&amp;rsquo;t get in our way, but if something breaks, we can easily see which commit broke it and on which browsers.&lt;/p&gt;
&lt;p&gt;The final piece in our testing puzzle takes its inspiration from the Haskell world. &lt;a href="http://www.cs.chalmers.se/~rjmh/QuickCheck/"&gt;QuickCheck&lt;/a&gt; is a tool that performs randomized testing. You supply a function testing a certain property of your code, and it generates random inputs of increasing size. For example, if you have written a function reversing a list, you can state the property (function) that given a list (the input argument which is randomly generated), reversing it twice produces the original list.&lt;/p&gt;
&lt;p&gt;A &lt;a href="http://willowbend.cx/2009/12/05/qc-js-quickcheck-javascript/"&gt;Javascript version of QuickCheck&lt;/a&gt; exists, and we had a good use case. We use so-called reactive lists, that is, lists which can be connected to keep each other updated. For example, we can create two of these lists, and connect one to the other saying it is always the reverse of the first. When elements are inserted into the first, it sends an event to the second, which processes it, and generates an appropriate insert for the reversed list. The interesting thing is that we have an easily testable static property. Regardless of the mutations on the original list, at any point in time, the reverse of its values must be the same as the values in the output list.&lt;/p&gt;
&lt;p&gt;QuickCheck has quickly proven its worth in testing these reactive list functions. Since it generates random values of increasing size, it quickly and reliably finds all the corner cases where bugs like to hide. We&amp;rsquo;ve written a small helper function around QuickCheck to integrate it with QUnit, so that we can write code like this:&lt;/p&gt;
&lt;pre&gt;function splitTest ()
{
 var l = new R.List();
 var splitfun = function (e) { return e === 1; }
 var s = l.split(splitfun);
 quickcheck([arbListOp(arbRange(0, 5))],
   function (c, op)
   {
     op(l);
     c.assert(QUnit.equiv(A.split(splitfun, l.unR()), s.unR()));
   });
}
&lt;/pre&gt;
&lt;p&gt;This creates two lists: an input list &lt;code&gt;l&lt;/code&gt;, and an output list &lt;code&gt;s&lt;/code&gt; that is the result of splitting the input list at elements equal to 1. We then call the &lt;code&gt;quickcheck&lt;/code&gt; function, which is our helper function. It takes a list of generators as a first argument. In Haskell, these generators can be derived from the types, but here, we have to supply them. In this case, we generate operations on reactive lists. This generator for list operations takes a generator as an argument, which it uses to generate list elements to insert. In this case we generate random integers between 0 and 4 to make sure we get at least a few 1&amp;rsquo;s.&lt;/p&gt;
&lt;p&gt;The second argument to the &lt;code&gt;quickcheck&lt;/code&gt; function is the test function to run. It gets a QuickCheck object as its first argument, followed by the output from all the generators. In this case it gets a function representing an operation on a reactive list. We perform this operation on the input list by calling the function. We then check if the values in the input list (the function &lt;code&gt;unR&lt;/code&gt; performs a deep conversion from reactive to normal lists) split on 1&amp;rsquo;s are equal to the values in the output list. Here, &lt;code&gt;A.split&lt;/code&gt; is a static splitting function, and &lt;code&gt;QUnit.equiv&lt;/code&gt; performs a structural equality check on arrays.&lt;/p&gt;
&lt;p&gt;Thanks to the availability of these three open source tools, setting all this up only took a few days, and has provided us with a testing setup that works, but doesn&amp;rsquo;t get in our way. In the future, we will probably also start testing DOM and user interface related code; hopefully, that will work&lt;/p&gt;</description><link>http://engineering.silk.co/post/31922696209</link><guid>http://engineering.silk.co/post/31922696209</guid><pubDate>Tue, 23 Feb 2010 00:00:00 -0500</pubDate><category>engineering</category><category>javascript</category></item><item><title>Reinventing XSLT in pure Javascript</title><description>&lt;p&gt;In this post we will explore some boundaries of functional programming in Javascript and show how easy it is to implement a set of combinators that can express functions similar to queries in XPath and similar to transformations in XSLT. We call the result a &lt;em&gt;combinator&lt;/em&gt; library because we implement a few primitive queries and transformations and allow combining these into bigger ones using some basic composition functions. As we will show, all functions will follow more or less the same structure.&lt;/p&gt;
&lt;p&gt;This post is really about Javascript, which will be the target language of this library. But most of the techniques and underlying thoughts actually come from a statically typed functional programming background. While reading this post it might be interesting to continuously keep in mind the types of the functions, which makes it much easier to understand what is going on and how this framework might be extended with more interesting transformations.&lt;/p&gt;
&lt;blockquote&gt;&lt;em&gt;&lt;strong&gt;For the Haskell programmer:&lt;/strong&gt;&lt;/em&gt; the framework we are about to describe here is very similar to the list arrows of the &lt;a href="http://hackage.haskell.org/package/hxt"&gt;Haskell XML Toolkit&lt;/a&gt;. Looking at the documentation of this package might gain some additional insight.&lt;/blockquote&gt;
&lt;p&gt;Some functions we will see are selection functions that can be used to select parts of a document, other functions can be seen as filtering functions that exclude parts of the output, some are creation functions that introduce new structure in the output. In these post we call all of these function just &lt;em&gt;transformations&lt;/em&gt;. The resulting framework is some hybrid comparable to XPath and XSLT.&lt;/p&gt;
&lt;h2&gt;Primitive transformations&lt;/h2&gt;
&lt;p&gt;Let&amp;rsquo;s first look at the structure of the transformations performed in XPath and XSLT. Both languages can be seen as ways of describing a function that takes &lt;em&gt;one input&lt;/em&gt;, the context node, and produces &lt;em&gt;a list of outputs&lt;/em&gt;, a node set. Primitive examples of such transformation functions are: &lt;code&gt;getChildren&lt;/code&gt; (&lt;code&gt;/*&lt;/code&gt; in XPath), &lt;code&gt;getParent&lt;/code&gt; (&lt;code&gt;..&lt;/code&gt; in XPath) and &lt;code&gt;getID&lt;/code&gt; (&lt;code&gt;@id&lt;/code&gt; in XPath). Implementing these three functions in Javascript is very easy:&lt;/p&gt;
&lt;pre&gt;function getChildren (ctx) toArray(ctx.childNodes);
function getParent   (ctx) [ctx.parentNode];
function getID       (ctx) [ctx.getAttribute("id")];
&lt;/pre&gt;
&lt;p&gt;For the ease of reading most functions in this post are written down as Javascript 1.8 lambdas. This syntax, which leaves away the braces and the return statement of functions, is currently only supported by Firefox&amp;rsquo;s SpiderMonkey engine. Note that all three functions have exactly the same type, they take one context node as input and return an array of nodes (node set) as output. There is a good reason to have a consistent type signature for these primitive transformations: this allows us to easily compose multiple primitive operations into more advanced transformations.&lt;/p&gt;
&lt;h2&gt;Sequential composition&lt;/h2&gt;
&lt;p&gt;To illustrate composition, lets see how we would like to write down a transformation that selects all the grandchildren of a context node. Ideally we would like to sequentially compose two invocations of getChildren into one. With sequential composition we mean applying some transformation to the result of some earlier transformation. Something like this:&lt;/p&gt;
&lt;pre&gt;var getGrandchildren = seq(getChildren, getChildren);
&lt;/pre&gt;
&lt;p&gt;But what would the &lt;code&gt;seq&lt;/code&gt; function look like? To make it more easy to come up with the correct implementation of the &lt;code&gt;seq&lt;/code&gt; function it might help to write down the types of all the functions involved and work from there. We first write down the Haskell style type signatures for our &lt;code&gt;getChildren&lt;/code&gt; function, which takes one context node and produces a list of child nodes:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt; getChildren :: Node -&amp;gt; [Node] &lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Because we want the result of the expression &lt;code&gt;seq(getChildren, getChildren)&lt;/code&gt; itself to be transformation with the same type again, the type of seq must be:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt; seq :: (Node -&amp;gt; [Node]) -&amp;gt; (Node -&amp;gt; [Node]) -&amp;gt; (Node -&amp;gt; [Node]) &lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Which means: take two transformations as input and produce one transformation as output. By only looking at the type we can already deduce the following skeleton of the seq function:&lt;/p&gt;
&lt;pre&gt;function seq (tr0, tr1) function (ctx) /* [some nodes] */ ;
&lt;/pre&gt;
&lt;p&gt;Note that this function is higher-order, it takes two functions and returns a new function. Now we only have to fill in the gap, which we can do by looking at the desired semantics. In the case of the &lt;code&gt;getGrandChildren&lt;/code&gt; function we want to apply the first &lt;code&gt;getChildren&lt;/code&gt; transformation to the context node and then apply the second transformation (&lt;code&gt;getChildren&lt;/code&gt; again) over all the results of the first transformation and group together the results. Translating this to code would become something like: apply &lt;code&gt;tr0&lt;/code&gt; to &lt;code&gt;ctx&lt;/code&gt;, than &lt;code&gt;map&lt;/code&gt; &lt;code&gt;tr1&lt;/code&gt; over all these results and &lt;code&gt;concat&lt;/code&gt;the output.&lt;/p&gt;
&lt;pre&gt;function seq(tr0, tr1) function (ctx) concat(tr0(ctx).map(tr1));
function concat (xs) [].concat.apply([], xs); // stand-alone concat
&lt;/pre&gt;
&lt;p&gt;Now we worked out the skeleton of the function by looking at the derived type and we have come up with an implementation by looking at the desired semantics. The function is now both type correct and works as expected!&lt;/p&gt;
&lt;blockquote&gt;&lt;em&gt;&lt;strong&gt;For the Haskell programmer:&lt;/strong&gt;&lt;/em&gt; we just worked out Kleisli composition for the list monad. This composition internally uses the monadic bind for the list monad instance, which happens to be the &lt;code&gt;concatMap&lt;/code&gt; function. The &lt;code&gt;concatMap&lt;/code&gt; function has type &lt;code&gt;[a] -&amp;gt; (a -&amp;gt; [b]) -&amp;gt; [b]&lt;/code&gt;, which (in our context) shows exactly how we apply a transformation (&lt;code&gt;a -&amp;gt; [b]&lt;/code&gt;) to the results of another transformation (&lt;code&gt;[a]&lt;/code&gt;) and come up with the result of the composition (&lt;code&gt;[b]&lt;/code&gt;).&lt;/blockquote&gt;
&lt;p&gt;To illustrate the generic behavior of our sequential composition we use it to define some other useful transformations.&lt;/p&gt;
&lt;pre&gt;getSiblings    = seq(getParent, getChildren);
getGrandParent = seq(getParent, getParent);
&lt;/pre&gt;
&lt;h2&gt;Alternative composition&lt;/h2&gt;
&lt;p&gt;Now we have defined sequential composition we can also try to define another form of composition which just sums up the results of two transformations working over the same context node. For example when we want to create a transformation that selects &lt;em&gt;both&lt;/em&gt; the grand parent and the grand children of a context node, we might want to write down something like this:&lt;/p&gt;
&lt;pre&gt;getGrands = alt(getGrandParent, getGrandChildren);
&lt;/pre&gt;
&lt;p&gt;We call this composition function &lt;code&gt;alt&lt;/code&gt; because it combines two alternative transformation paths into one. The type signature of the &lt;code&gt;alt&lt;/code&gt; function is similar to that of &lt;code&gt;seq&lt;/code&gt;, it takes two transformations as input and is itself again a transformation. The semantics of this transformation combinator is really easy, it just applies the two input transformation to the same context node and groups the results.&lt;/p&gt;
&lt;p&gt;function alt(tr0, tr1) function (ctx) concat([tr0(ctx), tr1(ctx)]);&lt;/p&gt;
&lt;blockquote&gt;&lt;em&gt;&lt;strong&gt;For the Haskell programmer:&lt;/strong&gt;&lt;/em&gt; we just worked out the &lt;code&gt;&amp;lt;+&amp;gt;&lt;/code&gt; function of the &lt;code&gt;ArrowPlus&lt;/code&gt; instance for the list arrow. This function is defined in terms of the &lt;code&gt;append&lt;/code&gt; function for lists, with type &lt;code&gt;[a] -&amp;gt; [a] -&amp;gt; [a]&lt;/code&gt;. The semantics are very similar to that of the &lt;code&gt;Alternative&lt;/code&gt; and &lt;code&gt;MonadPlus&lt;/code&gt; type classes. Where the &lt;code&gt;seq&lt;/code&gt; function is the algebraic &lt;em&gt;and, product, sequence, &lt;code&gt;/&lt;/code&gt;, times, etc&lt;/em&gt;, the &lt;code&gt;alt&lt;/code&gt; function is the algebraic &lt;em&gt;or, sum, alternative, &lt;code&gt;|&lt;/code&gt;, plus, etc&lt;/em&gt;.&lt;/blockquote&gt;
&lt;h2&gt;Deep recursion, filtering and creating.&lt;/h2&gt;
&lt;p&gt;Now we have two basic transformation combinators: sequential and alternative composition. Combining these two functions can get us some powerful transformation combinators. E.g. we can make the deep function that applies a transformation at arbitrary depth in document.&lt;/p&gt;
&lt;pre&gt;function deep (tr) alt(tr, seq(getChildren, lazy(deep, tr)));
function lazy (f, a) function (n) f(a)(n); // postponed function application
&lt;/pre&gt;
&lt;p&gt;The deep function applies a transformation and groups the results with a recursive deep invocation for all the child nodes of the current context. Because, unfortunately, JavaScript is a strict language we have to explicitly delay the recursion in order to prevent infinite loops. Before we will test the deep transformation combinator we define three other useful primitive transformations.&lt;/p&gt;
&lt;pre&gt;function self (ctx) [ctx];
&lt;/pre&gt;
&lt;p&gt;This transformation &lt;code&gt;self&lt;/code&gt; is a bit special, it is the identity transformation which outputs a singleton list containing the context node itself. This transformation can be compared to the dot (.) in XPath.&lt;/p&gt;
&lt;pre&gt;function isElem (name) function (ctx) ctx.nodeName == name ? [ctx] : [];
&lt;/pre&gt;
&lt;p&gt;This function &lt;code&gt;isElem&lt;/code&gt; is a filter transformation function which includes or excludes a context node based on the &lt;code&gt;nodeName&lt;/code&gt;. When the node name matches it is included as a singleton list, when there is no match the empty lists will be returned.&lt;/p&gt;
&lt;pre&gt;function mkElem (name) function (tr)
 function (ctx) [createElemWithChildren(name, tr(ctx))];
&lt;/pre&gt;
&lt;p&gt;The third function &lt;code&gt;mkElem&lt;/code&gt; is neither a selection or filtering functions, it is a creation function which takes the output node set of a transformation and surrounds it with a new element. This element will be returned as a singleton node set again.&lt;/p&gt;
&lt;p&gt;Using the set of primitives and combinators to create a transformation that produces a simple table of contents from an input document is now very easy. The following transformation shows how to select all &lt;code&gt;H1&lt;/code&gt; and &lt;code&gt;H2&lt;/code&gt; elements from the entire document. All headers will be surrounded by a &lt;code&gt;LI&lt;/code&gt; and inserted into an unordered list &lt;code&gt;UL&lt;/code&gt;.&lt;/p&gt;
&lt;pre&gt;var isHeader = sum(isElem("h1"), isElem("h2"));
var toc = mkElem("ul")(seq(deep(isHeader, mkElem("li")(self))));

var myToc = toc(document.body);
&lt;/pre&gt;
&lt;p&gt;Now &lt;code&gt;toc&lt;/code&gt; is a true JavaScript function that computes a table of contents from an input document.&lt;/p&gt;
&lt;h2&gt;Comparison to XPath&lt;/h2&gt;
&lt;p&gt;Here is a (far from complete) table showing a comparison between XPath queries and our transformation functions. With the very few combinators defined, we can already rebuild a powerful set of XPath queries. Some primitives shown below, like &lt;code&gt;hasAttr&lt;/code&gt; and &lt;code&gt;precedingSiblings&lt;/code&gt;, have not been defined in this post. Defining them is very easy, please try it for yourself and see how far you can get with other exotic XPath axes.&lt;/p&gt;
&lt;p&gt;XPath JavaScript &lt;code&gt;.&lt;/code&gt; &lt;code&gt;self&lt;/code&gt; &lt;code&gt;//div&lt;/code&gt; &lt;code&gt;deep(isElem("div"))&lt;/code&gt; &lt;code&gt;/*/p&lt;/code&gt; &lt;code&gt;seq(getChildren, isElem("p"))&lt;/code&gt; &lt;code&gt;(//div|//p)&lt;/code&gt; &lt;code&gt;sum(deep(isElem("div")), deep(isElem("p")))&lt;/code&gt; &lt;code&gt;//p/../..&lt;/code&gt; &lt;code&gt;deep(seq(seq(isElem("p"), getParent), getParent))&lt;/code&gt; &lt;code&gt;/p[@href]/em&lt;/code&gt; &lt;code&gt;seq(seq(isElem("p"), hasAttr("href")) , seq(getChildren, isElem("em")))&lt;/code&gt; &lt;code&gt;//table/preceding-sibling::h2&lt;/code&gt; &lt;code&gt;deep(seq(isElem("table"), seq(precedingSiblings, isElem("h2"))))&lt;/code&gt;&lt;/p&gt;
&lt;h2&gt;Conclusion and goal&lt;/h2&gt;
&lt;p&gt;So we have seen how easy it is to create a powerful XML query and transformation language with a bare minimum of code. The trick is to define some primitive transformation functions and only two powerful ways to compose transformations. By using functions from one context node to a list of outputs selecting, filtering and creating elements all becomes possible. Using the composition functions you can easily build more advanced and high-level transformations like the &lt;code&gt;deep&lt;/code&gt; function that traverses an entire document tree. The comparison between XPath and the selection primitives is quite clear. Adding an element creation function shows that the query language can quite easily become a true transformation language in which we can add new structure to the output.&lt;/p&gt;
&lt;p&gt;While building such a library yourself is fun, it might feel a bit useless at first sight. Why reinvent XPath or XSLT while most browsers have built-in support for these tools? There are two main reasons to perform transformations this way.&lt;/p&gt;
&lt;p&gt;The first reason is that is now very easy to add extra power to the library by plugging in new JavaScript functions. Using XSLT as a programming language, which unfortunately happens a lot in practice, almost always ends up shooting yourself in the foot with large and unmanageable recursive templates that fuzz what is really going on.&lt;/p&gt;
&lt;p&gt;The second reason is more subtle. Here at typLAB we have shown that it is possible to change the primitive transformations and the two composition functions with their &lt;em&gt;functional reactive&lt;/em&gt; counterparts. This enables us to incrementally rebuild the output of a transformation when only small parts of the input document change. This incremental reactivity is an extremely powerful paradigm that allows for very fast live queries over semantic and structured documents.&lt;/p&gt;
&lt;p&gt;As you can see, this framework is very simple. Try playing with it yourself!&lt;/p&gt;</description><link>http://engineering.silk.co/post/33295990901</link><guid>http://engineering.silk.co/post/33295990901</guid><pubDate>Thu, 10 Dec 2009 00:00:00 -0500</pubDate><category>engineering</category><category>javascript</category></item><item><title>Writing a generic XML pickler</title><description>&lt;p&gt;In a &lt;a href="http://engineering.silkapp.com/post/31921413529/haskell-data-types-and-xml"&gt;previous post&lt;/a&gt;, Sebas explained that we use so-called XML picklers to convert Haskell data types to XML. Since these picklers have a regular structure, we don&amp;rsquo;t write them by hand, but derive them automatically using generic programming techniques. In this post, I&amp;rsquo;ll explain how our generic XML pickler works. The code shown here has been made &lt;a href="http://hackage.haskell.org/package/regular-xmlpickler"&gt;available on hackage&lt;/a&gt;.  We use the &lt;a href="http://hackage.haskell.org/package/regular"&gt;regular library&lt;/a&gt; to represent data types generically. It allows you to build a structure for your type that is isomorphic to your type, but built up from standard building blocks, called functors. This is possible because Haskell data types have a standard structure: a choice of one of several constructors (this is often called a sum), each having a number of fields (this is often called a product).&lt;/p&gt;



&lt;p&gt;As an example, we will represent a simple user data type:&lt;/p&gt;



&lt;pre&gt;data User = User&#13;
         { name  :: String&#13;
         , email :: String&#13;
         , admin :: Bool&#13;
         }&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;The generic representation of this type is called a pattern functor. For the user data type, it will look like this:&lt;/p&gt;



&lt;pre&gt;type instance PF User = C User_User_&#13;
 (   S User_User_name_  (K String)&#13;
 :*: S User_User_email_ (K String)&#13;
 :*: S User_User_admin  (K Bool)&#13;
 )&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;Here we see a few of the functors that are the building blocks of our generic representation: The &lt;code&gt;C&lt;/code&gt; type marks constructors, the &lt;code&gt;S&lt;/code&gt; type marks record labels, &lt;code&gt;K&lt;/code&gt; marks the types that make up the fields of our record, and &lt;code&gt;(:*:)&lt;/code&gt; is the product mentioned earlier, and is very similar to the &lt;code&gt;(,)&lt;/code&gt; used for constructing tuples. There are three more functors in regular: &lt;code&gt;I&lt;/code&gt;, which marks recursive positions in a type; &lt;code&gt;(:+:)&lt;/code&gt;, which sums together different constructors; and &lt;code&gt;U&lt;/code&gt;, for constructors without fields.&lt;/p&gt;



&lt;p&gt;To write a generic function, we define a type class which contains this generic function, and give an instance for each of these functors. If we then also define conversion functions from our data type to the generic representation and back, we can apply the function to our data type. Note that for regular, these conversion functions, and the representation given above, can all be generated using Template Haskell.&lt;/p&gt;



&lt;p&gt;Since we want to write a generic XML pickler for use with the &lt;a href="http://hackage.haskell.org/package/hxt"&gt;HXT library&lt;/a&gt;, we&amp;rsquo;ll define a type class containing such a pickler:&lt;/p&gt;



&lt;pre&gt;class GXmlPickler f where&#13;
 gxpicklef :: PU a -&amp;gt; PU (f a)&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;This says that we can give a generic pickler for one of the functors &lt;code&gt;f&lt;/code&gt;, containing &lt;code&gt;a&lt;/code&gt;&amp;rsquo;s at the recursive positions, if we have a pickler for &lt;code&gt;a&lt;/code&gt;&amp;rsquo;s. I&amp;rsquo;ll now show the instances for all the functors, and explain them one by one.&lt;/p&gt;



&lt;p&gt;The first is the instance for &lt;code&gt;I&lt;/code&gt;, marking recursive positions. Since we get a pickler for the resursive positions, all we have to do is wrap it in the &lt;code&gt;I&lt;/code&gt; constructor when unpickling, and remove that constructor when pickling. HXT provides the &lt;code&gt;xpWrap&lt;/code&gt; function to transform a pickler like this.&lt;/p&gt;



&lt;pre&gt;instance GXmlPickler I where&#13;
 gxpicklef = xpWrap (I, unI)&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;Next, we&amp;rsquo;ll tackle &lt;code&gt;K&lt;/code&gt;. Here, we require that the type of the field has its own pickler, and use that. We again use &lt;code&gt;xpWrap&lt;/code&gt; to add and remove the constructor of the functor. We provide a special instance for &lt;code&gt;String&lt;/code&gt;, since it doesn&amp;rsquo;t have a standard pickler. We choose to use the pickler that allows empty strings.&lt;/p&gt;



&lt;pre&gt;instance XmlPickler a =&amp;gt; GXmlPickler (K a) where&#13;
 gxpicklef _ = (K, unK) `xpWrap` xpickle&#13;
&#13;
instance GXmlPickler (K String) where&#13;
 gxpicklef _ = (K, unK) `xpWrap` xpText0&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;U&lt;/code&gt; works similarly. We use the pickler for &lt;code&gt;()&lt;/code&gt;, and use xpWrap to go from a &lt;code&gt;()&lt;/code&gt; to a &lt;code&gt;U&lt;/code&gt; and back.&lt;/p&gt;



&lt;pre&gt;instance GXmlPickler U where&#13;
 gxpicklef _ = (const U, const ()) `xpWrap` xpUnit&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;The case for &lt;code&gt;(:+:)&lt;/code&gt; is interesting. Remember that &lt;code&gt;(:+:)&lt;/code&gt; represents a choice between two functors. If we have picklers for both of these, we can define a pickler for the sum as follows: during conversion to XML, we can pattern match on &lt;code&gt;L&lt;/code&gt; or &lt;code&gt;R&lt;/code&gt; (the constructors of &lt;code&gt;(:+:)&lt;/code&gt;) and choose the left or right pickler appropriately. During conversion from XML, we try the first pickler. If it fails (an unpickler returns a &lt;code&gt;Maybe&lt;/code&gt; value) we try the second. Since HXT doesn&amp;rsquo;t seem to have a combinator that follows this logic, let&amp;rsquo;s define one ourselves:&lt;/p&gt;



&lt;pre&gt;xpEither :: PU (f r) -&amp;gt; PU (g r) -&amp;gt; PU ((f :+: g) r)&#13;
xpEither (PU fl tl sa) (PU fr tr sb) = PU&#13;
 (\(x, st) -&amp;gt; case x of&#13;
                L y -&amp;gt; fl (y, st)&#13;
                R y -&amp;gt; fr (y, st))&#13;
 (\x -&amp;gt; case tl x of&#13;
          (Nothing, _) -&amp;gt; lmap (fmap R) (tr x)&#13;
          r            -&amp;gt; lmap (fmap L) r)&#13;
 (sa `scAlt` sb)&#13;
 where lmap f (a, b) = (f a, b)&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;Note that we take apart the &lt;code&gt;PU&lt;/code&gt; type, and create a pretty printer, a parser and a schema separately. We can use this function in the &lt;code&gt;GXmlPickler&lt;/code&gt; instance by supplying it with two picklers, created by recursive calls to &lt;code&gt;gxpicklef&lt;/code&gt;.&lt;/p&gt;



&lt;pre&gt;instance (GXmlPickler f, GXmlPickler g) =&amp;gt; GXmlPickler (f :+: g) where&#13;
 gxpicklef f = xpEither (gxpicklef f) (gxpicklef f)&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;The instance for &lt;code&gt;(:&lt;em&gt;:)&lt;/em&gt;&lt;/code&gt; is easy again. Since &lt;code&gt;(::)&lt;/code&gt; combines two functors, we can require that these two have a pickler. We use &lt;code&gt;xpPair&lt;/code&gt; to combine these into a pickler for a pair, and then use &lt;code&gt;xpWrap&lt;/code&gt; again to convert it into a pickler for &lt;code&gt;(:*:)&lt;/code&gt;.&lt;/p&gt;



&lt;pre&gt;instance (GXmlPickler f, GXmlPickler g) =&amp;gt; GXmlPickler (f :*: g) where&#13;
 gxpicklef f = (uncurry (:*:), \(a :*: b) -&amp;gt; (a, b))&#13;
               `xpWrap`&#13;
               (gxpicklef f `xpPair` gxpicklef f)&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;Note that so far, we haven&amp;rsquo;t created any XML tags ourselves. We&amp;rsquo;ve only combined picklers. Now we come to the instances for constructors and record selectors. Here, we&amp;rsquo;ll use the constructor or selector name (converted to lowercase) to generate and parse xml tags. This is done with the &lt;code&gt;xpElem&lt;/code&gt; combinator.&lt;/p&gt;



&lt;pre&gt;instance (Constructor c, GXmlPickler f) =&amp;gt; GXmlPickler (C c f) where&#13;
 gxpicklef f = xpElem (map toLower $ conName (undefined :: C c f r))&#13;
                 ((C, unC) `xpWrap` gxpicklef f)&#13;
&#13;
instance (Selector s, GXmlPickler f) =&amp;gt; GXmlPickler (S s f) where&#13;
 gxpicklef f = xpElem (map toLower $ selName (undefined :: S s f r))&#13;
                 ((S, unS) `xpWrap` gxpicklef f)&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;We now have picklers for all generic representations built from the standard functors in regular. We still need a top level function that works on our real data types, though. Regular provides the two functions &lt;code&gt;to&lt;/code&gt; and &lt;code&gt;from&lt;/code&gt; to convert between data types and their generic representation. We can use these together with &lt;code&gt;xpWrap&lt;/code&gt; to change our generic pickler into a pickler for real data types. Another matter we have not addressed is the first argument to &lt;code&gt;gpicklef&lt;/code&gt;, which is a pickler for the recursive positions. Here we pass the top level function, since the recursive position contains our original data type again.&lt;/p&gt;



&lt;pre&gt;gxpickle :: (Regular a, GXmlPickler (PF a)) =&amp;gt; PU a&#13;
gxpickle = (to, from) `xpWrap` gxpicklef gxpickle&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;And that&amp;rsquo;s all for the generic XML pickler. Using it is very simple. To use it for our user data type above, we import &lt;code&gt;Generic.Regular&lt;/code&gt;, &lt;code&gt;Generic.Regular.TH&lt;/code&gt;, &lt;code&gt;Generic.Regular.XmlPickler&lt;/code&gt; and &lt;code&gt;Text.XML.HXT.Arrow.Pickle&lt;/code&gt;. We can then derive the &lt;code&gt;Regular&lt;/code&gt; instance for user (note that we need a little bit of extra code, since Template Haskell cannot generate type family instances yet), and make an &lt;code&gt;XmlPickler&lt;/code&gt; instance:&lt;/p&gt;



&lt;pre&gt;$(deriveAll ''User "PFUser")&#13;
type instance PF User = PFUser&#13;
&#13;
instance XmlPickler User where&#13;
 xpickle = gxpickle&#13;
&lt;/pre&gt;

&lt;p&gt;&#13;&lt;/p&gt;

&lt;p&gt;And that&amp;rsquo;s it! The package is available as &lt;a href="http://hackage.haskell.org/package/regular-xmlpickler"&gt;regular-xmlpickler&lt;/a&gt; on hackage, and the source can also be found on &lt;a href="http://github.com/typLAB/regular-xmlpickler"&gt;github&lt;/a&gt;. The packages &lt;a href="http://hackage.haskell.org/package/regular"&gt;regular&lt;/a&gt; and &lt;a href="http://hackage.haskell.org/package/hxt"&gt;HXT&lt;/a&gt; can also be found on hackage.&lt;/p&gt;</description><link>http://engineering.silk.co/post/31922245822</link><guid>http://engineering.silk.co/post/31922245822</guid><pubDate>Tue, 10 Nov 2009 00:00:00 -0500</pubDate><category>engineering</category><category>xml</category><category>haskell</category></item><item><title>How I learned to stop worrying and love web development again</title><description>&lt;h3&gt;Why we don&amp;rsquo;t support Internet Explorer&lt;/h3&gt;



&lt;p&gt;We&amp;rsquo;ve already talked about some of the technology choices we&amp;rsquo;re making as a company. And while our &lt;a href="http://engineering.silkapp.com/post/31921413529/haskell-data-types-and-xml" title="Haskell data types and XML"&gt;choices&lt;/a&gt; on the back-end can hardly be labeled as &lt;a href="http://engineering.silkapp.com/post/31920990633/why-we-use-haskell" title="Why we use Haskell"&gt;mainstream&lt;/a&gt;, the most difficult choice we actually had to make was related to the client-side as it directly affects our users. Obviously, Javascript on the client is a given, and we love it. However, as most web developers know, the differences between browsers are enormous and developing for all of them is almost impossible (Try using Netscape 4.7 today, if you&amp;rsquo;re curious.). Still, current conventional wisdom dictates that you should support recents versions of Internet Explorer (Usually, this means Internet Explorer 6 and newer. However support for Internet Explorer 6 is slowly declining, as it&amp;rsquo;s dropped by major sites like &lt;a href="http://www.google.com/support/youtube/bin/answer.py?hl=en&amp;amp;answer=157259"&gt;YouTube&lt;/a&gt;, &lt;a href="http://arstechnica.com/microsoft/news/2009/08/googles-orkut-starts-phasing-out-support-for-ie6.ars"&gt;Orkut&lt;/a&gt; and products like &lt;a href="http://productblog.37signals.com/products/2008/07/basecamp-phasin.html"&gt;Basecamp&lt;/a&gt;.), Firefox and the WebKit based browsers (basically, Safari and Google Chrome). We, however, have decided to drop Internet Explorer support entirely (Note that what we&amp;rsquo;re discussing here is browser support for our &lt;em&gt;application&lt;/em&gt;. The content that resides inside the application will always be available to any web browser, whether it&amp;rsquo;s a text-based browser with no Javascript support, a low capability mobile browser or Internet Explorer 6. This is based on the principe of &lt;a href="http://dev.opera.com/articles/view/graceful-degradation-progressive-enhance/#gracefulprogressivenutshell"&gt;graceful degradation&lt;/a&gt;.).&lt;/p&gt;



&lt;p&gt;In general, the trade-off you face when choosing which platform to develop for is between development time and a larger potential customer base that&amp;rsquo;s associated with the platform. Looking at the web right now, Internet Explorer still leads the market by a large margin (Wikipedia has a &lt;a href="http://en.wikipedia.org/wiki/Usage_share_of_web_browsers#Summary_Table"&gt;nice summary of various sources&lt;/a&gt; with browser usage statisics.). So even if it seems annoying that you have to work around some CSS bugs or still write Internet Explorer specific event handling code for a web site, the payoff in user reach will usually still be worth it. Modern Javascript libraries such as &lt;a href="http://jquery.com/"&gt;jQuery&lt;/a&gt; or &lt;a href="http://mootools.net/"&gt;Mootools&lt;/a&gt; lower cross-browser development time even more by abstracting away lots of differences between browsers, tipping the equation even more in favor of Internet Explorer support.&lt;/p&gt;



&lt;p&gt;So why did we choose to ignore the most used browser on the planet? It&amp;rsquo;s because we decided that in our case, the development costs would simply not be worth it. Obviously, this assessment is very specific for the type of application we&amp;rsquo;re building. We didn&amp;rsquo;t just base this on a hunch, we actually have quite some experience in this field: most of us have worked on products like &lt;a href="http://xopus.com"&gt;Xopus&lt;/a&gt; before. Xopus is an awesome, browser based wysiwyg XML editor. It consist of more than &lt;a href="http://www.slideshare.net/ajaxexperience2009/laurens-van-den-oever-xopus-presentation"&gt;120,000 lines of client-side Javascript code&lt;/a&gt;. A non-trivial part of that is code that completely works around standard Internet Explorer behavior because of its bugginess or complete lack of support. This isn&amp;rsquo;t about your father&amp;rsquo;s unsupported CSS selectors or the lack of addEventListener. We&amp;rsquo;re talking about stuff like &lt;a href="http://xopus.com/devblog/2007/craziest-dhtml-hack-ever.html"&gt;having to write your own cursor&lt;/a&gt; becausecontentEditable becomes basically useless when working on complex documents. The amount of bugs that are related tocontentEditable, text ranges, drag and drop and the Document Object Model in general are staggering. Most JavaScript projects, including some popular libraries, don&amp;rsquo;t even deal with these advanced aspects of the browser at all.&lt;/p&gt;



&lt;p&gt;Now obviously, the other browsers aren&amp;rsquo;t all free of bugs. Which brings us to the second problem with Internet Explorer: lack of real progress and transparency. Even if you consider relatively easy and popular features (such as support foraddEventListener), it&amp;rsquo;s hard to understand why they haven&amp;rsquo;t been implemented yet and if there is any timeline at all to implement them. That makes the probability of low visibility improvements and bug fixes in the rendering or selection code practically zero. The contrast with the open development of Mozilla and WebKit is huge, where almost everything is publicly discussed and with a focus on constant improvement of the rendering engines and pretty clear timeline.&lt;/p&gt;



&lt;p&gt;The current state of the web is actually very exciting right now, if we ignore Internet Explorer for a moment. Thanks to HTML5 there is a lot of progress allowing us to make almost desktop class applications, with support for things like &lt;a href="http://www.alertdebugging.com/2009/08/16/on-html-5-drag-and-drop/"&gt;drag and drop from the desktop&lt;/a&gt;, &lt;a href="https://developer.mozilla.org/En/Using_web_workers"&gt;background processes&lt;/a&gt; and &lt;a href="http://developer.apple.com/safari/library/documentation/iPhone/Conceptual/SafariJSDatabaseGuide/OfflineApplicationCache/OfflineApplicationCache.html"&gt;offline support&lt;/a&gt; (To be fair, Internet Explorer 8 does provide some offline support.). All of this should greatly improve the user experience with web applications and bridge the gap with desktop applications. No amount of code or smart engineering will allow us to bring that level of experience to browsers that lack these features.&lt;/p&gt;



&lt;p&gt;Does this mean that no complex web application should support Internet Explorer? Obviously, it depends on many factors. There are large differences between applications that are targeted to tech-savvy users (where Internet Explorer is quickly becoming a minority browser) or to large organizations (where Internet Explorer 6 is still widely used). We&amp;rsquo;re a small team and we have to prioritize our development goals aggressively. Large teams with lots of resources are in a different situation altogether. That being said, we were pleased to see that the Google Wave team also chose to &lt;a href="http://googlewavedev.blogspot.com/2009/09/google-wave-in-internet-explorer.html"&gt;drop Internet Explorer support&lt;/a&gt; and having Google develope the &lt;a href="http://code.google.com/chrome/chromeframe/"&gt;Chrome Frame plug-in&lt;/a&gt;.&lt;/p&gt;



&lt;p&gt;Finally, although this might seem less important than the above considerations from a business perspective, there is the loss of friction and return of enjoyment in developing web applications again. Not developing for Internet Explorer means that we can do amazing things with &lt;a href="http://webkit.org/blog/138/css-animation/"&gt;CSS&lt;/a&gt;, use &lt;a href="https://developer.mozilla.org/en/Core_JavaScript_1.5_Guide/Creating_New_Objects/Defining_Getters_and_Setters"&gt;new Javascript features&lt;/a&gt; in our codebase, and in general rediscover the excitement of the possibilities of the web. And that makes us love web development again.&lt;/p&gt;</description><link>http://engineering.silk.co/post/32258824729</link><guid>http://engineering.silk.co/post/32258824729</guid><pubDate>Wed, 28 Oct 2009 00:00:00 -0400</pubDate><category>engineering</category></item><item><title>Mutation events: what happens?</title><description>&lt;p&gt;Since typ&lt;span class="lab"&gt;LAB&lt;/span&gt; is all about exploring new ways of creating and consuming online content we figured our software might want to keep track of what&amp;rsquo;s happening inside a document. All modern browsers have support for &lt;a href="http://www.w3.org/TR/DOM-Level-2-Events/events.html#Events-eventgroupings-mutationevents"&gt;W3C&amp;rsquo;s mutation events&lt;/a&gt;. Safari, Chrome, FireFox and Opera all do them. But not all do all of them. Notably &lt;a href="https://bugs.webkit.org/show_bug.cgi?id=8191"&gt;WebKit fails to fire DOMAttrModified&lt;/a&gt; events when an attribute is changed. It does however fire the &lt;code&gt;DOMSubtreeModified&lt;/code&gt; event after an attribute is modified. So at least that gives us something to work with until the good folks at WebKit squash the bug.&lt;/p&gt;
&lt;p&gt;Here is how we fixed the lack of &lt;code&gt;DOMAttrModified&lt;/code&gt;. First we need to detect whether the fix is needed:&lt;/p&gt;
&lt;pre&gt;var attrModifiedWorks = false;
var listener = function(){ attrModifiedWorks = true; };
document.documentElement.addEventListener("DOMAttrModified", listener, false);
document.documentElement.setAttribute("___TEST___", true);
document.documentElement.removeAttribute("___TEST___", true);
document.documentElement.removeEventListener("DOMAttrModified", listener, false);
&lt;/pre&gt;
&lt;p&gt;The code is straightforward. Add an attribute and have a listener register the firing of the subsequent &lt;code&gt;DOMAttrModified&lt;/code&gt; event. If the event is not fired our repair code kicks in:&lt;/p&gt;
&lt;pre&gt;if (!attrModifiedWorks) 
{&lt;/pre&gt;
&lt;p&gt;Next we store and override &lt;code&gt;HTMLElement.setAttribute&lt;/code&gt;:&lt;/p&gt;
&lt;pre&gt;HTMLElement.prototype.__setAttribute = HTMLElement.prototype.setAttribute

HTMLElement.prototype.setAttribute = function(attrName, newVal)
{
  var prevVal = this.getAttribute(attrName);
  this.__setAttribute(attrName, newVal);
  newVal = this.getAttribute(attrName);
  if (newVal != prevVal)
  {
    var evt = document.createEvent("MutationEvent");
    evt.initMutationEvent(
      "DOMAttrModified",
      true,
      false,
      this,
      prevVal || "",
      newVal || "",
      attrName,
      (prevVal == null) ? evt.ADDITION : evt.MODIFICATION
    );
    this.dispatchEvent(evt);
  }
}
&lt;/pre&gt;
&lt;p&gt;The new code fetches the current value of the attribute, soon to become the previous value. It then proceeds to set the attribute using the original &lt;code&gt;setAttribute&lt;/code&gt; method that we stored. We don&amp;rsquo;t know whether that method does fancy stuff to the new attribute value, so, just to be sure, we fetch the new value by calling &lt;code&gt;getAttribute&lt;/code&gt; once again. If and only if the new and previous value differ we proceed to dispatch the appropriately initialised mutation event. This covers added and modified attributes. But it won&amp;rsquo;t help us with removed attributes. For those we can override the &lt;code&gt;removeAttribute&lt;/code&gt; method of &lt;code&gt;HTMLElement&lt;/code&gt;:&lt;/p&gt;
&lt;pre&gt;HTMLElement.prototype.__removeAttribute = HTMLElement.prototype.removeAttribute;
HTMLElement.prototype.removeAttribute = function(attrName)
{
  var prevVal = this.getAttribute(attrName);
  this.__removeAttribute(attrName);
  var evt = document.createEvent("MutationEvent");
  evt.initMutationEvent(
    "DOMAttrModified",
    true,
    false,
    this,
    prevVal,
    "",
    attrName,
    evt.REMOVAL
  );
  this.dispatchEvent(evt);
}&lt;/pre&gt;
&lt;p&gt;This concludes our fix for the lack of &lt;code&gt;DOMAttrModified&lt;/code&gt; in WebKit. Is this fix perfect? Nope. Some known issues:&lt;/p&gt;
&lt;ul&gt;&lt;li&gt;a &lt;code&gt;DOMSubtreeModified&lt;/code&gt; event is fired before instead of after the (artificial) &lt;code&gt;DOMAttrModified&lt;/code&gt; event&lt;/li&gt;
&lt;li&gt;assigning a value to an attribute will not trigger our &lt;code&gt;setAttribute&lt;/code&gt; method. Most noticeably assigning a value to a &lt;code&gt;className&lt;/code&gt; or &lt;code&gt;id&lt;/code&gt; attribute will not result in the appropriate &lt;code&gt;DOMAttrModified&lt;/code&gt; event&lt;/li&gt;
&lt;/ul&gt;&lt;p&gt;We&amp;rsquo;re open to suggestions. But best would be if some WebKit developer would fix &lt;a href="https://bugs.webkit.org/show_bug.cgi?id=8191"&gt;the bug&lt;/a&gt; so we can throw this code away.&lt;/p&gt;</description><link>http://engineering.silk.co/post/31921750832</link><guid>http://engineering.silk.co/post/31921750832</guid><pubDate>Thu, 08 Oct 2009 00:00:00 -0400</pubDate><category>engineering</category><category>mutation</category><category>webkit</category></item><item><title>Haskell data types and XML</title><description>&lt;p&gt;Here at typ&lt;span class="lab"&gt;LAB&lt;/span&gt; it wasn&amp;rsquo;t evident from the beginning what would be the best choice for a storage back-end. We knew that we were about to build a web based editor and would be dealing with a lot of &lt;a href="http://www.whatwg.org/specs/web-apps/current-work/multipage/"&gt;HTML5&lt;/a&gt; documents with lots of meta data. After some careful consideration we decided to go for an XML database. More specifically, the &lt;a href="http://www.oracle.com/database/berkeley-db/xml/index.html"&gt;Berkeley XML Database&lt;/a&gt;, lovingly called DBXML by its authors. We figured that using DBXML would give us some important advantages:&lt;/p&gt;
&lt;ul&gt;&lt;li&gt;Collections of HTML5 documents will form the basis of data model. Only one trivial conversion from HTML5 to syntactically valid XML is needed to get our documents into the XML database. Once stored we can perform some interesting queries over our data.&lt;/li&gt;
&lt;li&gt;XML databases allow for the storage of complex data layouts without having a strict schema. Without a schema it will be easier to adjust our data model over time without instantly breaking our software.&lt;/li&gt;
&lt;li&gt;&lt;a href="http://www.w3.org/TR/xquery/"&gt;XQuery&lt;/a&gt; is a very expressive (almost-purely functional) querying language which is at least as powerful as SQL and far more flexible in the structure of the data to target.&lt;/li&gt;
&lt;li&gt;XML can be used to both encode strictly defined datatypes and store free-form documents in the same document collection. This will enable us to put both our meta data and our documents in the same database.&lt;/li&gt;
&lt;li&gt;A quick look on Hackage revealed there is an out-of-the-box easy-to-use &lt;a href="http://hackage.haskell.org/package/BerkeleyDBXML"&gt;Haskell binding&lt;/a&gt; available for the Berkeley XML database. No need to create custom bindings ourselves.&lt;/li&gt;
&lt;li&gt;We are in the advantage (or disadvantage) of having Haskell as our language of choice for our server software. Because of the hierarchical nature of both XML and Haskell algebraic datatypes, an XML database feels like a perfect fit.&lt;/li&gt;
&lt;/ul&gt;&lt;p&gt;Once we decided to go for an DBXML back-end we had to figure out how to easily get values form our Haskell program in and out of the database. The rest of this post will be dealing with the last point of our enumeration: how to get a nice mapping from Haskell&amp;rsquo;s algebraic datatypes to our DBXML back-end.&lt;/p&gt;
&lt;h2&gt;XML queries&lt;/h2&gt;
&lt;p&gt;The DBXML binding for Haskell is a shallow wrapper around the existing C++ API. This library allows us to perform the common create, read, update and delete queries for entire XML documents or parts of it. Communication with the XML database happens mainly via XQuery. Queries and query parameters are passed into the API (and results will come out) using Haskell &lt;a href="http://hackage.haskell.org/packages/archive/bytestring/0.9.1.4/doc/html/Data-ByteString.html#1"&gt;&lt;code&gt;ByteString &lt;/code&gt;&lt;/a&gt;s. It is up to the programmer to setup the queries with the right XML structure and encoding. Take for example the (somewhat simplified) type signature of the &lt;code&gt;query&lt;/code&gt; function:&lt;/p&gt;
&lt;pre&gt;query :: Collection -&amp;gt; Query -&amp;gt; Parameters -&amp;gt; IO [ByteString]&lt;/pre&gt;
&lt;p&gt;This function takes an identification of the XML collection, which is somewhat like a database handle, an XML query, a set of query parameters and returns a possibly empty list of XML snippets as &lt;code&gt;ByteString&lt;/code&gt;s. Too bad that all our domain objects are well-typed Haskell algebraic datatypes and not raw sequences of bytes. We need a simple XML (de)serialization tool for this.&lt;/p&gt;
&lt;h2&gt;XML picklers&lt;/h2&gt;
&lt;p&gt;The &lt;a href="http://hackage.haskell.org/package/hxt"&gt;Haskell XML Toolbox&lt;/a&gt;(HXT) is a library containing a (quite extended) collection of XML processing tools. The library has support for XML parsing, pretty printing, XPath queries, XSL stylesheets, DTD, XSD and RelaxNG schemas and a lot more. Interestingly, HXT exposes a type class and an accompanying set of combinators called XML picklers. XML picklers can be used to build conversion functions from Haskell datatypes to XML and vice versa. The type class looks like this:&lt;/p&gt;
&lt;pre&gt; class XmlPickler a where xpickle :: PU a&lt;/pre&gt;
&lt;p&gt;So for every type in the &lt;a href="http://hackage.haskell.org/packages/archive/hxt/8.3.1/doc/html/Text-XML-HXT-Arrow-Pickle.html#t%3AXmlPickler"&gt;&lt;code&gt;XmlPickler&lt;/code&gt;&lt;/a&gt; type class there is some &lt;a href="http://hackage.haskell.org/packages/archive/hxt/8.3.1/doc/html/Text-XML-HXT-Arrow-Pickle.html#t%3APU"&gt;&lt;code&gt;PU&lt;/code&gt;&lt;/a&gt;available. The &lt;code&gt;PU&lt;/code&gt; datatype is composed of a pair of pickle (serialize) and unpickle (deserialize) functions together with a schema description. Because we won&amp;rsquo;t be using the schema definitions we will ignore them for now. There is probably no need to ever touch the functions inside the &lt;code&gt;PU&lt;/code&gt; type, because the library supplies a vast amount of basic pickler combinators to be used instead. To illustrate the usage of HXT picklers take this simple Haskell datatype representing a single user in our system:&lt;/p&gt;
&lt;pre&gt;data User = User { name :: String , email :: String , password :: String , openID :: String }&lt;/pre&gt;
&lt;p&gt;Using some of the basic pickler combinators from the library it is very easy to come up with a suitable &lt;code&gt;XmlPickler&lt;/code&gt; instance:&lt;/p&gt;
&lt;pre&gt;class XmlPickler User where userPickle = xpElem "user" $ xpWrap ( (\(a, b, c, d, e) -&amp;gt; User a b c d e) , (\(User a b c d e) -&amp;gt; (a, b, c, d, e)) ) $ xp5Tuple (xpElem "username" xpText) (xpElem "name" xpText) (xpElem "password" xpText) (xpElem "email" xpText) (xpElem "openid" xpText0)&lt;/pre&gt;
&lt;p&gt;This instance uses the &lt;a href="http://hackage.haskell.org/packages/archive/hxt/8.3.1/doc/html/Text-XML-HXT-Arrow-Pickle.html#v%3Axp5Tuple"&gt;&lt;code&gt;xp5Tuple&lt;/code&gt;&lt;/a&gt; function to pickle five sub-picklers into a big tuple. The five fields will be appropriately named elements from which the text value will be used. The tuple will be converted into a value of the &lt;code&gt;User&lt;/code&gt; datatype using the &lt;a href="http://hackage.haskell.org/packages/archive/hxt/8.3.1/doc/html/Text-XML-HXT-Arrow-Pickle.html#v%3AxpWrap"&gt;&lt;code&gt;xpWrap&lt;/code&gt;&lt;/a&gt; function. This is all you to need to manually write XML serialization and deserialization code. A bit off topic but interesting to note is the fact that the &lt;code&gt;xpWrap&lt;/code&gt; function can be seen as a pickler specific and bidirectional version of the well known &lt;a href="http://haskell.org/ghc/docs/latest/html/libraries/base/Prelude.html#v:fmap"&gt;&lt;code&gt;fmap&lt;/code&gt;&lt;/a&gt; for &lt;a href="http://haskell.org/ghc/docs/latest/html/libraries/base/Prelude.html#t%3AFunctor"&gt;&lt;code&gt;Functor&lt;/code&gt;&lt;/a&gt;s. The &lt;code&gt;xpWrap&lt;/code&gt; is used to define true isomorphisms. When we generalize the type of &lt;code&gt;xpWrap&lt;/code&gt; to work arbitrary containers, lets call this function &lt;code&gt;bifmap&lt;/code&gt;, and compare the type signatures this similarity becomes obvious:&lt;/p&gt;
&lt;pre&gt;fmap :: (a -&amp;gt; b) -&amp;gt; f a -&amp;gt; f b bifmap :: (a -&amp;gt; b, b -&amp;gt; a) -&amp;gt; f a -&amp;gt; f b&lt;/pre&gt;
&lt;p&gt;So, taking the &lt;code&gt;XmlPickler&lt;/code&gt; instance for our &lt;code&gt;User&lt;/code&gt; datatype we can now easily convert users into XML and read them back in, like the following example:&lt;/p&gt;
&lt;pre&gt;User "jd" "John Doe" "secret" "john@doe" "none"&lt;/pre&gt;
&lt;pre&gt; jd John Doe secret john@doe none &lt;/pre&gt;
&lt;p&gt;Using the &lt;a href="http://hackage.haskell.org/packages/archive/hxt/8.3.1/doc/html/Text-XML-HXT-Arrow-Pickle.html#t%3AXmlPickler"&gt;&lt;code&gt;xpickle&lt;/code&gt;&lt;/a&gt; function from the type class and the &lt;a href="http://hackage.haskell.org/packages/archive/hxt/8.3.1/doc/html/Text-XML-HXT-Arrow-Pickle.html#v%3AxunpickleVal"&gt;&lt;code&gt;xunpickleVal&lt;/code&gt;&lt;/a&gt;from the HXT library we can now write a more suitable query function on top of the raw version. This version does not return &lt;code&gt;ByteString&lt;/code&gt;s, but values of any type we can convert to. Off course, the information in the database should match your datatype, otherwise the unpickler function will just produce parse errors resulting in an empty list.&lt;/p&gt;
&lt;pre&gt;query :: XmlPickler a =&amp;gt; Collection -&amp;gt; Query -&amp;gt; Parameters -&amp;gt; IO [a]&lt;/pre&gt;
&lt;p&gt;Although this pickler example for our user is a very simple one, even more complicated datatypes, including multi-constructor and possibly mutual recursive datatypes, can quite easily be made an instance of the &lt;code&gt;XmlPickler&lt;/code&gt; class. Unfortunately we still have to write them all by hand.&lt;/p&gt;
&lt;h2&gt;Going generic&lt;/h2&gt;
&lt;p&gt;After a few of years at the University of Utrecht we learned at least one valuable lesson, never write functions by hand when they can be &lt;a href="http://www.cs.uu.nl/wiki/GenericProgramming"&gt;derived generically&lt;/a&gt;. We decided to write a generic XML pickler function using the generic programming library &lt;a href="http://hackage.haskell.org/package/regular"&gt;Regular&lt;/a&gt;, developed (not entirely coincidentally) at the University of Utrecht. Regular is a relatively simple but powerful tool for writing data type generic functions. The library has support for deriving embedding projection pairs (conversions from and to a generic representation) using &lt;a href="http://www.haskell.org/th/"&gt;Template Haskell&lt;/a&gt;and provides enough reflection to inspect constructor names and record labels. The generic representation is encoded as a type family (the pattern functor, or &lt;code&gt;PF&lt;/code&gt;) over the original data type. The generic pickler function we developed has the following signature:&lt;/p&gt;
&lt;pre&gt;gxpickle :: (Regular a, GXmlPickler (PF a)) =&amp;gt; PU a&lt;/pre&gt;
&lt;p&gt;This means that for every type that we can convert to a generic representation (indicated by the &lt;code&gt;Regular&lt;/code&gt; type class) and for every type that has a &lt;code&gt;GXmlPickler&lt;/code&gt; instance for its generic representation, we can deliver a &lt;code&gt;PU&lt;/code&gt;. The &lt;code&gt;regular-xmlpickler&lt;/code&gt; package implements the &lt;code&gt;GXmlPickler&lt;/code&gt; type class and the instances for the types of which the representations are composed. So, all we need now for our &lt;code&gt;User&lt;/code&gt; datatype is to derive a generic representation and use the generic implementation for the &lt;code&gt;XmlPickler&lt;/code&gt; instance.&lt;/p&gt;
&lt;pre&gt;$(deriveAll ''User "PFUser") type instance PF User = PFUser instance XmlPickler User where xpickle = gxpickle&lt;/pre&gt;
&lt;p&gt;Using this automatically derived XML pickler and the query function described above we can now query the DBXML backend for all users that satisfy a certain property:&lt;/p&gt;
&lt;pre&gt;jd :: IO [User] jd = query myCollection "/user[username=$name]" [("name", "jd")]&lt;/pre&gt;
&lt;p&gt;Now we&amp;rsquo;re able to query a database for pieces of XML and reify these as true Haskell values with almost no boilerplate involved! Because of the bidirectional behavior of the &lt;code&gt;XmlPickler&lt;/code&gt; type class it shouldn&amp;rsquo;t be difficult to imagine that the same trick is applicable to inserting and updating database entries. We will discuss writing generic functions using the Regular library in a later post.&lt;/p&gt;</description><link>http://engineering.silk.co/post/31921413529</link><guid>http://engineering.silk.co/post/31921413529</guid><pubDate>Thu, 24 Sep 2009 00:00:00 -0400</pubDate><category>engineering</category><category>haskell</category></item><item><title>Why we use Haskell</title><description>&lt;p&gt;As a newly started company, we have a lot of technical decisions to make. One of the important ones is the choice of a programming language. Since we&amp;rsquo;re building a web application, this goes for both the client (i.e. the web browser) and the server.&lt;/p&gt;
&lt;p&gt;On the client, there wasn&amp;rsquo;t much discussion. Javascript is the only viable choice, unless you want to use Flash or similar plugin-based models. But on the server, we had more freedom. Popular choices for web applications are dynamically typed languages like Ruby, Python and PHP, and statically typed languages like Java and C#.&lt;/p&gt;
&lt;p&gt;However, there was another option. Two of us (me and Sebas) are enrolled in the Software Technology master program at Utrecht University, where we often use the programming language Haskell. Haskell is a functional programming language, which means that a program is composed of expressions instead of statements, as is the case in an imperative programming language.&lt;/p&gt;
&lt;p&gt;As you might have guessed from the title of this post, we decided to use Haskell for our server side programming. What are the features that made us choose Haskell?&lt;/p&gt;
&lt;p&gt;First, we like the functional programming model. Modeling programs based on what values are, instead of what bits to move where, is much closer to the way we think. We write down what we mean, and let the compiler figure out how to calculate it. This also leads to shorter code: Haskell is often surprisingly concise.&lt;/p&gt;
&lt;p&gt;Another important consideration is the strong type system that Haskell has. It has powerful features that prevent you from running incorrect programs, while not getting in the way, as is often the case in traditional statically typed languages like Java or C#. This is due to the fact that Haskell has type inference: it can deduce the types of variables and functions without programmer specifications.&lt;/p&gt;
&lt;p&gt;Something that sets Haskell apart from almost all other programming languages, is its purity: an expression in Haskell is not allowed to do anything on evaluation other than return its value (functions are not allowed to have side-effects). This has many implications. For example, data is immutable: if you modify a list, for example, you create a new copy (although smart compilers share duplicate parts for efficiency). This means that you never have to think about other parts of the code changing your data: you can always reason locally. Another implication is that your language can be lazy (and in fact, Haskell is). This means that expressions are not evaluated unless their results are needed. This can lead to performance improvements, and allows you to use infinite data structures.&lt;/p&gt;
&lt;p&gt;If we disallow side effects, how do we perform I/O, for example? Haskell uses an abstraction called a monad to do this. A monad shows up in the type, indicating that we are performing side effects. The I/O monad allows us to force an ordering on actions, allowing us to safely perform the actions in a lazy setting. We can also use monads to keep state, have mutable variables, and many other things.&lt;/p&gt;
&lt;p&gt;Purity also makes concurrency a lot easier: if data is immutable, there is no risk in sharing it between different threads. If you need to share mutable data between different threads, Haskell offers a few ways of dealing with this, the most interesting of which is software transactional memory (STM), a lock-free way of building composable atomic actions.&lt;/p&gt;
&lt;p&gt;Since Haskell is a functional language, functions are first class: you can create them on the fly, pass them as arguments, return them as results, store them in a data structure, etc. This allows you to easily create powerful abstractions. For example, one often-used Haskell function is map. It takes an argument that turns a value of type a into a value of type b, and uses this to transform a list of a&amp;rsquo;s into a list of b&amp;rsquo;s by applying it to each element. This function abstracts away looping over a list. In a similar way we can create our own abstractions. This often leads to the creation of lightweight EDSLs (embedded domain specific languages), especially since Haskell allows the creation of (infix) operators as well.&lt;/p&gt;
&lt;p&gt;The combination of conciseness, locality of data, strong types and explicit side-effects has a very important consequence: it makes refactoring easy and risk-free. You can change your code, and if it still type checks, you can often be sure that it still works. Combined with the power of abstraction this means we can quickly develop programs that can be easily understood and modified by others.&lt;/p&gt;
&lt;p&gt;Finally, another positive aspect of Haskell is its community. There is an &lt;a href="http://planet.haskell.org/" title="planet haskell"&gt;active blog community&lt;/a&gt;, a large and helpful IRC channel (#haskell) and an &lt;a href="http://hackage.haskell.org/" title="Hackage"&gt;online package repository&lt;/a&gt;. There is an &lt;a href="http://www.haskell.org/ghc/" title="GHC"&gt;industrial strength compiler&lt;/a&gt;, a &lt;a href="http://www.haskell.org/cabal/" title="Cabal"&gt;dependency tracking package manager&lt;/a&gt; and built-in tools like code coverage.&lt;/p&gt;
&lt;p&gt;Of course, there are also a few risks involved in choosing a language like Haskell. Since it doesn&amp;rsquo;t have a user base as large as some other languages, there are less libraries available. This is alleviated in part by a strong C interface that Haskell provides. Because of laziness, performance problems can be harder to diagnose and fix. And of course, there are less programmers who have experience with Haskell, so finding people to help you can be harder.&lt;/p&gt;
&lt;p&gt;But all in all, we feel that the benefits Haskell can bring us outweigh the risks. And most of all, Haskell makes programming fun!&lt;/p&gt;</description><link>http://engineering.silk.co/post/31920990633</link><guid>http://engineering.silk.co/post/31920990633</guid><pubDate>Tue, 22 Sep 2009 00:00:00 -0400</pubDate><category>engineering</category><category>haskell</category></item><item><title>This is our blog</title><description>&lt;p&gt;This is our lablog, where we will write about the proceedings of typ&lt;span class="lab"&gt;LAB&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;‘What’s typ&lt;span class="lab"&gt;LAB&lt;/span&gt; then?’, you might ask.&lt;/p&gt;
&lt;p&gt;typ&lt;span class="lab"&gt;LAB&lt;/span&gt; is our initiative to investigate and develop new ways of creating and consuming web-based content. We aim to explore the boundaries of what’s possible using contemporary internet technologies. The result of our endeavors should enable you to read and write rich, dynamic and smart articles like you’ve never read nor written them before.&lt;/p&gt;
&lt;p&gt;Next question. ‘Who are those “we” you keep referring to?’&lt;/p&gt;
&lt;p&gt;We are Erik Hesselink, Sebastiaan Visser, Salar al Khafaji and Lon Boonen. That’s the team; small, smart, focused and eager to learn and share. Most of us share a history at &lt;a href="http://www.xopus.com/"&gt;Xopus&lt;/a&gt;, the web based XML editor.&lt;/p&gt;
&lt;p&gt;One more question? ‘What about lablog, what can we expect from it?’&lt;/p&gt;
&lt;p&gt;Through lablog we want to inform you of our ponderings and wonderings. Our steps forward, backward and sideways. We’ll write about our progress, our discoveries and the things that make us tick.&lt;/p&gt;
&lt;p&gt;You can expect articles on HTML5, CSS3, web development in general, Haskell, XQuery, usability, semantics, JavaScript and all stuff that makes a nerd’s heart jump a beat. Or two.&lt;/p&gt;
&lt;p&gt;And we&amp;rsquo;d love to hear from you. Through the comments, &lt;a href="http://twitter.com/silkapp"&gt;Twitter&lt;/a&gt; or plain old &lt;a href="mailto:info@silkapp.com"&gt;e-mail&lt;/a&gt;.&lt;/p&gt;</description><link>http://engineering.silk.co/post/31920889003</link><guid>http://engineering.silk.co/post/31920889003</guid><pubDate>Wed, 16 Sep 2009 00:00:00 -0400</pubDate><category>engineering</category></item></channel></rss>
