James Long 2014-08-20T00:00:00Z http://jlongster.com James Long Blog Rebuild: Build Systems & Cross-Compiling 2014-08-14T00:00:00Z 2014-08-14T00:00:00Z http://jlongster.com/Blog-Rebuild--Build-Systems---Cross-Compiling This is an entry in a series about rebuilding my custom blog with react, CSP, and other modern tech. Read more in the blog rebuild series.

A few years ago I remember being surprised at how popular grunt was getting. Not because it wasn't great software, but because I didn't understand what problem it solved. If I needed to process a few things like CSS before deploying to production, make seemed to work just fine.

Back then I thought things like build steps for JavaScript were an unnecessary complexity. I couldn't have been more wrong. A build system adds some complexity, yes, but a good one like gulp or broccoli is simple enough, and the returns are enormous. A complex Makefile for a JavaScript project would be a mistake, but these build tools are great.

tl;dr I chose gulp as my build system and webpack as my client-side module bundler. My final setup is on github, specifically gulpfile.js and webpack.config.js.

A Practical Approach

]]>
This is an entry in a series about rebuilding my custom blog with react, CSP, and other modern tech. Read more in the blog rebuild series.

A few years ago I remember being surprised at how popular grunt was getting. Not because it wasn't great software, but because I didn't understand what problem it solved. If I needed to process a few things like CSS before deploying to production, make seemed to work just fine.

Back then I thought things like build steps for JavaScript were an unnecessary complexity. I couldn't have been more wrong. A build system adds some complexity, yes, but a good one like gulp or broccoli is simple enough, and the returns are enormous. A complex Makefile for a JavaScript project would be a mistake, but these build tools are great.

tl;dr I chose gulp as my build system and webpack as my client-side module bundler. My final setup is on github, specifically gulpfile.js and webpack.config.js.

A Practical Approach

I'm going to be as practical as possible during this rebuild. I'm going to investigate newer things like ES6 modules, but if the tools are too immature I will fallback to something like CommonJS. I want something that works now with little effort.

What I need:

  1. A common module format for the client and server. Node uses CommonJS, and currently browsers do not enforce modules.
  2. For client-side code, a way to compile modules to run in the browser.
  3. An extensible pipeline for hooking in compilation stages for both server and client JS. This lets me hook in various JS transformations that I need.
  4. A watcher that will automatically trigger the necessary compilations to get updates automatically (and only re-compile the necessary files)
  5. Ability to define a few basic build tasks for moving files around and running the app

There are lots of things involved in the above requirements: compilation strategies, module bundling, and build task management. I don't know yet which combination of projects will work out, so let's investigate various solutions.

The main drive for a compilation pipeline is to compile out ES6 features into ES5. I don't want to hook something big like Traceur in because there are projects that compile out specific features better. For example, I want to use regenerator to compile out generators and then defs to compile out let. I've always enjoyed this post about ClojureScript's compilation pipeline, and I'm reminded of it when I think of this strategy of incrementally compiling an AST. Ideally, we will pass an AST around, but we'll see if the tools are good enough for that yet.

Of course, I'm a big fan of sweet.js so that will be the first compilation phase. I may compile out some ES6 features with the es6-macros project, but the reality is that the JS community has written mature ES6 transformations in the form of compilers, so it might make sense just to use them. I will still use macros for user-land syntax extensions, which I'll talk more about in future posts.

The Core Problem

I think the core problem is that the client and server are very different beasts. Node requires CommonJS and modules separated out into individual files. Browsers don't have modules and it's desirable to bundle everything together into a single JS file to deploy. To make things harder, everything should be sourcemapped.

The first question to ask is how a build system can help. Since we want to work with modules, we need support for N:M files at each build step. That means that given N files, a build step can produce M files. For example, given 1 file, a module plugin will return 10 files (all the dependencies), and then the next step could bundle them all together into 1 file.

This is important for watching and incremental builds. If a dependency changes, even if it's not listed directly in the files to watch, the build system should recompile. Additionally, it should only recompile the necessary changes, so it should cache each dependency, even if it's not explicitly listed in the original sources.

The second question to ask is what tools are out there for working with modules. The build system is the backbone, but we need plugins for actually doing things with modules. How well the build system supports N:M files affects how much the module loaders need to do.

Lastly, there's one more desirable feature. There are several transformations I want to do to my code (like sweet.jsregeneratordefs). It would be far better to pass an AST through this process rather than passing strings. This means we probably don't want to hook up this whole pipeline through whatever build system we choose, but wrap it up into a single plugin.

Gulp + Webpack

Gulp is a build system built around streams. One thing I like is that it's very simple to use and define new tasks. (Note: I'm going to skip over grunt because its config syntax is really bad and I just don't like it.)

Gulp supports the N:M file builds in the form of stream events. A plugin can take a single file from a stream and output multiple files. If you add a caching layer with gulp-cache, and use the more advanced gulp-watch, you could effectively pass in one JS file and have it watch and rebuild all of its dependencies.

I'm not sure a lot of people understand that you can do this, which emits 2 files for every file that comes down the stream:

function explode() {
  return es.through(function(file) {
    this.emit('data', new gutil.File({
      base: file.base,
      cwd: file.cwd,
      path: path.join(file.base, 'foo.js'),
      contents: new Buffer('boo')
    }));

    this.emit('data', file);
  });
}

gulp.task("explode", function() {
  gulp.src('input/main.js')
    .pipe(explode())
    .pipe(gulp.dest('output'));
});

Not very many projects use this to help with module bundling, though. There is one project, amd-optimize, that does basic dependency tracing for AMD modules. Still, the more sophisticated gulp-watch is needed if you want to watch new files from the stream (you could apply it after explode()); it is not builtin. Generally, there is very little mature code that integrates a module bundler into gulp. You have to work at it. So this doesn't really solve our problem of compiling modules for client-side. Everyone just uses browserify or webpack.

Additionally, you really only care about your local dependencies, not ones pulled from npm. You don't need to run your code transformations on npm dependencies. So it's really easy to just give the native watch all of your modules by just doing gulp.src('src/**/*.js'). Because of this, and the fact that server-side code doesn't require module bundling, gulp works well for transforming server-side code. This code transforms each file from src and generates files in the build folder with sourcemaps.

function makeNodeStream(src, withoutSourcemaps) {
  var stream = src.pipe(cache('src'))
      .pipe(sourcemaps.init())
      .pipe(sweetjs({
        readableNames: true,
        modules: ['es6-macros']
      }))
      .pipe(regenerator())
      .pipe(jsheader('var wrapGenerator = require("regenerator/runtime/dev").wrapGenerator;'))
      .pipe(jsheader('require("source-map-support");'));

  if(!withoutSourcemaps) {
    stream = stream.pipe(sourcemaps.write('.'));
  }
  return stream;
}

gulp.task("src", function(cb) {
  es.merge(
    makeNodeStream(gulp.src('src/**/*.js'))
      .pipe(gulp.dest('build')),
    makeNodeStream(gulp.src('static/js/shared/**/*.js'))
      .pipe(gulp.dest('build/shared')),
    gulp.src(['src/**/*', '!src/**/*.js']).pipe(gulp.dest('build'))
  ).on('end', function() {
    nodemon.restart();
    cb();
  });
});

An additional complexity is that I have a shared folder that also needs to be transformed and output to a different directory. As for as I could tell, I couldn't combine that into a single gulp.src and gulp.dest, so I created makeNodeStream to run it on both. I also copy anything that's not a JS file from src to the build folder. Lastly, when it's finished it restarts the node process using nodemon.

My transformation pipeline here goes like this: sweet.js → regenerator → header append. I will likely add more steps in the future. This is passing around strings, which I talked about before, when we really should pass around ASTs. One thing I could do is use esnext instead and integrate sweet.js with it, and then do a single pipe to it. It would probably be much faster.

It takes about 2 seconds to compile my whole src directory, which is a bunch of code. But who cares? You don't need to recompile everything when just one file changes! Note that I use the cache('src') step first from gulp-cached; this will cache all files coming through the stream, and only re-emit files that have changed. That means we only transform new files, and it only takes a few hundred ms now.

Client-side

What about client-side code? As mentioned before, even though gulp could be used as a module bundler, nobody does that since mature projects like browserify and webpack exist. I chose to use webpack since I like the API and documentation better (and it has more features).

This basically requires me to use CommonJS modules for the browser. This route is well-established in the JS community so I benefit from mature tools. Eventually I'd like to use ES6 modules, but the ecosystem isn't quite there yet. I'm being conservative here so that I don't spend too much time on my tools.

Now that I'm using webpack, all of my problems for client-side development are solved. It has everything, from code splitting to hot module replacement. Here is my webpack config:

var config = {
  cache: true,
  entry: './static/js/main.js',
  output: {
    filename: './static/js/bundle.js'
  },
  resolve: {
    extensions: ['', '.js', '.sjs'],
    fallback: __dirname
  },
  module: {
    loaders: [
      {test: /\.js$/,
       exclude: [/static\/js\/lib\/.*\.js$/,
                 /node_modules\/.*/],
       loader: 'regenerator!sweetjs?modules[]=es6-macros'},
      {test: /\.less$/, loader: "style!css!less"},
      {test: /\.css$/, loader: "style!css"}
    ]
  }
};

Webpack is explicitly a module bundler, so all it needs is just one file and it will walk the dependencies. Everything will be bundled together into a single file bundle.js. This happens by default, so you can see why this doesn't work for server-side code where we just need a 1:1 file mapping.

This uses a loader on JS files to run them through sweet.js and regenerator. Again, I really should look into esnext so that I don't keep re-parsing the code.

It also uses some really cool loaders to deal with stylesheets. less-loader compiles out lesscss. css-loader is an awesome loader that converts all @import and url statements to require so that everything is resolved the same way, and lets you apply loaders on those resources being loaded, allowing things like inlining the url content straight into the stylesheet. Having everything go through the same mechanism (and able to pull from npm dependencies) is extremely liberating.

To top it all off, style-loader is a loader that automatically adds a style tag to the page when the css file is requireed. It also inlines all the CSS into your JavaScript bundle, but you can also make it reference an external CSS file. Either way, all you have to do is require('css/main.css') in your JavaScript and it just works.

There are a few other things I do with gulp and webpack, mostly to get integration with a few modules pulled down from npm (like React) working. I also have a run task that starts my app and uses nodemon to track it so it can be restarted whenever a change happens.

View my final setup on github.

Broccoli + ES6 modules

Broccoli is a rather new build tool that operates on tree structures, so it gets good incremental rebuilds and watches for free. See the annoucement blog post for more details.

I'm not sure if broccoli competes more with gulp or webpack. It sits somewhere in the middle. It doesn't have any concept of tasks, so I can't make a run task that restarts my server on changes. But it's also not nearly as specific as webpack, and doesn't dictate anything specific about modules or how things are bundled.

I think broccoli makes it a lot easier to write something like webpack, and that's the idea. Basically, in broccoli plugins are always passing around whole trees of files, and a plugin can easily expand a tree into a much bigger tree if needed. This makes it easy to expand dependencies but still leverage the build system to handle them. So watching for changes in dependencies works great, and incremental builds are really fast because it can easily figure out what to do. Webpack has to figure all of this stuff out itself.

I like the idea of broccoli, and because working with modules is easy people are doing a lot of great work to get a workflow for compiling ES6 modules. This plugin integrates es6-module-transpiler with broccoli and does all the dependency stuff.

The thing broccoli could solve for me is not only using ES6 modules, but also to unify the JS transformation between server-side and client-side. Using gulp and webpack, I have two completely separate processes.

This was my first Brocfile.js to see how it would work out:

var pickFiles = require('broccoli-static-compiler');
var sweetjs = require('broccoli-sweetjs');
var transpileES6 = require('broccoli-es6-module-transpiler');

var src = pickFiles('src', {
  srcDir: '/',
  destDir: '/'
});

src = sweetjs(src, {
  modules: ['es6-macros']
});

src = transpileES6(src, { type: 'cjs' });
module.exports = src;

Unfortunately, I immediately ran into a bug and it wouldn't compile my code. Somehow I was using an older version that didn't work with nested yields (I guess a newer version needs to be pushed to npm). These kinds of bugs can easily be fixed.

I also ran into a bigger issue though: that project does not have a good story for integration with npm dependencies yet (more discussion here). With webpack, I could require just require dependencies and it would look in node_modules, and it worked awesomely. I don't know why we can't do something similar with ES6 modules.

There was also another big issue in general with broccoli: sourcemaps. The sourcemap story for broccoli is very vague (es6-module-transpiler supports them just fine, but I don't know how to expand with sweet.js and pass it the result & sourcemaps and make it combine them). The standard project broccoli-filter which is supposed to be used by plugins that simply map files 1:1 states right in the README that is does not support sourcemaps. That is insane to me and I can't think about using broccoli until sourcemaps are deeply integrated through and through. Also see this discussion.

In gulp, it's really easy with the awesome gulp-sourcemaps project. You just hook into the stream and write sourcemaps to a directory:

src.pipe('src/**/*.js')
  .pipe(sourcemaps.init())
  .pipe(sweetjs())
  .pipe(regenerator())
  .pipe(sourcemaps.write('.'));

Plugins have a standard method of applying sourcemaps. The sourcemap is attached to the File instances that are passed through the stream, and combined using vinyl-sourcemaps-apply. It looks like this:

var applySourceMap = require('vinyl-sourcemaps-apply');
// ...
if(myGeneratedSourceMap) {
  applySourceMap(file, myGeneratedSourceMap);
}

That incrementally combines sourcemaps as they are applied through the streams. It has worked out really well for me.

Even without all these problems, the story in general for browser-side module bundling isn't nearly as strong as browserify or webpack, which have tons of features specific for browser modules. So until we get a solid build system that has plugins that implement most of those features of a module bundler, right now using gulp/broccoli + browserify/webpack works pretty darn well.

Most likely, I will switch my project to ES6 modules when can find a good cross-compiler that works well with CommonJS and my current build system.

I could use broccoli and webpack, but at this point I'm just going to stick with gulp. It's easy to use and works really well with server-side transformation and sourcemaps. As for broccoli, I understand the design and I like it, but it does make plugin development very complicated and I'm not entirely sold on it, especially when you can do N:M compilations with gulp. Lastly, it uses temporary files so gulp is potentially faster with streams.

Stream of Thought EOF

There are several other build systems out there and a million ways to combine them. I can't possibly cover all of them, but I hope this gave some insight into my process for researching. I have something that works well, and the only thing I'll improve in the future is using ES6 modules instead of CJS.

View the full repo to see all the glorious code. Specifically, check out the full gulpfile.js and webpack.config.js. What's neat about this set up is I can run webpack from the CLI like normal, but it's also defined as a task so gulp webpack will work and it can be used as a task dependency (for tasks like gulp all). I can switch between the systems easily.

I'm sure I have made some errors in this post, as it was mostly stream of thought as I was doing my research. If something is completely off, let me know.

]]>
Blog Rebuild: A Fresh Start 2014-07-30T00:00:00Z 2014-07-30T00:00:00Z http://jlongster.com/Blog-Rebuild--A-Fresh-Start About two years ago I wanted to start blogging more seriously, focusing on in-depth tech articles and tutorials. Since then I've successfully made several posts like the one about games and another about react.js.

I decided to write my own blog from scratch to provide a better blogging experience, and it has served me well. I didn't want something big and complicated to maintain like Wordpress, and I had used static generators before but in my opinion you sacrifice a lot, and there's too much friction for writing and updating posts.

Back then I wanted to learn more about node.js, redis, and a few other things. So I wrote a basic redis-backed node.js blogging engine. In a few months (working here and there), I had a site with all the basic blog pages, a markdown editor with live preview, autosaving, unpublished drafts, tags, and some basic layout options. Here is the current ugly editor:

Redis is an in-memory data store, and node handles multiple connections well by default, so my simple site scales really well. I've have posts reach #1 on hacker news with ~750 visitors at the same time for hours (reaching about 60,000 views) with no problem at all. It may also help that my linode instance has 8 cores and I load up 4 instances of node to serve the site.

]]>
About two years ago I wanted to start blogging more seriously, focusing on in-depth tech articles and tutorials. Since then I've successfully made several posts like the one about games and another about react.js.

I decided to write my own blog from scratch to provide a better blogging experience, and it has served me well. I didn't want something big and complicated to maintain like Wordpress, and I had used static generators before but in my opinion you sacrifice a lot, and there's too much friction for writing and updating posts.

Back then I wanted to learn more about node.js, redis, and a few other things. So I wrote a basic redis-backed node.js blogging engine. In a few months (working here and there), I had a site with all the basic blog pages, a markdown editor with live preview, autosaving, unpublished drafts, tags, and some basic layout options. Here is the current ugly editor:

Redis is an in-memory data store, and node handles multiple connections well by default, so my simple site scales really well. I've have posts reach #1 on hacker news with ~750 visitors at the same time for hours (reaching about 60,000 views) with no problem at all. It may also help that my linode instance has 8 cores and I load up 4 instances of node to serve the site.

You may wonder why I don't just use something like ghost, a modern blogging platform already written in node. I tried ghost for a while but it's early software, includes complex features like multiple users which I don't need, and most importantly it was too difficult to implement my ideas. This is the kind of thing where I really want my site to be my code; it's my area to play, my grand experiment. For me, it's been working out really well (check out all of my posts).

But the cracks are showing. The code is JavaScript as I wrote it 2 years ago: ugly callbacks, poor modularity, no tests, random jQuery blobs to make the frontend work, and more. The site is stable and writing blog posts works, but implementing new features is pretty much out of the question. Since this is my site and I can do whatever I want, I'm going to commit the cardinal sin and rewrite it from scratch.

I've learned a ton over the past two years, and I'm really excited to try out some new techniques. I have a lot of the infrastructure set up already, which uses the following software:

  • react — for building user interfaces seamlessly between client/server
  • react-router — advanced route handling for react components
  • js-csp — CSP-style channels for async communication
  • mori — persistent data structures
  • gulp — node build system
  • webpack — front-end module bundler
  • sweet.js — macros
  • es6-macros — several ES6 features as macros
  • regenerator — compile generators to ES5

Check out the new version of the site at new.jlongster.com. You can see my progress there (right now it's just a glimpse of the current site). I will put it up on github soon.

I thought it would also be interesting to blog throughout the development process. I'm using some really interesting libraries in ways that are very new, so I'm eager to dump my thoughts quite often. You can expect a post a week, explaining what I worked on and how I'm using a library in a certain way. It will touch on everything such as build systems and cross-compiling, testing, front-end structuring. Others might learn something new as well.

Next time, I'll talk about build systems and cross-compiling infrastructure. See you then!

]]>
Compiling JSX with Sweet.js using Readtables 2014-07-02T00:00:00Z 2014-07-02T00:00:00Z http://jlongster.com/Compiling-JSX-with-Sweet.js-using-Readtables JSX is a Facebook project that embeds an XML-like language in JavaScript, and is typically used with React. Many people love it and find it highly useful. Unfortunately it requires its own compiler and doesn't mix with other language extensions. I have implemented a JSX "compiler" with sweet.js macros, so you can use it alongside any other language extensions implemented as macros.

I have a vision. A vision where somebody can add a feature as complicated as pattern matching to JavaScript, and all I have to do is install the module to use it. With how far-reaching JavaScript is today, I think this kind of language extensibility is important.

It's not just important for you or me to be able to have something like native goroutines or native syntax for persistent data structures. It's incredibly important that we use features in the wild that could potentially be part of a future ES spec, and become standardized in JavaScript itself. Future JavaScript will be better because of your feedback. We need a modular way to extend the language, one that works, seamlessly with any number of extensions.

I'm not going to explain why sweet.js macros are the answer to this. If you'd like to hear more, watch my JSConf 2014 talk about it. If you are already writing a negative comment about this, please read this first.

So why did I implement JSX in sweet.js? If you use JSX, now you can use it alongside any other macros available with jsx-reader. Want native syntax for dealing with persistent data structures? Read on...

]]>
JSX is a Facebook project that embeds an XML-like language in JavaScript, and is typically used with React. Many people love it and find it highly useful. Unfortunately it requires its own compiler and doesn't mix with other language extensions. I have implemented a JSX "compiler" with sweet.js macros, so you can use it alongside any other language extensions implemented as macros.

I have a vision. A vision where somebody can add a feature as complicated as pattern matching to JavaScript, and all I have to do is install the module to use it. With how far-reaching JavaScript is today, I think this kind of language extensibility is important.

It's not just important for you or me to be able to have something like native goroutines or native syntax for persistent data structures. It's incredibly important that we use features in the wild that could potentially be part of a future ES spec, and become standardized in JavaScript itself. Future JavaScript will be better because of your feedback. We need a modular way to extend the language, one that works, seamlessly with any number of extensions.

I'm not going to explain why sweet.js macros are the answer to this. If you'd like to hear more, watch my JSConf 2014 talk about it. If you are already writing a negative comment about this, please read this first.

So why did I implement JSX in sweet.js? If you use JSX, now you can use it alongside any other macros available with jsx-reader. Want native syntax for dealing with persistent data structures? Read on...

The Problem

JSX works like this: XML elements are expressions that are transformed to simple JavaScript objects.

var div = <div>
  <h1>{ header }</h1>
</div>;

This is transformed into:

var div = React.DOM.div(null, React.DOM.h1(null, header));

JSX could not be implemented in sweet.js until this week's release. What made it work? Readtables.

I will briefly explain a few things about sweet.js for some technical context. Sweet.js mainly works with tokens, not an AST, which is the only way to get real composable language extensions *. The algorithm is heavily grounded in decades of work done by the Lisp and Scheme communities, particularly Racket.

The idea is that you work on a lightweight tree of tokens, and provide a language for defining macros that expand these tokens. Specialized pattern matching and automatic hygiene make it easy to do really complex expansion, and you get a lot for free like sourcemaps.

The general pipeline looks like this:

  • read - Takes a string of code and produces a tree of tokens. It disambiguates regular expression syntax, divisions, and all kinds of other fun stuff. You get back a nice tree of tokens that represent atomic pieces of syntax. It's a tree because anything inside a delimiter (one of {}()[]) exists as children of the delimiter token.
  • expand - Takes the token tree and walks through it, expanding any macros that it finds. It looks for macros by name and invokes them with the rest of the syntax. This phase also does quite a bit of light parsing, like figuring out if something is a valid expression.
  • parse - Takes the final expanded tree and generates a real JavaScript AST. Currently sweet.js uses a patched version of esprima for this.
  • generate - Generate the final JavaScript code from the AST. sweet.js uses escodegen for this.

The expand phase essentially adds extensibility to the language parser. This is good for a whole lot of features. There are a few features, like types and modules, that require knowledge of the whole program, and those are better off with an AST somewhere in between the parse and generate phase. Currently we don't have extensibility there yet.

There are very rare features that require extensibility to the read phase, and JSX is one of them. Lisp has had something called readtables for a long time, and I recently realized that we needed something like that in sweet.js in order to support JSX. So I implemented it!

There are a few reasons JSX needs to work as a reader and not a macro:

  1. Most importantly, the closing tag </name> is completely invalid JavaScript syntax. The default reader thinks it's starting a regular expression but it can't find the end of it, so it just errors in the read phase.
  2. JSX has very specific rules about whitespace handling. Whitespace inside elements needs to be preserved, and a space is added between sibling expressions, for example. Macros don't know anything about whitespace.

Reader extensions allow you install a custom reader that is invoked when a specific character is encountered in the source. You can read as much of the source as you need, and return a list of tokens. Reader extensions can only be invoked on punctuators (symbols like < and #), so you can't do awful things like change how quotes are handled.

A readtable is a mapping of characters to reader extensions. Read more about how this works in here in the docs.

* I have a lot more opinions about this which I will expand in future posts. Working with ASTs are great for features requiring whole programs, like types or optimization passes. Read this for a great explanation about these ideas.

Available on npm: jsx-reader

Now that we have readtables, we can implement JSX! I have done exactly that with jsx-reader. It is a literal port of the JSX compiler, with all the whitespace rules and other edge cases kept in tact (hopefully).

To load a reader extension, pass the module name to the sweet.js compiler sjs with the -l flag. Here are all the steps you need to expand a file with JSX:

$ npm install sweet.js
$ npm install jsx-reader
$ sjs -l jsx-reader file.js

Of course, you can load any other macro with sjs as well and use it within your file. That's the beauty of composable language extensions. (Try out es6-macros.)

I have also created a webpack loader and a gulp loader that is up-to-date that supports readtable loading.

This is beta software. I have tested it across the small test cases that the original JSX compiler includes, in addition to many large files, and it works well. However, there are likely small bugs and edge cases which need to be fixed as people discover them.

Not only do you get reliable sourcemaps (pass -c to sjs), but you also get better error messages than the original JSX compiler. For example, if you forget to close a tag:

var div = <div>
    <h1>Title</h1>
    <p>
</div>

You get a nice, clean error message that directly points to the problem in your code:

SyntaxError: [JSX] Expected corresponding closing tag for p
4: </div>
     ^
    at Object.readtables.readerAPI.throwSyntaxError (/Users/james/tmp/poop/node_modules/sweet.js/lib/parser.js:5075:23)

One downside is that this will be slower than the original JSX compiler. A large file with 2000 lines of code will take ~.7s to compile (excluding warmup time, since you'll be using a watcher in almost all projects), while the original compiler takes ~.4s. In reality, it's barely noticeable, since most files are a lot smaller and things compile in a matter of a couple hundred milliseconds most of the time. Also, sweet.js will only get more optimized with time.

Example: Persistent Data Structures

React works even better when using persistent data structures. The problem is that JavaScript doesn't have them natively, but luckily libraries like mori are available. The problem is that you can't use object literals anymore; you have to do things like mori.vector(1,2,3) instead of [1,2,3].

What if we implement literal syntax for mori data structures? Potentially you could use #[1, 2, 3] to create persistent vectors, and #{x: 1, y: 2} to create persistent maps. That would be awesome! (Unfortunately, I haven't actually done this yet, but I want to.)

Now everyone using JSX will be able to use my literal syntax for persistent data structures with React. That is truly a powerful toolkit.

The JSX Read Algorithm

Adding new syntax to JavaScript, especially a feature with a large surface area like JSX, must be done with great care. It has to be 100% backwards-compatible, and done with future ES.next features in mind.

Both jsx-reader and the original JSX compiler look for the < token and trigger a JSX expression parse. There's a key difference though. You may have noticed that jsx-reader, as a reader, is invoked from the raw source and has no context. The original JSX compiler monkeypatches esprima to invoke < only when parsing an expression, so it is easier to guarantee correct parsing. < is never valid in expression position in JavaScript, so it can get away with it.

jsx-reader is invoked on < whenever it occurs in the source, even if it's a comparison operator. That sounds scary, and it is; we need to be very careful. But I have figured out a read algorithm that works. You don't always need a full AST.

jsx-reader begins parsing < and anything after it as a JSX expression, and if it finds something unexpected at certain points, it bails. For the most part, it's able to figure out extremely early whether or not < is really a JSX expression or not. Here's the algorithm:

  1. Read <
  2. Read an identifier. If that fails, bail.
  3. If a > is not next:
    1. Skip whitespace
    2. Read an identifier. If that fails, bail.
    3. If a > is next, go to 4.1
    4. Read =
    5. If a { is next, read a JS expression similar to 4.3.1
    6. If a < is next, go to 1
    7. Otherwise, read a string literal
    8. If a > is next, go to 4.1
    9. Otherwise, go to 3.1
  4. Otherwise:
    1. Read >
    2. Read any raw text up until { or <
    3. If a { is next:
      1. Read {
      2. Read all JS tokens up until }
      3. Read }
      4. Go to 4.2
    4. If a < is next:
      1. If a < and / is next, go to 5
      2. Otherwise, go to 1
    5. If at end of file, bail.
  5. Read <
  6. Attempt to read a regular expression. If that succeeds, bail.
  7. Read /
  8. Read an identifier. Make sure it matches opening identifier.
  9. Read >

I typed that out pretty fast, and there's probably a better way to format it, but you get the idea. The gist is that it's easy to disambiguate the common cases, but all the edge cases should work too. The edge cases aren't very performant, since our reader could do a bunch of work and then trash it, but though should never happen in 99.9% of code.

In our algorithm, if we say read without a corresponding "if it fails, bail", it throws an error. We can still give the user good errors while disambiguating the edge cases of JavaScript.

Here are a few cases where our reader bails:

  • if(x < y) {} - it bails because it looks for an attribute after y, and ) is not a valid identifier character
  • if(x < y > z) {} - it bails because it reads all the way to the end of the file and doesn't find a closing block. This only happens with top-level elements, and is the worst case performance-wise, but x < y > z doesn't do what you think it does and nobody ever does that.
  • if(x < div > y < /foo>/) {} - this is the most complicated case, and is completely valid JavaScript. It bails because it reads a valid regex at the end.

We take advantage of the fact that expressions like x < y foo don't make sense in javascript. Here, it looks for either a = to parse an attribute or a > to close the element, and errors if it doesn't find it.

Are You Concerned?

Macros invoke mixed feelings in some people, and many are hesitant to think they are a good thing to use. You may think this, and argue that things like readtables are signs that we have gone off the deep end.

I ask that you think hard about sweet.js. Give it 5 minutes. Maybe give it a couple of hours. Play around with it: set up a gulp watcher, install some macros from npm, and use it. Don't push back against it unless you actually understand the problem we are trying to solve. Many arguments that people give don't make sense (but some of them do!).

Regardless, even if you think this isn't the right approach, it's certainly a valid one. One of the most troubling things about the software industry to me is how vicious we can be to one another, so please be constructive.

Give jsx-reader a try today, and please file bugs if you find any!

]]>
Removing User Interface Complexity, or Why React is Awesome 2014-05-13T00:00:00Z 2014-05-13T00:00:00Z http://jlongster.com/Removing-User-Interface-Complexity,-or-Why-React-is-Awesome I've been studying frameworks and libraries like Ember, Angular, and React the past several months, and given Web Components a lot of thought. These all solve similar problems to varying degrees, and are in conflict in some ways and in harmony in others.

I'm mostly concerned with the core problems of data binding and componentizing UIs. After much research and usage of the technologies mentioned above, I found React to provide the best solution.

I ask that you set aside your framework prejudices and read this post with an open mind. This post is not to evangelize React specifically, but to explain why its technique is profound. I want developers steeped in other technologies to take a hard look at these techniques, particularly those involved in Web Components.

This post actually started as a call to address problems with Web Components. But I've come to realize a few things: the bottom half of the Web Components effort (unlocking all the builtin stuff so that even native tags could be rewritten) is a great thing, Web Components are a step up from current technology regardless, and I don't want to do harm to that effort. My concerns are quite vague as well, so it doesn't make a good blog post. I can't reconcile what I want components to be with Web Components, and it's much better if I fully explain what I think is the best solution and leave it up to the readers to apply if they choose.

]]>
I've been studying frameworks and libraries like Ember, Angular, and React the past several months, and given Web Components a lot of thought. These all solve similar problems to varying degrees, and are in conflict in some ways and in harmony in others.

I'm mostly concerned with the core problems of data binding and componentizing UIs. After much research and usage of the technologies mentioned above, I found React to provide the best solution.

I ask that you set aside your framework prejudices and read this post with an open mind. This post is not to evangelize React specifically, but to explain why its technique is profound. I want developers steeped in other technologies to take a hard look at these techniques, particularly those involved in Web Components.

This post actually started as a call to address problems with Web Components. But I've come to realize a few things: the bottom half of the Web Components effort (unlocking all the builtin stuff so that even native tags could be rewritten) is a great thing, Web Components are a step up from current technology regardless, and I don't want to do harm to that effort. My concerns are quite vague as well, so it doesn't make a good blog post. I can't reconcile what I want components to be with Web Components, and it's much better if I fully explain what I think is the best solution and leave it up to the readers to apply if they choose.

The Bloop Library

I'm not going to focus on React specifically. In fact, just in case it would be distracting, we're not going to use React at all.

We're going to build our own library from scratch that is highly inspired by React. This lets us play with all kinds of ideas with a very small amount of code. This is prototype-quality, of course, and wouldn't scale to large UIs, but using something like React would make it scale.

Let's call our library Bloop. Our library should let us change data and transparently keep the UI in sync. It should allow us to structure the UI into components, and flow data between them sanely.

First, we need a way to represent behavior and structure. It's easy to write behaviors as JavaScript, but what about the structure? We could use a templating language that looks like HTML, but it's much easier just to represent it in JavaScript too. It keeps the component together, makes it trivial to wire up the structure with behaviors, and allows us to easily share the component. Just imagine being able to use JavaScript for importing a component:

var MyToolbar = require('shared-components/toolbar');

You can't even do something as basic as that with Web Components.

This is what a component looks like with our library. It shows a number and a button that, when pressed, increments the number. getInitialState returns the initial state of the object that can be accessed with this.state.

var Box = Bloop.createClass({
  getInitialState: function() {
    return { number: 0 };
  },

  updateNumber: function() {
    this.state.number++;
  },

  render: function() {
    return dom.div(
      this.state.number,
      dom.button({ onClick: this.updateNumber }, 'Increment')
    );
  }
});

Next, we need a way to render it into the page. You can use renderComponent to render a component instance into a DOM element.

Bloop.renderComponent(Box(), document.body);

There's a problem with this though: it doesn't rerender whenever the state changes. The Box component mutates its state directly, so we don't know when something has changed. We could add a set method that tells us what is changing, but then we need to figure out which pieces of the DOM to update, which gets really complicated. Here's an idea: rerender the entire app whenever a change is made. That way, it's really easy to keep everything in sync.

But how do we know when a change is made? A set method to change state could trigger a rerender, but there's an even easier way to react to changes: continuously call renderComponent with requestAnimationFrame to repaint the UI, and only change the DOM when the component returns different content. React does not do this by default; it has a setState method to change state that triggeres a repaint. Let's just have some fun with requestAnimationFrame, even though you would never do this in production. View the full source code for this example here.

var box = Box();

function render() {
  Bloop.renderComponent(box, document.body);
  requestAnimationFrame(render);
}

render();

Rerendering everything (and only applying it to the DOM when something actually changed) vastly simplifies the architecture of our app. Observables+DOM elements is a leaky abstraction, and as a user I shouldn't need an intimate knowledge of how the UI is kept in sync with my data. This architecture opens up lots of various ways to optimize the rendering, but it's all completely transparent to the user.

The Bloop library is only 250 lines of JavaScript, and simply renders a component by setting innerHTML if changed. Since you usually have a top-level App component, that means the entire app is rendered with innerHTML. I told you it was prototype-quality, right? In a twisted, absurd way, it's shocking how far you can go with this. And the kicker is you can easily swap this out with React to get much better rendering performance, since React has a virtual DOM and will only touch the real DOM when needed. That also solves various problems like rerendering forms and other controls which have focus.

Since everything is rerendered on update, we've decoupled data binding and views and libraries can reuse it in many different ways like Om. Even though React implements complicated optimizations, the mental footprint is as small as our Bloop library which is a breathe of fresh air amid other complicated data binding technologies.

Data Flow

We expressed our component's structure in JavaScript instead of trying to mangle it into the DOM. This makes data flow very natural, since we have direct access to the component instance. If you use the DOM directly, it's common to have to wire up the data flow in JavaScript afterwards. A templating engine with automatic bindings helps, but it still creates more complexity than necessary. The days of having one big HTML file are gone; components and their behaviors are intimately dependant and should be encapsulated like so.

Aren't you tired of having to query the DOM tree and manually manage the structure to create UIs? Web Components doesn't solve this at all, it just tries to encapsulate the work. The problem is that building apps is building components, so you inevitably are forced back into the manual DOM management to create your app-specific components (like how you constantly have to create directives in Angular). You also need to jump into JavaScript to configure and wire up any Web Components you used. It's a very messy abstraction, and fools you into desiring a pure HTML-based declarative way to write apps, which is like wanting steak but eating liver.

Frameworks like Ember and Angular help a lot with this. However, templates and controllers are still separated and data binding still leaks into your app (especially in Angular, $scope.$apply is the devil). Ember does a better job than most, but you still have to declare data dependencies manually (computed properties) and ultimately accept its way of modelling data. That's not a bad thing but there are things you can't do that we will study later in this article. It simply comes down to the framework vs. library debate.

In Bloop, there's a clear and simple flow of data: data is passed down and events flow up. Here is the same example as before, but with multiple components to demonstrate data flow. In all of my examples, you can assume dom is set to Bloop.dom and that the App component is continuously rendered with requestAnimationFrame like you saw in the first section.

There are two components: Toolbar which makes a few buttons that change the number, and App which is our top-level component that uses Toolbar. App has state: the current value of the number. It passes this state into Toolbar, so that toolbar can decrement and increment the number. But Toolbar never touches our app state; it can make a new number, and call the onChange handler with the new number, but it can't do anything else. It's up to the App component to bind the onChange handler to one of its methods which takes the new number and actually modifies the state.

This introduces another aspect of Bloop: properties. Properties are available as this.props and represent the data passed into the component. Components should never mutate their properties. View the full source for this example here.

var App = Bloop.createClass({
  getInitialState: function() {
    return { number: 0 };
  },

  updateNumber: function(value) {
    this.state.number = value;
  },

  render: function() {
    return dom.div(
      dom.span(this.state.number),
      Toolbar({
        number: this.state.number,
        onChange: this.updateNumber
      })
    );
  }
});

var Toolbar = Bloop.createClass({
  render: function() {
    return dom.div(
      dom.button({
        onClick: this.props.onChange.bind(null, this.props.number - 1)
      }, 'decrement'),
      dom.button({
        onClick: this.props.onChange.bind(null, this.props.number + 1)
      }, 'increment')
    );
  }
});

State flows down the components, and events flow up. Note that while properties should never be changed, state is mutable. Properties can't be changed because they are inherited every time the component is rendered, so any changes will be lost.

The difference between state and properties can be useful. It makes it clear what state the component owns. It's best to keep most of your components stateless, and isolate state into as few places as possible. This makes it easy to rationalize about your app and how it changes over time. We will explore this more in the next section.

This example is trivial and so far we've been pretty abstract. You will see more complex examples throughout this post. Communication between components is difficult to do right, and Bloop and React provide you with this simple data flow for handling it, even if it may be limiting in certain cases (you many find yourself adding a lot of trivial methods to App to change app state). Check out this and this page from React about it. Near the end of article, we will explore modifications to this approach.

Let's quickly look at a more complex example. This app has tabs that switch between content, and a settings pane that lets you customize it. View the code here. There is a Tabs component that responds to tab changes by calling the onChange handler, and we bind that to the changeTab method on our top-level App component.

Additionally, there is a Settings component that renders the settings form. Whenever a setting changes, all it does is call its onChange property, which we bound to the changeSetting method on App.

Fundamentally, this is a declarative way to construct components. The render method continuously constructs a new UI based on simple JavaScript objects, binding specific events to component methods.

Explicit App State

You're probably already familiar with the pattern of attaching event handlers to components. As described above, Bloop takes this even further though, making it clear that events flow upward, and since everything is in JavaScript it's dead simple to take a JavaScript function and bind it right onto the component. You do this declaratively within render when creating the component.

There's a deeper reason why it's so important to make data flow clear and simple: it encourages you to keep state in as few places as possible and make most of your components stateless. It's easy to create complex data flows in Bloop with many small components, and keep track of what's going on. Additionally, instead of setting state directly on a component instance, Bloop makes it explicitly a separate JavaScript object. Components that do have state can access it with this.state. A component that uses state must implement a getInitialState method that returns the initial state object.

Tearing apart state from the component instance turns out to be really powerful. It fits well with the model that most of our state is held at the top-level, since most of your UI is now described in one simple JavaScript object. This has far-reaching consequences.

  1. It's adaptable. The state object doesn't have to be a native JavaScript object; it can be anything you return in getInitialState (or your own way of passing state around, if you choose). Want to use persistent data structures instead? Go ahead!
  2. It's easy to snapshot. Given a specific state, you can guarantee what the resulting HTML of a component will be. If you simply save the state object somewhere, you can load it up later and render your component exactly like it was when you saved it. An undo system is trivial: just save the state, and restore it (which is especially trivial with persistent data structures).
  3. It's easy to test and prerender. Similar to point #2, you can easily test components by rendering them with a specific state to an HTML string and comparing the output. You can even manually fire off event handlers which change state and test the changes. Finally, prerendering on the server is as trivial as it sounds: render the top-level component to a string and serve it up, and when loaded on the client the library will bind all the event handlers to the prerendered DOM.

The principle to learn is that things like DOM elements are basically native objects, like an open file instance. You don't stick user-land state onto file instances, do you? They are unserializable and slow. Since the DOM doesn't contain our app state, we just have to deal with a simple JavaScript object, declaratively render a structure based on it, and let the library figure out how to reify it into DOM elements.

The kicker is that the opportunity for great performance falls out of this naturally. React creates a lightweight Virtual DOM every time components are rendered, and diffs them to figure out the smallest amount of real DOM changes needed. The result is astonishing performance since DOM changes are the slowest part. Of course, you don't have to do this; Bloop naively sets innerHTML if the contents have changed, but the abstraction is there to allow great optimization.

Ok, enough philosophizing, let's get to some examples.

Looking at our app with tabs again, there is a Tabs component. This is a completely stateless component, and the top-level App component actually handles the tab change and changes selectedItem in the app state. You might think that Tabs should handle the state to be reusable, but if you think about it, something needs to be hooked up to change the panes when a tab is changed. This makes the dependency on that state explicit and easy to rationalize about.

In fact, all of our app state is a single object attached to the App component. It's easy to expose this as an editor, which is what you see below. This is the raw state of the app, and is synced both ways. Change it manually in the textarea below (change bigFonts to true, for example), and click around the app to watch changes come in.

When you change it in the textarea above, it is sent to the app and applied by running app.state = JSON.parse(msg.data). It's that easy.

Generally, your app state will roughly correspond to your UI structure. I think this is what efforts like Web Components are trying to do (hide details about the UI structure), but it doesn't really work if you insist on still using the DOM. This is what you really want: a stripped down, bare representation of your app. Your UI structure falls out it of this, not the other way around.

The app state is so simple and easy to access. Let's implement an undo system and see if we were telling the truth about how easy it is. Let's use a different example app for this: a basic twitter clone (code here).

Type some messages into the text input, and press "enter" to submit them. Click the star to "star" a few of them. Now press the "undo" button several times and you will watch your previous actions wash away.

How is this implemented? Here is the code:

var prevStates = [JSON.stringify(appState)];

function undo() {
  while(1) {
    var state = JSON.parse(prevStates.pop());

    if(!prevStates.length) {
      // This is the initial app state, so unconditionally apply it
      // and push it back onto the history so we don't lose it
      appState.feed = state.feed;
      prevStates.push(JSON.stringify(state));
      break;
    }
    else if(JSON.stringify(appState.feed) !== JSON.stringify(state.feed)) {
      // We found a state where the feed has changed, so apply it
      appState.feed = state.feed;
      break;
    }
  }
}

function render() {
  app.state = appState;
  var changed = Bloop.renderComponent(app, document.body);
  if(changed) {
    prevStates.push(JSON.stringify(appState));
  }
  requestAnimationFrame(render);
}

render();

All we have to do is save the app state when it is changed, and apply a previous state when an undo is requested. There are a few architecture-specific details here: Bloop doesn't currently have an event when the state changes, but Bloop.renderComponent does return if the rendered output has changed, so we use that to detect when we should save the state. And since we are using simple JavaScript objects, we use JSON.parse and JSON.stringify to take snapshots of the state. This is very simplistic, but you could implement more powerful ways to track changes like using persistent data structures.

Note that we only undo changes to the feed. In undo, we walk back through the history and find an app state where the feed structure has changed, skipping over any other state changes. It's up to you to determine what you want to track and undo.

If you're undoing UI that is backed by a data store, you also need to perform the undo in the backend. You can use a versioned data store, which makes it just as trivial. Or you can diff the app state and generate actions to perform the undo. This is something that needs to be explored more, and is the reverse of usual undo methods, where you manually undo changes in the database and then re-fetch data and refresh the whole UI. That might be just fine for your app, but it gets tedious to hand-code undo code paths for every single model. This is far more powerful because it's automatically applicable to any component.

Forcing your data through the backend data store to allow undo is also limiting. What if you want to not actually persist the action for 30 seconds, and give the user the chance to undo before it even hits the backend? Our architecture makes it trivial to go ahead and update the UI as if it has persisted in the data store, but let the user undo the action before it actually hits the backend. We get all of this for free.

This pixel editor named Goya, written in Om which is built on top of React, leverages this technique for tracking history, and allowing you to travel through changes. Here's all the code it took to implement undo/redo. When it's independent of your data structure (as a good abstraction should be!) it's trivial!

One last example of explicit app state: prerendering and testing. Instead of using Bloop.renderComponent, you can use Bloop.renderComponentToString to render a component with properties to a string of HTML. This makes it trivial to test, just render the component and compare the output. This is the toolbar from the example on the right (if you open the example in a tab, you can run this code from the console):

Bloop.renderComponentToString(Toolbar({ username: 'foo' }))

// Output:
// "<div class="toolbar"><em>Logged in as foo</em><button>settings</button><button>undo</button></div>"

You might not want to compare strings directly, but you could do specific tests to make sure that changes in state relate to changes in structure.

If you want to prerender your app, you could simply render the top-level App component with the initial app state on the server and send it to the client. The library can then simply bind event listeners to the already rendered DOM (Bloop does not support this, but React does). It's literally just a few lines of code.

Bloop itself is extremely simplistic, but we already get most of this for free. React makes it easy to use these techniques for real apps because it handles all the hard stuff, too. There's a lot of win here, and a lot of opportunity for interesting advancements.

Game Loop

So far, we've said that the structure of a component created within the render method is declarative. This is because you generate the structure based off of the app state, nothing else. As the app state changes, so does your structure. However, it may not look like traditionally declarative code, since any JavaScript can be run:

Bloop.createClass({
  render: function() {
    var items = this.props.items.filter(function(item) {
      return item.isVisible;
    });

    return items.map(function(item) {
      return dom.div(item.name);
    });
  }
})

There is a declarative current running underneath this code. But we can look at it in a different light: almost as if we're operating in an immediate rendering mode, as if the dom.div and such functions painted the element instantly. In fact, since we're using requestAnimationFrame to repaint the UI, this is extremely similar to how game developers render UI in games.

Game developers discovered immediate-mode graphical user interfaces years ago (watch that video, it's awesome). Their power lies in the fact that you just run normal JavaScript to paint your UI: conditional elements are literally expressed as if(shown) { renderItem(); }, and that data is always synced with the UI because the UI is always redrawn.

The web traditionally operates in retained mode, where the DOM exists as an in-memory representation of the current UI, and you poke it to make changes. Since we can't throw away the current web, our library still creates DOM elements using our declarative forms. So we're basically operating in an immediate mode on top of a retained mode, and I'm starting to think that it actually gets us the best of both worlds. React provides "lifecycle" methods which trigger at various stages within the retained DOM, which gives you an escape hatch when you need to handle things like focus. Even if it might be better for React if there was a lower-level rendering API, just using the DOM works pretty well.

If our library can make edits to the retained DOM fast enough, we can actually treat our render methods as if they were in immediate mode. That means we can implement performance-sensitive things like 60 FPS animations, or a UI that changes when scrolling. You may think it's taboo not to use CSS for animations, but with requestAnimationFrame and other advancements, people are finding out that you can actually use JavaScript for better and even more performant animations, as seen with Velocity.js.

React, with its Virtual DOM, is fast enough to implement animations that depend on user input, like scrolling or cursor position. Bloop is dumb and uses innerHTML so it's not nearly as good, but on desktop it's at least good enough to show an example.

A wonderful thing about immediate mode is that it's easy to do things like occlusion culling. Another corollary to graphics engines, occlusion culling is an algorithm to optimize rendering by figuring out which elements are actually visible, and only rendering them. Imagine you have a list of 5000 items. If you create a big <ul> with all of them, the DOM will grow large, take up lots of memory, and scrolling will be degraded (especially on mobile). If you know only 25 are on the screen at once, why do we need to create 5000 DOM elements?

You should only need 25 DOM elements at one time, and fill them out with the 25 elements that pass the occlusion test. I made this component in just a few minutes to make this work (view the full code here):

var App = Bloop.createClass({
  getInitialState: function() {
    return { pageY: 0,
             pageHeight: window.innerHeight };
  },

  componentDidRender: function() {
    var numItems = this.props.items.length;
    document.querySelector('.list').style.height = numItems * 31 + 'px';
    var ul = document.querySelector('ul');
    ul.style.top = this.state.pageY + 'px';
  },

  render: function() {
    var pageY = this.state.pageY;
    var begin = pageY / 31 | 0;
    // Add 2 so that the top and bottom of the page are filled with
    // next/prev item, not just whitespace if item not in full view
    var end = begin + (this.state.pageHeight / 31 | 0 + 2);

    var offset = pageY % 31;

    return dom.div(
      { className: 'list',
        style: 'position: relative; top: ' + (-offset) + 'px' },
      dom.ul(
        this.props.items.slice(begin, end).map(function(item) {
          return dom.li(null, item.title);
        })
      )
    );
  }
});

This assumes each list item has a height of 31px, and can calculate which set of items are visible given the page scroll position and window height. The render method only returns a fixed size of <li> elements always (probably ~25), no matter how large the list is. The <ul> is shifted as the user scrolls so it actually stays in one place, but the outer container is given the full height of the list so that we still have an accurate scrollbar.

Go ahead, right-click and inspect one of the list elements in your browser's developer tools. Look around and you'll only see a modest number of <li> elements.

Here's the code for initializing and rendering this component:

// application code

var items = [];
for(var i=0; i<5000; i++) {
  items.push({
    title: 'Foo Bar ' + i
  });
}

var app = App({ items: items });

window.addEventListener('scroll', function(e) {
  app.state.pageY = Math.max(e.pageY, 0);
  app.state.pageHeight = window.innerHeight;
});

// render

function render() {
  Bloop.renderComponent(app, document.body);
  requestAnimationFrame(render);
}

render();

When the scroll event is fired, we simply update pageY and pageHeight and the new DOM elements are filled with the right data, giving the illusion that the user is scrolling down a large list. This basic implementation isn't perfect, but it certainly could be with some better edge case handling.

This is all just with my stupid Bloop library, just imagine what you could do with React's optimizations.

Contrast this with what it would take to implement with Web Components. You would have to manually manage all of those DOM nodes yourself, and take special care to remove ones outside of the viewport, or even better reuse them and reposition them. Retained mode is a sucky way of doing UIs, and I think we'd all be better off if we switched to thinking in immediate mode.

Additional Abstractions

Cortex

We went over how data flows in Bloop and React: data is passed down and events are triggered up through event handlers. This is a good way for components to talk to each other, but it has drawbacks. Sometimes it's annoying to create many trivial event handlers, and you also want to be able to wrap state management into components instead of it all being top-level.

There are many ways to improve this, and React actually encourages its community to build interesting abstractions on top of React. Cortex is one such enhancement of handling state.

Cortex is a way to have one single data structure for app state, but have the ability to take pieces of it and hand it off to child components. Child components have the ability to change state themselves, and we get update notifications. It's basically a type of "observable", but the difference is we don't care what has changed. When we get an update notification, we just trigger a rerender of the whole app.

Here's what using Cortex looks like:

var data = { settings: { username: "James "},
             number: 0 };

var cortex = new Cortex(data, function() {
  // Called whenever an update happened
  Bloop.renderComponent(app, document.body);
});

In my component, if I had a Settings component I could pass it down like so:

var App = Bloop.createClass({
  render: function() {
    return dom.div(
      MainContent(),
      Settings({ state: this.state.settings })
    );
  }
});

And the Settings component could update it like so:

var Settings = Bloop.createClass({
  updateUsername: function(username) {
    this.state.username.set(username);
  },

  render: function() {
    // ...
  }
});

The callback we passed to Cortex would be called and the app would be rerendered. Now we can do more sophisticated state management, letting components manage the state themselves but still having full access to the state from the top-level through our normal data object. It's like proper Object-Oriented Programming!

Indeed, if you are familiar with functional lenses this sounds all too familiar to you. Unfortunately, this is still a mutable data structure, but it still solves the encapsulation problem.

The above code is Bloop-specific. In React, you would need to access the cortex object in props in components that get state passed down. Bloop allows you to specify a state property when creating the component, and it is used as the component's initial state.

Here is the full increment/decrement example using Cortex (or as a gist):

var dom = Bloop.dom;

// components

var App = Bloop.createClass({
  render: function() {
    return dom.div(
      dom.span(this.state.number.val()),
      Toolbar({
        number: this.state.number
      })
    );
  }
});

var Toolbar = Bloop.createClass({
  updateNumber: function(value) {
    this.props.number.set(value);
  },

  render: function() {
    return dom.div(
      dom.button({
        onClick: this.updateNumber.bind(null, this.props.number.val() - 1)
      }, 'decrement'),
      dom.button({
        onClick: this.updateNumber.bind(null, this.props.number.val() + 1)
      }, 'increment')
    );
  }
});

var data = { number: 0 };

var cortex = new Cortex(data, function() {
  // Called whenever an update happened
  Bloop.renderComponent(app, document.body);
});

// render

var app = App({ state: cortex });
Bloop.renderComponent(app, document.body);

Bacon.js

If functional reactive programming (FRP) is your thing, I'm sure you've heard of bacon.js. Since Bloop doesn't care where your data comes from, it's trivial to use FRP to construct data flows and update the UI whenever something comes down the stream. This post describes how to do just that with bacon.js.

Addon: Immutability Helper

React actually comes with an addon that lets you update data structures persistently, the immutability helper. The neat thing is that you can still use native JavaScript data structures, but create new objects when performing updates instead of mutating them directly.

It's a little unwieldy to use, however, but with some macro magic it could be quite handy.

Om

Om is a much more sophisticated abstraction on top of React. It is a ClojureScript interface to React that introduces a different way of defining components and managing state. Since ClojureScript uses persistent data structures natively, app state is immutable and persistent. This immutability makes it trivial to check what has changed, since you just have to compare pointers.

Om uses requestAnimationFrame to render the app continuously batch rendering (all requests for rerendering happen just once on the next animation frame). If something has changed, it's very quick to detect the components that have changed with a few more pointer checks and rerender those components.

Immutability turns out to be an incredible companion to React, making not only app state explicit but also changes over time. Keeping a history involves only keeping pointers to previous app states.

I would love to dive into this more, but that is for a future post.

Mori

Mori is a library for persistent data structures for JavaScript. These are same data structures used by ClojureScript. If you want a lot of the same benefits that Om takes advantage of, like optimized rendering and easy history and undo, you can use this library to manage app state as a persistent data structure.

I haven't seen too many persistent JavaScript react apps yet, but there is this post about using a different persistent data structure library for building apps.

Finale

We've shown how React chooses a level of abstraction that is powerful, and also adaptive. We haven't explored the details of React's optimizations with the virtual DOM, but you can find more about that in the docs and around the web. I wanted to focus on the abstract idea itself, and show how well it works even with my stupidly simple Bloop library.

Bloop follows most of React's APIs and conventions, with the following differences: in React, the properties object is required when creating DOM elements, so you have to pass null if there aren't any (dom.div(null, 'text')). There also is no componentDidRender in React, but there is componentDidUpdate.

This post assumes Web Components as a way to build applications; some people look at Web Components as a low-level way to share custom components. Even so, everything is stuck in a global namespace and you miss out on all the goodness of modules. Also, it's a hard sell when something like React can't even use it, and it's hard to load in components from a completely different system, especially when you want to take advantage of what you already have.

There's no doubt that you will need more than this for building apps: we haven't mentioned routing, data stores, controllers, and all that stuff. I like the ability to choose which libraries to use and see how they are all pieced together. See React's post about the flux architecture, a router, and more.

It's no coincidence that immutability and persistence was repeatedly referenced throughout this post. Using persistent data structures with this architecture really does allow for powerful features. However, even with simple mutable JavaScript objects, React brings a powerful UI and component system to the table.

Read more:

  • Bloop
  • Source code for all the demos
  • React docs
  • React provides JSX, an extension to JavaScript, which allows you to embed HTML directly in JavaScript. This makes it look like you are writing HTML, but all it does it transform to the native dom.* calls. I prefer not to use it, but it's helpful if you are working with designers.
  • Om is a ClojureScript interface to React
  • Mori is a library of persistent data structures for JavaScript
  • Cortex provides a way to centrally manage data
  • Mercury is an attempt to rebuild something like React, separating functionality into lots of modules like virtual-dom. It features immutable-by-default state and virtual DOM, see more comparison here.
  • Mithril is another framework that has similar ideas, and the author has commented and ported my examples here.

]]>
Moving to the Developer Tools Team 2014-03-18T00:00:00Z 2014-03-18T00:00:00Z http://jlongster.com/Moving-to-Developer-Tools I've been with Mozilla for over 3 years now, and it's been an amazing experience. I get to work with people smarter than me, have freedom to hack on all sorts of projects, and feel like I'm contributing to a greater good (I've worked on http://mozilla.org/ and a bunch of stuff related to web apps). That said, for a while I haven't been challenged and I've been looking for different team to utilize my skills.

As of this week, I'm moving to the Firefox Developer Tools team, specifically the scripting and performance team. I couldn't be more excited! I think it's a perfect fit for me: complex JavaScript engineering challenges and a product that helps millions of developers.

I will help with the next version of our tools for measuring performance of the web, from JavaScript to layout jank and all sorts of other components. I will also help improve our debugger and basically anything that I find time to help with.

We have a lot of work to do, and I'm really excited to be a part of it. Firefox devtools are going to rock even more than they already do. Feedback is crucial, so let me know what you love/hate about our developers tools, and I'll make sure my team knows about it. Keep rockin' the free web!

]]>
I've been with Mozilla for over 3 years now, and it's been an amazing experience. I get to work with people smarter than me, have freedom to hack on all sorts of projects, and feel like I'm contributing to a greater good (I've worked on http://mozilla.org/ and a bunch of stuff related to web apps). That said, for a while I haven't been challenged and I've been looking for different team to utilize my skills.

As of this week, I'm moving to the Firefox Developer Tools team, specifically the scripting and performance team. I couldn't be more excited! I think it's a perfect fit for me: complex JavaScript engineering challenges and a product that helps millions of developers.

I will help with the next version of our tools for measuring performance of the web, from JavaScript to layout jank and all sorts of other components. I will also help improve our debugger and basically anything that I find time to help with.

We have a lot of work to do, and I'm really excited to be a part of it. Firefox devtools are going to rock even more than they already do. Feedback is crucial, so let me know what you love/hate about our developers tools, and I'll make sure my team knows about it. Keep rockin' the free web!

]]>