Progress events

Progress events are becoming so common that they’re actually written up in a separate specification^[1]. These events are designed to generically indicate the progress of data transfers. Such transfers occur when requesting data from the server, but also when requesting data from disk, which is what FileReader does.

There are six progress events:

• loadstart – indicates that the process of loading data has begun. This event always fires first.
• progress – fires multiple times as data is being loaded, giving access to intermediate data.
• error – fires when loading has failed.
• abort – fires when data loading has been canceled by calling abort() (available on both XMLHttpRequest and FileReader).
• load – fires only when all data has been successfully read.
• loadend – fires when the object has finished transferring data. Always fires and will always fire after error, abort, or load.

Two events, error and load, were discussed in my previous post. The other events give you more fine-grained control over data transfers.

Tracking progress

When you want to track progress of a file reader, use the progress event. The event object for this event contains three properties to monitor the data being transferred:

• lengthComputable – a boolean indicating if the browser can determine the complete size of the data.
• loaded – the number of bytes that have been read already.
• total – the total number of bytes to be read.

The intent of this data is to allow for progress bars to be generated using the information from the progress event. For example, you may be using an HTML5 <progress> element to monitor the progress of reading a file. You can tie the progress value to the actual data using code like this:

var reader = new FileReader(),
     progressNode = document.getElementById("my-progress");

reader.onprogress = function(event) {
    if (event.lengthComputable) {
        progressNode.max = event.total;
        progressNode.value = event.loaded;
    }
};

reader.onloadend = function(event) {
    var contents = event.target.result,
        error    = event.target.error;

    if (error != null) {
        console.error("File could not be read! Code " + error.code);
    } else {
        progressNode.max = 1;
        progressNode.value = 1;
        console.log("Contents: " + contents);
    }
};

reader.readAsText(file);

This is similar to the approach that Gmail uses for its drag and drop file upload implementation, where you see a progressbar immediately after dropping a file onto the email. That progressbar indicates how much of the files has been transferred to the server.

Dealing with errors

Even though you’re reading a local file, it’s still possible for the read to fail. The File API specification^[2] defines four types of errors:

• NotFoundError – the file can’t be found.
• SecurityError – something about the file or the read is dangerous. The browser has some leeway as to when this occurs, but generally if the file is dangerous to load into the browser or the browser has been performing too many reads, you’ll see this error.
• NotReadableError – the file exists but can’t be read, most likely due to a permissions problem.
• EncodingError – primarily when trying to read as a data URI and the length of the resulting data URI is beyond the maximum length supported by the browser.

When an error occurs during a file read, the FileReader object’s error property is assigned to be an instance of one of the above mentioned errors. At least, that’s how the spec is written. In reality, browsers implement this as a FileError object that has a code property indicating the type of error that has occurred. Each error type is represented by a numeric constant value:

• FileError.NOT_FOUND_ERR for file not found errors.
• FileError.SECURITY_ERR for security errors.
• FileError.NOT_READABLE_ERR for not readable errors.
• FileError.ENCODING_ERR for encoding errors.
• FileError.ABORT_ERR when abort() is called while there is no read in progress.

You can test for the type of error either during the error event or during loadend:

var reader = new FileReader();

reader.onloadend = function(event) {
    var contents = event.target.result,
        error    = event.target.error;

    if (error != null) {
        switch (error.code) {
            case error.ENCODING_ERR:
                console.error("Encoding error!");
                break;

            case error.NOT_FOUND_ERR:
                console.error("File not found!");
                break;

            case error.NOT_READABLE_ERR:
                console.error("File could not be read!");
                break;

            case error.SECURITY_ERR:
                console.error("Security issue with file!");
                break;

            default:
                console.error("I have no idea what's wrong!");
        }
    } else {
        progressNode.max = 1;
        progressNode.value = 1;
        console.log("Contents: " + contents);
    }
};

reader.readAsText(file);

Up next

The FileReader object is a fully-featured object with a lot of functionality and a lot of similarities to XMLHttpRequest. By following these last three posts, you should now be able to read data from files using JavaScript and send that data back to the server if necessary. However, the File API ecosystem is quite a bit larger than has been already discussed in this series, and in the next part you’ll learn about a powerful new features designed to work with files.

References

1. Progress Events
2. File API

The FileReader type

The FileReader type has a single job: to read data from a file and store it in a JavaScript variable. The API is intentionally designed to be similar to XMLHttpRequest since both are loading data from an external (outside of the browser) resource. The read is done asynchronously so as not to block the browser.

There are several formats that a FileReader can create to represent the file data, and the format must be requested when asking the file to be read. Reading is done through calling one of these methods:

• readAsText() – returns the file contents as plain text
• readAsBinaryString() – returns the file contents as a string of encoded binary data (deprecated – use readAsArrayBuffer() instead)
• readAsArrayBuffer() – returns the file contents as an ArrayBuffer (good for binary data such as images)
• readAsDataURL() – returns the file contents as a data URL

Each of these methods initiates a file read similar to the XHR object’s send() method initiating an HTTP request. As such, you must listen for the load event before starting to read. The result of the read is always represented by event.target.result. For example:

var reader = new FileReader();
reader.onload = function(event) {
    var contents = event.target.result;
    console.log("File contents: " + contents);
};

reader.onerror = function(event) {
    console.error("File could not be read! Code " + event.target.error.code);
};

reader.readAsText(file);

This example simply reads the contents of a file and outputs it in plain text to the console. The onload handler is called when the file is successfully read whereas the onerror handler is called if the file wasn’t read for some reason. The FileReader instance is available inside of the event handler via event.target and it’s recommended to use that instead of referencing the reader variable directly. The result property contains the file contents on success and error contains error information about the failed operation.

Reading data URIs

You can use the same basic setup for reading to a data URI. Data URIs (sometimes called data URLs) are an interesting option if you want to, for example, display an image that was just read from disk. You could do so with the following code:

var reader = new FileReader();
reader.onload = function(event) {
    var dataUri = event.target.result,
        img     = document.createElement("img");

    img.src = dataUri;
    document.body.appendChild(img);
};

reader.onerror = function(event) {
    console.error("File could not be read! Code " + event.target.error.code);
};

reader.readAsDataURL(file);

This code simply inserts an image that was read from disk into a page. Since the data URI contains all of the image data, it can be passed directly into the src attribute of an image and displayed on the page. You could, alternately, load the image and draw it onto a <canvas> as well:

var reader = new FileReader();
reader.onload = function(event) {
    var dataUri = event.target.result,
        context = document.getElementById("mycanvas").getContext("2d"),
        img     = new Image();

    // wait until the image has been fully processed
    img.onload = function() {
        context.drawImage(img, 100, 100);
    };
    img.src = dataUri;
};

reader.onerror = function(event) {
    console.error("File could not be read! Code " + event.target.error.code);
};

reader.readAsDataURL(file);

This code loads the image data into a new Image object and then uses that to draw the image onto a canvas (specifying both the width and height as 100).

Data URIs are generally used for this purpose, but can be used on any type of the file. The most common use case for reading a file into a data URI is to display the file contents on a web page immediately.

Reading ArrayBuffers

The ArrayBuffer type^[1] was first introduced as part of WebGL. An ArrayBuffer represents a finite number of bytes that may be used to store numbers of any size. The way data is read from an ArrayBuffer is by using a specific view, such as Int8Array, which treats the underlying bytes as a collection of 8-bit signed integers or Float32Array, which treats the underlying bytes as a collection of 32-bit floating point numbers. These are called typed arrays^[2], which force you to work with a specific numeric type rather than containing any type of data (as with traditional arrays).

You use an ArrayBuffer primarily when dealing with binary files, to have more fine-grained control over the data. It’s beyond the scope of this post to explain all the ins and outs of ArrayBuffer, just realize that you can read a file into an ArrayBuffer pretty easily if you need it. You can pass an ArrayBuffer directly into an XHR object’s send() method to send the raw data to the server (you’ll have to read this data from the request on the server to reconstruct the file), so long as your browser fully supports XMLHttpRequest Level 2^[3] (most recent browsers, including Internet Explorer 10 and Opera 12).

Up next

Reading data from a file using a FileReader is pretty simple. If you know how to use XMLHttpRequest, there’s no reason you can’t also be reading data from files. In the next part of this series, you’ll learn more about using the FileReader events and understanding more about possible errors.

References

1. ArrayBuffer
2. Typed Array Specification
3. XMLHttpRequest Level 2

The File type

The File type is defined in the File API^[1] specification and is an abstract representation of a file. Each instance of File has several properties:

• name – the filename
• size – the size of the file in bytes
• type – the MIME type for the file

A File object basically gives you essential information about the file without providing direct access to the file contents. That’s important because reading from files requires disk access, and depending on the size of the file, that process has the potential to take a significant amount of time. A File object is just a reference to a file, and getting data from that file is a separate process altogether.

Getting File references

Of course, access to user files is strictly forbidden on the web because it’s a very obvious security issue. You wouldn’t want to load up a web page and then have it scan your hard drive and figure out what’s there. You need permission from the user in order to access files from their computer. There’s no need for messy permission windows, however, because users grant permission for web pages to read files all the time when they decide to upload something.

When you use a <input type="file"> control, you’re giving the web page (and the server) permission to access that file. So it makes sense that the first place you can retrieve File objects is through a <input type="file"> control.

HTML5 defines a files property for all <input type="file"> controls. This collection is a FileList, which is an array-like structure called FileList containing File objects for each selected file in the control (remember, HTML5 allows multiple file selection in these controls). So at any point in time, you can get access to the files a user has selected using code similar to this:

<input type="file" id="your-files" multiple>
<script>
var control = document.getElementById("your-files");
control.addEventListener("change", function(event) {

    // When the control has changed, there are new files

    var i = 0,
        files = control.files,
        len = files.length;

    for (; i < len; i++) {
        console.log("Filename: " + files[i].name);
        console.log("Type: " + files[i].type);
        console.log("Size: " + files[i].size + " bytes");
    }

}, false);
</script>

This relatively simple code listens for the change event on the file control. When the event fires, it signifies that the file selection has changed, and the code iterates through each File object and outputs its information. Keep in mind that the files property is always accessible from JavaScript, so you don’t have to wait for change to try to read it.

Drag and drop files

Accessing files from form controls still requires the form control and the associated user action of browsing to find the files of interest. Fortunately, HTML5 Drag and Drop^[2] provides another way for users to grant access to their files: by simply dragging a file from the desktop into the web browser. All you have to do to take advantage is listen for two events.

In order to read files that are dropped onto an area of the page, you must listen for the dragover and drop events and cancel the default action of both. Doing so tells the browser that you are handling the action directly and it shouldn’t, for example, open an image file.

<div id="your-files"></div>
<script>
var target = document.getElementById("your-files");

target.addEventListener("dragover", function(event) {
    event.preventDefault();
}, false);

target.addEventListener("drop", function(event) {

    // cancel default actions
    event.preventDefault();

    var i = 0,
        files = event.dataTransfer.files,
        len = files.length;

    for (; i < len; i++) {
        console.log("Filename: " + files[i].name);
        console.log("Type: " + files[i].type);
        console.log("Size: " + files[i].size + " bytes");
    }

}, false);
</script>

The event.dataTransfer.files is another FileList object that you can access to get file information. The code is almost exactly the same as using the file form control and the File objects can be accessed in the same way.

Ajax file upload

Once you have a reference to the file, you’re able to do something that’s pretty cool: upload a file via Ajax. This is all possible due to the FormData object, which is defined in XMLHttpRequest Level 2^[3]. This object represents an HTML form and allows you to add key-value pairs to be submitted to the server via the append() method:

var form = new FormData();
form.append("name", "Nicholas");

The great thing about the FormData object is that you can add a file directly to it, effectively mimicking a file upload by HTML form. All you have to do is add the File reference with a specific name, and the browser does the rest. For example:

// create a form with a couple of values
var form = new FormData();
form.append("name", "Nicholas");
form.append("photo", control.files[0]);

// send via XHR - look ma, no headers being set!
var xhr = new XMLHttpRequest();
xhr.onload = function() {
    console.log("Upload complete.");
};
xhr.open("post", "/entrypoint", true);
xhr.send(form);

Once the FormData object is passed into send(), the proper HTTP headers are automatically set for you. You don’t have to worry about setting the correct form encoding when using files, so the server gets to act as if a regular HTML form has been submitted, reading file data from the “photo” key and text data from the “name” key. This gives you the freedom to write processing code on the backend that can easily work with both traditional HTML forms and Ajax forms of this nature.

And all of this works in the most recent version of every browser, including Internet Explorer 10.

Up next

You now know the two methods of accessing File information in the browser: through a file upload control and through native drag and drop. There will likely be other ways to access files in the future, but for now, these are the two you need to know. Of course, reading information about files is just part of the problem. The next step is read data from those files, and that’s where part 2 will pick up.

References

1. File API specification (editor’s draft)
2. HTML5 Drag and Drop
3. XMLHttpRequest Level 2

I really and truly wish that The Linux Command Line had been available to me at that time. This is exactly what a Linux beginner needs to get up to speed quickly. The book goes beyond simply walking through all of the command line utilities, and ventures into the realm of theory and how things work together. I’m definitely not a Linux expert, but I have been using it on a day-to-day basis for about six years, and I still felt like I learned an incredible amount from this book.

The author approaches all of the topics in a very friendly way. All of the examples are easy to follow and he even takes time to explain some of the differences across various flavors of Linux. He covers all of the common command line tools, taking you through in a logical order with great narrative.

This book has earned a permanent place on my desk, as I find myself repeatedly going back to it when I get stuck on something. If you’re a Linux beginner, or just really want to understand the system better, I can’t recommend this book enough. I’ll probably be rereading it a couple more times, myself, to make sure that everything sticks. In the meantime, I’ve already gone through and updated a bunch of bash scripts that I now realize aren’t coded as well as they should be.

Wanting to get to the bottom of it, I^[2] and John Allsopp^[3] wrote blog posts trying to provide numbers around localStorage. John’s post focused on quantifying the amount of time it takes to perform a single read and a single write, which gave us good initial numbers for these operations. My post focused on comparing localStorage reads and writes to cookie reads and writes from JavaScript. My theory was that cookies are the closest appromixation of localStorage due to the fact that its contents are stored on disk and are shared by all tabs pointing to the same origin. Both John and I concluded by saying that localStorage doesn’t have an appreciably bad affect on performance either as an aggregate rating or in comparison to cookies.

More details

Subsequent to that, I started a conversation with Jonas Sicking from Mozilla, who actually worked on the localStorage implementation for Firefox and so has a unique perspective. He started from the position that there is a performance problem and I started from the position that there is not, based on the numbers from John and I. Jonas pointed out a key piece of information I wasn’t aware of: the performance issue isn’t with individual reads and writes, it’s with the initial read into memory.

Firefox starts out by reading all of the data from localStorage into memory for the page’s origin. Once the data is in memory, reads and writes should be relatively fast (though they do still appear slower than reading and writing to a native JavaScript object – not sure why), so our measuring of reads and writes doesn’t capture the full picture. Jonas’ assertion is that reading the data from localStorage on page load is the concern.

As Jonas kept telling me (and finally it stuck), the real problem with localStorage is that it’s a synchronous API, which makes the implementors decide between a limited number of options. One option is to load all the data as the page is loading, but that has a side effect of slowing down initial page load because JavaScript using localStorage can’t execute until the data for localStorage has been completely read. That means a large amount of data in localStorage could actually increase page load time because JavaScript needs to wait before executing.

The other option isn’t much better. If you were to wait until the first time localStorage was used, it would require a full (blocking) stop while the data was read from disk initially. Once again, this could be noticeable if there’s a large amount of data on disk. What’s more, you could argue that a delay on calling localStorage.getItem() is unexpected, because there is an assumption that you’re already working in memory and so the operation should be fast. This is why Firefox loads the data on page load.

In reality, this becomes the same problem as cookies. Cookies are stored on disk and read into memory upon page load as well. The difference is in the size of the data. Cookies are still fairly limited in size (around 4KB) where localStorage is much large (5MB). Of course, reading a 5MB file from the file system will be faster than downloading it over the internet, but who’s to say if it would significantly affect page load time?

Benchmarks?

I tried to run some benchmarks but was met with a technical limitation: no one is sure if our current testing tools are accurately taking the initial localStorage read into account. Without that information, it’s hard to know whether or not localStorage is actually a performance problem for initial page load. It definitely isn’t a performance issue for reads and writes after the fact (though it doesn’t come without some cost, as noted previously).

A new API?

The call to create a new API to replace localStorage seems a bit permature, but is basically centered around three main ideas:

1. The browser shouldn’t need to read a large amount of data from disk on page load.
2. The read from disk should be asynchronous and not block the UI thread.
3. The developer should be able to indicate when the read should happen.

This led Jonas to suggesting several alternatives APIs on Chris’ original post. The one I like the best is this:

getBetterLocalStorage(function(storage) {
    x = storage.foo;
    storage.bar = calculateStuff(y);
    storage.baz++;
});

Ignoring the name, the getBetterLocalStorage() function signals the browser that it’s time to read everything into memory, so the storage object can be used as any other object. Once the callback function is finished executing, the changes would be written back to disk. Though I’m not ready to throw out localStorage completely, I do like the direction of this API. In fact, it closely follows a proposal I made for improving localStorage with expiration dates and encryption.^[4]

Conclusion

Whether or not localStorage is a performance issue on page load is still a question. It’s hard for to know for sure if this is a real issue until we can get some good benchmarks from browsers. Unfortunately, this will likely have to come from browser developers who can look at the code and figure out whether localStorage is already being accounted for, and if not, how to measure it.

In the meantime, IndexedDB is definitely not a suitable replacement for localStorage in almost every case. IndexedDB could be used, as Jonas pointed out, to create a solution similar to the one he proposed. However, it’s still a bit of overhead to write that out. My advice: don’t worry too much about localStorage for now…but don’t go storing 5MB of data in it either, just in case.

References

1. There is no simple solution for localStorage by Chris Heilmann
2. In defense of localStorage by Me
3. localStorage, perhaps not so harmful by John Allsopp
4. Towards more secure client-side data storage by Me

Unfortunately, it’s a bit of a beast to install on Windows. Every time I have to install Ant on another Windows machine I end up searching the web yet again for a good set of instructions. So this post is primarily for myself, so that I don’t need to search too far.

Prerequisites

Before beginning, make sure you have the latest JDK installed. If not, go download it from Sun^[2] and install it. It’s better to install the JDK instead of just the JRE because some Ant tasks require the JDK.

Step 1: Download and install

The first step, as with most software, is to download Ant. Go to the Ant homepage and click to download the binary. Because we’re talking about Windows, choose to download the ZIP file rather than any of the others. Scroll down to where it says “Current release of Ant” and click on the ZIP filename.

Once downloaded, unzip the file. You’ll now need to choose a permanent home for Ant on the computer. I tend to use c:\java\ant for simplicity, but you can use whatever you want. I do recommend, however, that the path have no spaces in it (spaces make things more complicated).

Step 2: Set environment variables

This is the part that I always forget. Because you’re installing Ant by hand, you also need to deal with setting environment variables by hand.

For Windows XP: To set environment variables on Windows XP, right click on My Computer and select Properties. Then go to the Advanced tab and click the Environment Variables button at the bottom.

For Windows 7: To set environment variables on Windows 7, right click on Computer and select Properties. Click on Advanced System Settings and click the Environment Variables button at the bottom.

The dialog for both Windows XP and Windows 7 is the same. Make sure you’re only working on system variables and not user variables.

The only environment variable that you absolutely need is JAVA_HOME, which tells Ant the location of your JRE. If you’ve installed the JDK, this is likely c:\Program Files\Java\jdk1.x.x\jre on Windows XP and c:\Program Files(x86)\Java\jdk1.x.x\jre on Windows 7. You’ll note that both have spaces in their paths, which causes a problem. You need to use the mangled name^[3] instead of the complete name. So for Windows XP, use C:\Progra~1\Java\jdk1.x.x\jre and for Windows 7, use C:\Progra~2\Java\jdk1.6.0_26\jre if it’s installed in the Program Files(x86) folder (otherwise use the same as Windows XP).

That alone is enough to get Ant to work, but for convenience, it’s a good idea to add the Ant binary path to the PATH variable. This variable is a semicolon-delimited list of directories to search for executables. To be able to run ant in any directory, Windows needs to know both the location for the ant binary and for the java binary. You’ll need to add both of these to the end of the PATH variable. For Windows XP, you’ll likely add something like this:

;c:\java\ant\bin;C:\Progra~1\Java\jdk1.x.x\jre\bin

For Windows 7, it will look something like this:

;c:\java\ant\bin;C:\Progra~2\Java\jdk1.x.x\jre\bin

Done

Once you’ve done that and applied the changes, you’ll need to open a new command prompt to see if the variables are set properly. You should be able to simply run ant and see something like this:

Buildfile: build.xml does not exist!
Build failed

That means Ant is installed properly and is looking for a build.xml file.

References

1. Apache Ant homepage
2. Java SE Downloads
3. Microsoft filenames

What does it do?

Strict mode makes a lot of changes to how JavaScript runs, and I group these into two categories: obvious and subtle. The subtle changes aim to fix subtle problems, and I’m not going to delve into those here; if you’re interested in those details, please see Dmitry Soshnikov’s excellent, ECMA-262-5 in Detail. Chapter 2. Strict Mode^[1]. I’m far more interested in talking about the obvious changes: the ones you should know about before using strict mode, and the ones that will most likely help you the most.

Before getting into specific features, keep in mind that one of the goals of strict mode is to allow for faster debugging of issues. The best way to help developers debug is to throw errors when certain patterns occur, rather than silently failing or behaving strangely (which JavaScript does today outside of strict mode). Strict mode code throws far more errors, and that’s a good thing, because it quickly calls to attention things that should be fixed immediately.

Eliminates with

To begin, strict mode eliminates the with statement. It is now considered invalid JavaScript syntax and will throw a syntax error when it appears in strict mode code. So first step to using strict mode: make sure you’re not using with.

// Causes a syntax error in strict mode
with (location) {
    alert(href);
}

Prevents accidental globals

Next, variables must be declared before they can be assigned to. Without strict mode, assigning a value to an undeclared variable automatically creates a global variable with that name. This is one of the most common errors in JavaScript. In strict mode, attempting to do so throws an error.

// Throws an error in strict mode
(function() {

    someUndeclaredVar = "foo";

}());

Eliminates this coercion

Another important change is a this-value of null or undefined is no longer coerced to the global. Instead, this remains its original value, and so may cause some code depending on the coercion to break. For example:

window.color = "red";
function sayColor() {
    alert(this.color);
}

// Throws an error in strict mode, "red" otherwise
sayColor();

// Throws an error in strict mode, "red" otherwise
sayColor.call(null);

Basically, the this-value must be assigned a value or else it remains undefined. That means constructors accidentally called without new are also affected:

function Person(name) {
    this.name = name;
}

// Error in strict mode
var me = Person("Nicholas");

In this code, this is undefined when the Person constructor is called without new. Since you can’t assign a property to undefined, this code throws an error. In non-strict mode, this would be coerced to the global and so name would be assigned as a global variable.

No duplicates

It can be quite easy to duplicate properties in objects or named arguments in functions if you’ve been doing a lot of coding. Strict mode throws an error when it comes across either pattern:

// Error in strict mode - duplicate arguments
function doSomething(value1, value2, value1) {
    //code
}

// Error in strict mode - duplicate properties
var object = {
    foo: "bar",
    foo: "baz"
};

These are both syntax errors and so the error is thrown before the code is executed.

Safer eval()

Even though eval() wasn’t removed, it has undergone some changes in strict mode. The biggest change is that variables and functions declared inside of an eval() statement are no longer created in the containing scope. For example:

(function() {

    eval("var x = 10;");

    // Non-strict mode, alerts 10
    // Strict mode, throws an error because x is undeclared
    alert(x);

}());

Any variables or functions created inside of eval() stay inside of eval(). You can, however, return a value from eval() if you wish to pass a value back out:

(function() {

    var result = eval("var x = 10, y = 20; x + y");

    // Works in strict and non-strict mode (30)
    alert(result);

}());

Errors for immutables

ECMAScript 5 also introduced the ability to modify property attributes, such as setting a property as read only or freezing an entire object’s structure. In non-strict mode, attempting to modify an immutable property fails silently. You’ve probably run into this issue with some native APIs. Strict mode ensures that an error is thrown whenever you try to modify an object or object property in a way that isn’t allowed.

var person = {};
Object.defineProperty(person, "name", {
    writable: false,
    value: "Nicholas"
});

// Fails silently in non-strict mode, throws error in strict mode
person.name = "John";

In this example, the name property is set to read only. In non-strict mode, assigning to name fails silently; in strict mode, an error is thrown.

Note: I very strongly encourage you to use strict mode if you’re using any of the ECMAScript attribute capabilities. If you’re changing the mutability of objects, you’ll run into a lot of errors that will fail silently in non-strict mode.

How do you use it?

Strict mode is very easily enabled in modern browsers using the following pragma:

"use strict";

Even though this looks like a string that isn’t assigned to a variable, it actually instructs conforming JavaScript engines to switch into strict mode (browsers that don’t support strict mode simply read this as an unassigned string and continue to work as usual). You can use it either globally or within a function. That being said, you should never use it globally. Using the pragma globally means that any code within the same file also runs in strict mode.

// Don't do this
"use strict";

function doSomething() {
    // this runs in strict mode
}

function doSomethingElse() {
    // so does this
}

This may not seem like a big deal, however, it can cause big problems in our world of aggressive script concatenation. All it takes is one script to include the pragma globally for every script its concatenated with to be switch into strict mode (potentially revealing errors you never would have anticipated).

For that reason, it’s best to only use strict mode inside of functions, such as:

function doSomething() {
    "use strict";
    // this runs in strict mode
}

function doSomethingElse() {
    // this doesn't run in strict mode
}

If you want strict mode to apply to more than one function, use an immediately-invoked function expression (IIFE):

(function() {

    "use strict";

    function doSomething() {
        // this runs in strict mode
    }

    function doSomethingElse() {
        // so does this
    }
}());

Conclusion

I strongly recommend everyone start using strict mode now. There are enough browsers supporting it that strict mode will legitimately help save you from errors you didn’t even know where in your code. Make sure you don’t include the pragma globally, but use IIFEs as frequently as you like to apply strict mode to as much code as possible. Initially, there will be errors you’ve never encountered before – this is normal. Make sure you do a fair amount of testing after switching to strict mode to make sure you’ve caught everything. Definitely don’t just throw "use strict" in your code and assume there are no errors. The bottom line is that it’s time to start using this incredibly useful language feature to write better code.

Update (14-Mar-2012): Added note about using strict mode pragma with non-conforming JavaScript engines.
Update (21-Mar-2012): Fixed typo.

References

1. ECMA-262-5 in Detail. Chapter 2. Strict Mode by Dmitry Soshnikov

What is slow?

So does localStorage have a performance problem? Quite simply, I don’t know. Is storing and retrieving data from localStorage slower than that of a regular, in-memory object? Yes. I wrote a post about this for 2011 Performance advent calendar^[2]. In fact, it’s quite a bit slower when reading data out. My conclusion was that you should try to limit reads by storing multiple pieces of data in the same key. But as with most performance metrics, this really only matters when you’re performing the same operation multiple times in a row. If you’re only ever reading one value or writing one value, you’ll likely never run into a performance issue regardless of the data size or what’s going on with your system.

So localStorage is slower than using an in-memory object. Cars are slower than airplanes. What does that tell us? Not a whole lot.

Pain points

The fact of the matter is that localStorage reads from and writes to disk, which is always slower than an in-memory operation because there’s hardware involved. That’s the first problem. The second problem is the per-origin nature of localStorage. This characteristic means that two browser windows or tabs open to the same origin can both be reading from or writing to the same localStorage at the same time. That, in turn, means the browser needs to be incredibly smart about how it performs each operation. If tab A is writing to localStorage around the same time that tab B is reading from localStorage, which operation should happen first?

Each operation, read and write, then needs to lock localStorage to ensure data integrity. This is actually a big issue. Interacting with files is also dicey since another system process might also be working the same file. Consider a simple write operation:

localStorage.setItem("foo", "bar");

This operation is synchronous, meaning that the UI thread is blocked in the browser until the write is complete. No further JavaScript will execute and no further UI updates drawn. During this one method call, several things happen:

1. If localStorage is busy with another window or tab, then wait. This is problematic since there’s no way to know how long this will be.
2. If the file is busy, then wait. The files may be scanned by antivirus, be included in a backup job, etc., and may therefore be unavailable until such operations complete. This is also problematic because it’s unpredictable.
3. Open the file for writing.
4. Seek the right spot to write to the file.
5. Write to the file.
6. Close the file.

Only after all of that completes can the browser continue on to execute other statements. So really, the issue isn’t that localStorage is slow, it’s that localStorage must necessarily block on each operation to maintain data integrity.

Compared to cookies

The closest comparable API for JavaScript is the cookie API (though calling document.cookie an API is incredibly generous). Cookies are also name-value pairs, albeit with some additional metadata, which uses files as storage and must be synchronized across browser windows and tabs. I was surprised that Chris didn’t compare localStorage to cookies since the API was clearly meant to move us from storing client-only data in cookies to storing it in localStorage. It’s no accident that the localStorage API looks a lot like various cookie APIs.

When I created a benchmark^[3] to test localStorage against cookies for reading and writing, the results were quite interesting. Internet Explorer, Chrome, and Safari (including iOS), reading cookies was slower than reading from  localStorage and writing to cookies was much slower than writing to localStorage. Firefox and Opera exhibit the same performance characteristics on writes as the others (with cookies being slower), but reading from a cookie is faster. So in many cases across browsers, localStorage is actually a performance improvement over using cookies with JavaScript.

APIs

The reason localStorage is popular is partly due to its simplicity. That simplicity is by design and was first designed and implemented by browser vendors, which is why it seems strange that a browser vendor would now lead the charge against an API it had a hand in creating. Yes, humans create browsers and humans can make mistakes, but I don’t think the design of localStorage is a mistake.

As I was reading over Chris’ plea to look for alternatives, my engineer brain kept repeating to myself, “this is an implementation issue, not an interface issue”. Firefox is choosing to preload the localStorage data to improve read performance later, but that’s an implementation issue. Likewise, the need to read and write synchronously is an implementation issue – many forget that Internet Explorer 8′s implementation of localStorage actually wrote asynchronously. That an implementation-specific detail. Why not make all writes happen asynchronously and just keep a copy of the data in memory so it can always be read correctly regardless of the write state?

I’m not saying that this is necessarily an easy problem to solve; what I am saying is that the API design works well for developers, and for that reason it’s worth looking at the implementation details to figure out if there’s an efficient way to hide the warts of the system from web developers.

The proposed alternative, IndexedDB, is perhaps one of the worst API designs I’ve ever seen. To read and write a single piece of data requires way too many lines of code, ensuring that the API won’t be used by most developers until someone comes up with a library to abstract away the horribleness. I understand the rationale behind providing such a low-level, asynchronous API (I was even part of the discussions held at Mozilla around web databases), but this absolutely stinks of browser developers creating an API that’s easy to implement rather than creating an API that’s easy to consume. This is the opposite of how good APIs are made. IndexedDB will never be a replacement for localStorage, it’s just too complicated for most uses.

Non-blocking localStorage

As discussed previously, the real issue is that localStorage blocks on reads and writes, and the amount of time it blocks can’t be determined ahead of time. If this turns out to be a concern for you (after benchmarking, of course), then the solution is to use a non-blocking  localStorage mechanism. When you hear the term “non-blocking” these days, you should immediately be thinking about Web Workers.

In the near future, I believe that client-side architectures that perform I/O should perform all of that I/O in Web Workers. That means all of your localStorage, XMLHttpRequest, Web Socket, etc., I/O should be done inside of a worker. Basically, you should be able to do something like this:

var worker = new Worker("io.js"); 

worker.postMessage({
    type: "write",
    src: "localStorage",
    key: "foo",
    value: "bar"
}); 

worker.postMessage({
    type: "read",
    src: "localStorage",
    key: "foo"
}); 

worker.onmessage = function(event) {
    alert(event.data.value);
};

All of the reading and writing would be done off of the UI thread, so blocking really doesn’t matter. I know I’m not alone in thinking this is the way of the future as the IndexedDB spec has a whole section on synchronous APIs available in workers^[5]. Having synchronous APIs for IndexedDB makes it less horrible to deal with, but you need to use them in a worker. This hasn’t been implemented by all browsers yet, but should be coming soon. Add to that the concept of shared workers, web workers that are shared amongst all tabs with pages from the same origin, and you have a great recipe for resolving a lot of I/O issues.

Workers currently have access to XMLHttpRequest, Web Sockets, File Readers, and the such…and yet no access to localStorage. Why? This is really the solution to the problem: don’t throw away a great API because in some cases it will cause issues. Instead, make it available in workers so that we have an option for moving the reading/writing off of the UI thread.

Note: It’s possible that the cross-domain localStorage approach I wrote about previously^[6] might provide some non-blocking benefits. The cross-frame postMessage() API is asynchronous, but I’ve not figured out a good way to test if the containing page freezes if an iframe from the same domain accesses localStorage .

Conclusion

Asking web developers to give up localStorage is ridiculous. Are there problems with the API? Yes, indeed there are. Are they bad enough to abandon using it altogether? Absolutely not. Claims of terrible performance haven’t been substantiated. Despite the complaints of browser developers as to the technical difficulties, there are no good alternatives to localStorage. We could always go back to using cookies, but as the previous benchmark shows, that doesn’t necessarily guarantee better performance. And IndexedDB is a non-starter because the API is too complex for most use cases.

So to Mozilla, and the other browser vendors out there, you’re a victim of your own success. You wanted to create an API that could be used in place of cookies for storing client-side data, and you created something great. The interface is friendly to web developers and that’s why it’s had such rapid adoption. You’re all quite smart, capable people and I’m sure you can come up with better ways to implement the API than what we have today. And also, make localStorage accessible in Web Workers, please.

Update (8 March 2012): Fixed typos and added shared worker reference.

References

1. There is no simple solution for localStorage by Chris Heilmann
2. localStorage Read Performance by Nicholas C. Zakas
3. localStorage vs. cookies by Nicholas C. Zakas
4. Introduction to Web Storage by MSDN
5. Indexed Database – Synchronous APIs
6. Learning from XAuth: Cross-Domain localStorage by Nicholas C. Zakas

Technical debt, or tech debt for short, is quite simply the amount of technical work remaining on a project. It’s something that is hard to quantify, typically existing in one or more developer minds at any given point in time. There’s the initial debt of hours or days to completion which is usually the estimate delivered to the manager. After that, every technical decision that’s made comes with some sort of tech debt.

Deadlines rarely align with the amount of time needed to complete work, and so technical compromises are made along the way. There’s a short-term tradeoff to get something working initially. The developer knows the lifetime of the solution and accepts that he or she will have to go back and create the longer-term solution later on. While those looking in from the outside might consider this to be sloppy engineering, it’s actually what allows products to be released on time. And also how tech debt accumulates.

Tech debt doesn’t show up on any product roadmap and so developers frequently try to address it as they’re working on other features. In some organizations, a savvy manager may be able to buy the developers some time pay down their tech debt, but that is the exception more than the rule. Those making business decisions typically don’t like giving time for paying tech debt because, in their mind, the work isn’t resulting in any new features. Why should they give you time to work on something when the end result isn’t a new feature?

In the worst case, tech debt can accumulate to the point where the web application is destined to fail. A classic tech debt issue is scalability. Initially, the goal is to get the web application up and running so that anyone can use it. Decisions are made to allow that to happen. But an app that can handle 100 users isn’t built the same way as an app to handle 1,000,000 users. In the back of the developers’ minds are a list of things that need to happen in order to scale out: hit the database less, implementing several layers of caching, reduce the size of responses, figure out a faster way to process orders, and so on.

Just like monetary debt, tech debt is best dealt with before it gets too large and overwhelming. Regularly tending to tech debt is a process I like to call code hygiene. If you don’t go to the dentist for ten years and then finally go, chances are you’ll be in for a nasty surprise by not practicing proper dental hygiene. Code hygiene is the same. Keeping on top of your tech debt means regularly going in and addressing what you can with the time available. Good code hygiene means regularly:

1. Deleting unused code – I always say that any day you can delete multiple lines of code is a good day, and one where you earned your paycheck. Unused code shouldn’t be allowed to live in a codebase because, just like bacteria, all it needs is one opening to cause a big problem. If you suspect code isn’t being used, verify and then remove it. Don’t make a note to do it later, thus creating more tech debt, do it immediately. Less code means less to go wrong and faster builds and deployments.
2. Refactor code – The time to refactor is as soon as you notice a problem. Don’t allow sloppy or bad code to remain in the code base. Code has a way of duplicating itself without warning. New developers come in looking for an example, find bad code, copy it, and now you have two pieces of bad code instead of one. That’s more tech debt. Make sure that all code you check in is worthy of being copied to prevent future tech debt. When you’re building off of existing code, make sure to include some time in your estimates for refactoring.
3. Rewrite code – It’s rare for developers to be able to rewrite something from scratch, but it’s a necessary part of code hygiene. Good code should be able to run for five years without being rewritten; bad code needs to be rewritten as soon as possible. You should always try to refactor first, but if that fails, it’s time to rewrite. Making the business case for rewriting is difficult, but if you can tie the rewrite to a new feature, you have a better chance of getting the time.

Technical debt is a part of every project, and one that needs to be better understood by all parties involved. Building time into every release for code hygiene is a good idea and can take many forms: bug bashes, where everyone takes a day or two and just works on bugs, engineering-driven features, which are features of the platform rather than end-user features, and code reviews, where engineers inspect each other’s code, are all good practices to have on the team to ensure tech debt doesn’t get out of control.