Mad, Beautiful Ideas

Changes

2011-07-13T13:00:00Z

The last few weeks have been very long, but still very, very good. Almost exactly two months ago, I found myself with the opportunity to leave my employer of nearly four years, Washington State University. WSU has proven to be an excellent incubator for me over these last few years, but I know full well I had been ready to go for months, even prior to beginning my job search in earnest. At some point, I had begun to feel that the University had become an impediment to my further professional growth, and I increasingly found myself strongly disagreeing with the direction of the higher leadership at the institution, whom it seemed was continually making decisions that I felt were neither sustainable, nor fiscally responsible for a state-run institution.

While I was ready to move on, the decision to seek a new job was also strongly driven by a new opportunity my wife had created for herself. Due to an unfortunate situation with her advisor for her graduate program, she decided to stop pursuing a Ph.D. at WSU, and instead complete her Master’s degree in Zoology and complete her Ph.D elsewhere. Earlier this year, she was given an excellent opportunity at the University of Louisiana at Lafayette, working under Dr. Darryl Felder, a researcher focusing on decapod crustaceans. It is an amazing opportunity, that will have us moving, six days after this post, to the heart of cajun country.

As for me, I have spent the past six weeks or so working for Meebo. Meebo has been finding itself in a period of really aggressive growth over the first half of this year, with a half dozen people joining on the front-end JavaScript team alone this year, including myself. It’s been an exciting place to be, and though Meebo’s engineering is based in Mountain View, California and New York City, I have been lucky enough to be brought into the company as one of the first full-time remote engineers.

Working remote is definitely a change, though my experience in the open source community over the last decade or so, and especially on the YUI project over the last three or so years, had taught me a lot about working with people you communicate with primarily via e-mail and chat. Still, it has been an adjustment, as my desk is ten feet from my bed, and as my fellow YUI contributor and recent Meebody, Tony Pipkin (@apipkin) recently tweeted:

New office attire: basketball shorts and a plain white t

At Meebo, I have transitioned to being a pure JavaScript programmer. When I need a server-side component, I pass those tasks off to someone else, which is a bit awkward. I have to e a lot more proactive about making sure that my server-side counterpart is aware of my requirements early enough that they can be scheduled, since I’m not in Mountain View, I need to communicate really clearly and with written specifications, because miscommunication can result in the wrong thing being implemented because of ambiguous language.

I’ve been assigned to the Ads product at Meebo, which means that any where you go with the Meebo Bar, when the ad pops up, that’s code I now own running. Advertising is a nuanced business, but I have long been convinced that the best model we have at scale for monetizing content is ad sales (it doesn’t scale down to, say, the size of this blog, however), thought there is an incredibly amount of nuance to that business that I had no idea existed. Comments for another post, however.

In six days, Catherine and I will watch as everything we own gets loaded onto a truck, before we follow that truck out of town for a drive across the country with our two cats. The kind of change that we’re looking at has grown to be incredibly intimidating, even though it’s exciting. Starting work on the 25th, right after we get down there (and incidentally, possibly a week before our possessions arrive. A 6-14 day delivery window is really inconvenient).

I’ll be looking to get involved in a developer group down in Lafayette, and I’m looking forward to getting familiar with the area. And I definitely plan to start blogging regularly again come August.

Confessions of a Public Speaker

2011-05-11T13:00:00Z

Scott Berkun¹ is a former Microsoft Engineer who severed ties with the mothership and went full-time public speaker some years back. His book, Confessions of a Public Speaker² is sort of his manifesto on how he felt comfortable making that change and how he feels he’s found success.

Scott acknowledges the simple fact that he feels comfortable giving away these secrets and hard earned knowledge simply because he knows that most people will never do the necessary work to become a truly great speaker. I know that I’m not apt to immediately. I feel that presentation is important, and I often present at regional Code Camp events and for WSU’s Application Developer’s group, something I began doing in part because I was tired of going to events and sitting through talks that I felt had little value either due to poor presentation, or just because I felt I knew more than the presenter.

More than that, I think that it’s important to share information within our community. Between techniques and tools, we are certainly spoiled for choice, but without discussion and presentation, the majority of people developing software today have no chance to get expose to any idea that isn’t backed by a marketing budget (this is a notable problem in the .NET and Java communities, but that’s another post).

If you’ve done anything with public speaking in the past (and odds are you’ve taken a class at some time that did something with public speaking), you’ve no doubt heard much of this advice before. Practice, prepare more content than you think you need (but be prepared to cut content on the fly if necessary), practice, learn to harness your nervous energy, practice, show up early, etc. However, Berkun goes into a depth on this material that I’m not sure I’ve ever seen before.

He debunks common myths, like “people would rather die than speak in public,” by showing where such myths came from and the inherent ridiculousness in such statements. He presents many cases from his own career of things not going well at all, like giving a presentation at the same time and across the hall from Linus Torvalds’, who’s crowd was overflowing into the hall, while he had less than a dozen sitting in his enormous conference hall.

While Berkun does stress that preparation is the key to making any public speaking gig succeed, it’s the flexibility to deal with surprises that makes the best speakers as good as they are. Quick thinking doesn’t trump preparation, but it’s necessary sometimes to avert disaster.

One bit of advice that I’d like to share is what to do in that circumstance where no one shows up to hear you speak. Berkun suggests getting the crowd to move and sit near one another so that you can at least pretend the space is smaller than it truly is, while also making it easier to engage directly with the audience, perhaps turning it more into an informal directed conversation than a full-blown presentation with slides.

It is clear that Berkun was in technology, and still primarily speaks about tech, but despite his background, and the examples that he uses from his own career that refer to that, this is absolutely a public speaking book, and I think it’s accessible to anyone who wants to improve their public speaking, even if you’re not interested in turning it into a career.

References: 1. http://www.scottberkun.com/ 2. http://oreilly.com/catalog/9780596802004

Phyrric Victories are still Victories

2011-05-04T16:54:08Z

This past weekend, President Barack Obama was able to announce¹ to the world that, ten and a half years after taking responsibility for the single worst day of attacks on American soil in our history, Osama Bin Laden had been killed.

I am not going to discuss the morality of killing Bin Laden. We are a nation which practices the death penalty, and whether the operation which finally caught up with Bin Laden would capture or kill him, he was going to die. At least this way, we will not be subjected to a mockery of a trial, as the world was given following the capture of Saddam Hussein².

Some have made much of the fact that Bin Laden had been at the compound in Northern Pakistan which was assaulted on Saturday for some time, certainly long enough that JSOC³, the special forces unit directly answerable to the President which was designed around this sort of mission, had time to build and train in a replica of the compound for one full month. But I think that it’s a simple answer to why, in the end, this turned out to be relatively easy.

Osama Bin Laden was a great many things, but I do not believe he could ever have been categorized a fool. He choose to attack in the manner in which he did in 2001 because he knew that in no way could Al-Qaeda stand up against the American Armed Forces. By taking principle responsibility for the attack, he put himself in our line of fire, he began living on borrowed time, and prepared himself for martyrdom.

For a time, there as clearly value in remaining alive to release statements and further antagonize the West, but in order to be a Martyr, it would eventually become necessary to die.

This is why I believe he had lived so long in North Pakistan with a relatively small retinue of defenders. When dealing with a strike team like JSOC, five militants would stand about as well as twenty, but it would be easier to live, and live in relative comfort with fewer. No doubt there were other activities in progress, training new leuitenants for instance. Plus, a martyr does not walk into death, they must wait for it to find them.

However, even if you don’t believe, as I have come to, that this death was, in some way, prepared for by Bin Laden, there is no true victory in his death for us as a nation.

Ten years ago, we as a nation had certain harsh realities thrust upon us. It was demonstrated that we were not as insulated as we’d believed from the realities of global politics, or the terrible truth of the distrust and resentment created by our government’s historic policy of convenient involvement in other nations’ affairs. The world did not change that day, but American’s view of our place in it certainly did.

And what do we have to show for it today? The death of an enemy for whom 66% of American’s age 13-17 didn’t even know who they were (which, if anything, shows just how irrelevant Bin Laden had become). The legacy of the PATRIOT Act. The formation of the TSA. Erosion of our civil rights. Three Middle Eastern wars, two of which were justified as linked to those September 11, 2001 attacks.

I do not mean to downplay the actions of JSOC and the SEAL team which is responsible for Operation Geronimo. It was a well executed military action, particularly that we only suffered the loss of a single helicopter to mechanical failure. They executed their assigned mission, by all accounts, professionally and expertly.

I am not even terribly concerned with the lack of respect for the deceased shown by the decision to dump the body into the sea, while attempts were being made to follow all other tenets of Islam surrounding the handling of the dead. The concern that Bin Laden’s grave would become a place of pilgrimage for extremists was an understandable one, though the claim they couldn’t find anyone to take the body is absurd, given the size of Osama Bin Laden’s family.

I do not wholly misunderstand the jubilation felt by many at the news, particularly in the city of New York. I myself was in New York City, standing on the roof of the World Trade Center in July of 2011. When those towers fell I was awash in a surreal feeling over that experience. But I didn’t know anyone who lost their lives in the attack. I haven’t watched the growing health problems of the first responders. And I certainly haven’t lived with a daily reminder of the tragedy in the form of a damaged skyline and gaping hole at ground zero.

No, I do not begrudge the celebration, especially in New York.

And while this will no doubt drive many in Al-Qaeda into a new level of fervor, improved communication and analysis of intelligence, disruption of what command structure Al-Qaeda has, and a new level of proactiveness and willingness to respond among Americans has greatly reduced their chances at success. Combined with the real and reasonble security improvements, it is untrue to say we have nothing to fear, but the risk is likely better mitigated today than at any time in our past.

I just can’t help but think that, until we as a nation decide that we will not trade essential liberty for the illusion of security (and much of it has been illusion), than that enemy has still won. Terrorism is not defeated by killing terrorists. It is defeated by refusing to be terrorized.

Notes: 1. http://www.whitehouse.gov/blog/2011/05/02/osama-bin-laden-dead 2. http://en.wikipedia.org/wiki/TrialofSaddam_Hussein 3. http://en.wikipedia.org/wiki/JSOC

Thoughts on Void Safety in JavaScript

2011-04-29T01:46:07Z

A thread recently arose on the es-discuss mailing list¹ regarding the idea of adding an ‘existential operator’² to JavaScript. This was in line with some thinking I’d been doing lately, but which I was uanble to fully formulate to bring into the discussion on that thread, and now that the thread has been dead for a while, I’m choosing to use this forum to put down my thoughts before I decide to either start a new thread or raise the old one.

The argument for an ‘existential’ operator is an interesting one, with the initial poster, Dmitry Soshnikov proposing the following:

street = user.address?.street

// equivalent to:

street = (typeof user.address != "undefined" && user.address != null)
     ? user.address.street
     : undefined;

An interesting proposal, and functionally similar to my considerations to proposing void-safety into the language. Let’s first define what I mean by, ‘void-safety’, a term I first read during the interview with the creator of Eiffel³ in Masterminds of Programming⁴. To be void-safe, a simple rule would be added to the language that any attempt to access a property on an undefined value would itself equal undefined. In other words, it would be like the above example, but it would not have the ‘?’ character, and it would apply to all accesses of any value.

I oppose Dmitry’s suggestion to address this issue through syntax, as I think that such an addition would have a lot more value being a fundemental change to the language, than an optional change requiring a potentially obscure bit of syntax. Plus, this proposal is derived directly from Coffeescript⁵, which is a cool language, but was designed to translate directly into JavaScript, meaning that it’s solutions need to work with JavaScripts limitations in how it solves problems.

Either of these solutions helps to break a common pattern of data access, especially prevalent with configuration objects. However, there is at least one question that I’ve raised in my own head that has led to a bit of reluctance to post this to es-discuss. Imagine the following, fairly common pattern:

function action(config) {
    config || config = {};
    config.option || config.option = "default";
    ... Do Something ...
};

With void-safe JavaScript, you could do to check for existence on config.option without raising a TypeError, however, the assignment would proceed to raise a TypeError because config is not a value which can have a property added to it. In essence, this requires the first line, resulting in no win for providing void-safety. But then, existential operator isn’t really useful in this case either.

It is an interesting idea to be able to say that ‘if config doesn’t exist, create it and then append the option property to it, but that has the potential to create brittle code by accidentally initializing values as objects that should have been created as functions (and then had properties set) or other imaginable bad side-effects. And while it may be nice to consider functionality like the Namespace function within YUI, such a thing should always be opt-in.

There is one place where the existensial operator requests functionality that I’m not sure I’d want to see in void-safe JavaScript, and that is around functions. When applied to a funciton call, the existential operator will not call the function and just proceed past it in the property chain (potentially raising a TypeError later).

At first, I felt that applying void-safety to functions was a bad idea, one likely to cause brittleness in programs. To some extent, I still feel that way, as JavaScript functions are allowed to have side effects. The question then because is it better to raise a ReferenceError by trying to execute undefined, stopping script execution, or to continue on having essentially completed a no-op? Plus, with JavaScript’s varaiable inference, where a typo can result in a new variable, there are times you’d want to have that TypeError raised, espeically during debugging.

Of course, the varibale creation by inference is disabled in strict mode, and several of these potential threats are caught by tools such as JSLint⁶, which can be more easily intergrated into your work process today than ever before.

The concern for me, therefore, is that the behaviour I want in development will not necessarily be the behaviour I want in production. Where void-safety comes in the most useful is likely in the processing of a JS object pulled in from an external source, be that a web service, iframe, or WebWorker, where the simplified data access with increased safety is potentially very useful.

I seem to remember seeing Brendan Eich and the Mozilla team (I’m not sure how involved the rest of the EcmaScript community is yet) discussing a new ‘debugging’ namespace for JavaScript, though I’m having trouble finding the source. I think the void-safety could be a good flag in this environment. By default, turn void-safety on. It makes scripts safer as the browser won’t abort script execution as frequently. But developers could turn it off for their browser, allowing them more powerful debugging.

I’m still on the fence about this proposal. It can make data lookups simpler and safer, without adding new syntax, which is a win. But there are definitely circumstances where it can potentially hide bugs, thus making a developer’s life more difficult if it can’t be disabled. I do think I will raise the issue on es-discuss, as I think it at least warrants discussion by that community, and it may be that there are good historical reasons to not change this behaviour that others who have been buried in these problems longer than I will be familiar with.

References: 1. https://mail.mozilla.org/listinfo/es-discuss 2. https://mail.mozilla.org/pipermail/es-discuss/2011-April/013697.html 3. http://eiffel.com/ 4. http://oreilly.com/catalog/9780596515171 5. http://jashkenas.github.com/coffee-script/ 6. http://jslint.com/

Introducing connect-conneg

2011-04-20T17:30:00Z

Content Negotiation is the practice of using metadata already included via the HTTP specification¹ to customize what your web server returns based on client capabilities or settings. It has been oddly absent in a lot of major sites, with the Twitter API² requiring you specify the format of the return as part of the URI, instead of using the HTTP Accept header, and http://google.fr/ returning the French representation, regardless of the Accept-Language header (to be fair, http://google.com/ does localize).

While there are benefits (largely security benefits, unfortunately) to owning your domain in every country code, it is cost-prohibitive to many organizations, and your customers are already telling you what language they want your content in. Admittedly, many sites may wish to have a way to override the browser’s language settings, but this should be handled via user configuration, not URI.

Where I find content negotiation to be most useful is in the space of language customization. My elder sister is getting married soon, and guests are coming from the US, Italy, and Mexico, which has required all the web-based material to be made available in all three languages. For her main wedding site, the top-level sidebar looks like this:

Here, we have three links that are functionally identical, each taking the user to a localized version of the page content, but hiding said content behind at least one link, and exposing the user to the fact that all three languages are available, something they probably do not care about. Now, with the CMS that they are using, this is the best solution that I can see. Fact is, most CMSes do a terrible job of allowing for multiple language content, but that is an issue for another post.

However, for their RSVP system that I am building on NodeJS³ using ExpressJS⁴, I didn’t view this as an acceptable solution.

Express does make one nod to Content-Negotiation, in the form of it’s ‘request.accepts’⁵ method, which enables the following:

if (request.accepts('application/json')) {
    response.send(jsonObject);
} else (request.accepts('text/html')) {
    response.render(templateName, data);
}

However, this implementation, in many ways, misses the point. The MIME types in the Accept header (or the language codes in the Accept-Language header) can provide was are called ‘q-values’, or numbers between 0 and 1 to indicate preference order. Consider the two header options.

Accept: application/json, text/html
Accept: application/json;q=0.8, text/html

What this tells the server is that a response either in JSON or HTML is acceptable, but in the first case, JSON is preferred, while HTML is preferred in the second. However, for the above code, this preference is ignored. Using Express’ accepts method, I’ve decided that if they want JSON at all that’s what I’m sending, even if they might prefer a different representation I offer.

For Acceptable Types, this is less relevant, but for languages, it’s very important. Most every user will have ‘English’ as one of their accepted languages, even though for many it won’t be their preferred. Which is why q-value sorting is so important.

Connect-conneg⁶, which is available on Github right now, is pretty simple right now, but I have plans to add helper methods for common activities. Basic usage for languages is as such when using the Connect or Express frameworks:


In the above example, language, acceptableTypes, and charsets are statically
exported functions, built using the same method exposed as conneg.custom. For
each method, this will pull the HTTP Header, and sort the values per the rules in
RFC 2616. These lists will be mapped to the following properties on the request object.


Accept-Language -> languages
Accept -> acceptableTypes
Accept-Charset -> charsets


These are exposed as separate methods, so that you can 'mix and match', for my use,
I'm only caring about languages for right now. Frankly, I can't imagine a circumstance
right now where you'd want to use any charset instead of UTF-8, but it's there for
completeness.

What I haven't implemented in connect-conneg just yet are the helper methods to
determine what the 'right' thing to do is. For languages, I'm using the following
method right now:

function getPreferredLanguage(acceptedLanguages, providedLanguages, defaultLanguage) {
    defaultLanguage || (defaultLanguage = providedLanguages[0]);

    var index = 999;
    acceptedLanguages.forEach(function(lang) {
        var found = providedLanguages.indexOf(lang);
        if (found !== -1 && found < index) { index = found; }
        found = providedLanguages.indexOf(lang.split('-')[0]));
        if (found !== -1 && found < index) { index = found; }
    });

    if (index === 999) { return defaultLanguage; }
    return providedLanguages[index];
}


At the moment, I’m still thinking through how this will be implemented as
library code. The above certainly works, but I’m not sure I understand the
structure of Connect as well just yet to build this in the most efficient
way. For languages, provided and default could (and probably should) potentially
be defined on the object created by connect, at which point, should I present
the list, or just the language they want? How do I deal with different
endpoints having different content-negotiation requirements?

I will be continuing to hack on this, and I’m going to try to get it on NPM
soon, though the git repo is installable via NPM if you clone it. So, please,
look, file bugs, make comments, changes, pull requests, whatever. I think this
is a useful tool that helps provide a richer usage of HTTP using the excellent
connect middleware.

Links: 
1. http://www.ietf.org/rfc/rfc2616.txt
2. https://developer.twitter.com/doc/get/statuses/public_timeline
3. http://nodejs.org/
4. http://expressjs.com/
5. http://expressjs.com/guide.html#req.accepts%28%29
6. https://github.com/foxxtrot/connect-conneg

Why I Use YUI3

2011-04-13T18:00:00Z

The JavaScript Community is an interesting one. It grew up from a language which is unique in that, as Douglas Crockford¹ says, no one bothers to learn before using. It’s success as a language is indicative of how good a language it is, when you are able to get past the DOM and a few of it’s less-well considered features. And that flexibility has been amazing in terms of innovation. Look at the plethora of modules available for the barely year-old NodeJS², the dozens of script loaders and feature shims, and the many libraries for DOM abstraction like YUI³ and jQuery⁴.

It is, therefore, that I find it interesting that when Crockford was on Channel 9⁵ Live for MIX 2011⁶ yesterday, that when he suggested YUI, it responded in so much surprise and nascent criticism from the many, many jQuery proponents inside of the Microsoft Developer community. The comment is hardly a surprise, and not because the Crock-en⁷ works for Yahoo! He’s not on the YUI project, and while I’m sure he participates in code reviews, his name does not appear in the commit history of either YUI2 or 3. He has, however, been critical of jQuery and it’s creator, John Resig⁸, in the past, often making snide remarks about ‘ninjas’.

I am not defending Crockford for his criticisms, or even seriously claiming that words from the mouth of Douglas should be taken as gospel. Admittedly, Douglas is a mythic figure these days, and he is very smart and has done great work creating and promoting best practices that have led directly to today’s Golden Age of JavaScript.

I am also not trying to say “don’t use jQuery”, though I tend to think you shouldn’t. My concern is the apparent bifurcation of the JavaScript community into ‘people who use jQuery’ and ‘everyone else’. Now, part of the reason I am a bit anti-jQuery is because most people I know who are heavy users of the library, don’t actually write much JavaScript, they mostly perform copy-paste programming of other people’s code, and often don’t develop much of an understanding of the language or it’s abilities. Incidentally, they like it that way.

I had started JavaScript doing Pure DOM work, and it was everything that makes people hate JavaScript (when really, they usually hate the DOM, and it’s inconsistent implementation). My needs, however, had been very basic, I wasn’t even doing XHR at that point, so it worked. The JavaScript I wrote at that time also wasn’t very good, looking back. Like Crockford, I didn’t really bother to learn the language. I had done plenty of Java and C++ in my university work, and so JavaScript’s visual familiarity led me to a lot of assumptions that were simply untrue.

Eventually, I needed a bit more. I required a date selection widget for a new project, and had also been reading a lot of the performance tips shared by Yahoo, which ended up leading me to YUI2. YUI2 felt quite a bit like Pure DOM, so it was familiar, and it provided a good set of high-quality widgets that did everything I needed, and quite a bit more. Though I started using YUI2, and read JavaScript: The Good Parts¹⁰, YUI2 definitely had some major weaknesses, which led to the negative attitudes many people seem to have to YUI to this day. The library was verbose, deciding what components you needed could be difficult, even when using the Configurator tool. And good luck writing your own widgets, it was time consuming and immensely repetitive due to the lack of any sort of standard framework.

But these weaknesses were all identified, and by the time I’d started using YUI2, YUI3 was already in it’s design phase, and when it’s first previews were released, I knew it was something special. It brought Loader, a tool I was intimately familiar with from YUI2, into the forefront making it simple to use. It defined a set of building blocks that promised to make widget creation, perhaps not trivial, but dramatically easier. It integrated CSS Selectors, the killer feature that everyone was so excited about in the jQuery world. It provided a plugin and custom event architecture that allows for easy composition and customization in a way that I hadn’t seen in any other library.

To this day, many of the Widgets in YUI2 haven’t been released in the Core of YUI3 (though many of Gallery¹¹ counterparts of varying levels of functionality and quality), which a many people see as a weakness. However, this is similar to how other projects operate, where the UI Widgets are a different project from the internal core, and that’s great. The fact is that there are more people building cool things for YUI3 than ever were for YUI2, and for those that have worked in other libraries, they almost all say that they find it easier and faster to build their code than in the other options available.

It is sometimes frustrating that the tools I want don’t always just exist, or perhaps aren’t quite right, but I have found very few problems that I haven’t been able to prototype in at most a few hours of work using YUI3, including the first run of my attempt to re-think multiselect. Of course it takes longer than a few hours to polish the idea and make it shine, but rapid prototyping is immensely useful. Plus, for a well-polished widget that does, say, 75% of what I require, it is easy using the framework to extend the behavior I require without needing to directly modify the code for the core widget. There are exceptions to this flexibility, but they are definitely the minority in my experience.

I don’t anticipate that this will directly buy any converts. I have shown no code. I have made comments that will likely offend someone. This post is more a collection of my thoughts on how I ended up using this particular library, and why, when I leave my current position, I’ll continue to advocate for YUI wherever I end up. I am not so inflexible as to refuse other options, but I like to use tools that I know are a good idea, and not just ones that look like it¹².

http://crockford.com/
http://nodejs.org/
http://yuilibrary.com/
http://jquery.com/
http://channel9.msdn.com/
http://live.visitmix.com/
https://mail.mozilla.org/pipermail/es-discuss/2011-March/013415.html
http://ejohn.org/
Resig is working on a book Secrets of the JavaScript Ninjas
http://oreilly.com/catalog/9780596517748/
http://yuilibrary.com/gallery/
http://boagworld.com/technology/dustin-diaz/

Building a YUI3 File Uploader: A Case Study

2011-04-06T17:30:00Z

Off and on for the last few weeks, I’ve been trying to build a file uploader taking advantage the new File API¹ in modern browsers (Firefox 4, newer versions of Webkit). It’s up on my Github², and unfortunately, it doesn’t quite work.

The first revision attempted to complete the upload by Base64 encoding the file and custom building a multipart-MIME message including the base64 encoded file representation using the Content-Transfer-Encoding header. This resulted in the NodeJS³ server using Formidable⁴ for form processing saving the file out as Base64. At first, I considered this a Bug, but per the HTTP/1.1 RFC (2616)⁵:

19.4.5 No Content-Transfer-Encoding

HTTP does not use the Content-Transfer-Encoding (CTE) field of RFC 2045. Proxies and gateways from MIME-compliant protocols to HTTP MUST remove any non-identity CTE (“quoted-printable” or “base64”) encoding prior to delivering the response message to an HTTP client.

Proxies and gateways from HTTP to MIME-compliant protocols are responsible for ensuring that the message is in the correct format and encoding for safe transport on that protocol, where “safe transport” is defined by the limitations of the protocol being used. Such a proxy or gateway SHOULD label the data with an appropriate Content-Transfer-Encoding if doing so will improve the likelihood of safe transport over the destination protocol.

The reason for this seems to stem from the fact that HTTP is a fully 8-bit protocol, while MIME was designed to be more flexible than that. One of the CTE options is ‘7-bit’, which would complicate an HTTP server more than most would like. Why 7-bit? ASCII⁶. ASCII is a 7-bit protocol for transmitting the English alphabet. Eventually it was extended to 8-bit with the ‘expanded’ character set, but in the early days of networking, a lot of text was sent in 7-bit mode. Which made sense, in that it amounts to a 12.5% reduction in data size. These days, when best practice is to encode our HTTP traffic as UTF-8 instead of ASCII (or other regional character sets), the problem seems to be largely gone.

I still take issue with the exclusion of Base64 encoding. Base64 is 8-bit safe, and while it makes the files larger, it had seemed a safe way to build my submission content using JavaScript, which stores it’s strings in Unicode.

And I wasn’t wrong. My next attempt, based on a blog post about the Firefox 3.6 version of the File API⁷ attempted to read the file as a Binary string and append that into my message. This also failed, but more subtly. The message ended up having a few bytes, which some hexdump analysis seems to suggest was related to some bytes being expanded from 1 byte to 2 based on UTF-16 rules. Regardless, the image saved by the server was unreadable, though I could see bits of data reminiscent of the standard JPEG headers.

A bit more looking brought me to the new XMLHttpRequest Level 2⁸ additions, supported again in Firefox 4 and Chromium. Of particular interest was the FormData object introduced in that interface. It’s a simple interface, working essentially as follows:

var fd = new FormData(formElem);
fd.append("key", "value");

It’s simple. Pass the constructor an optional DOM FORM element, and it will automatically append all of it’s inputs. You can call the ‘append’ method with a key and value (value can be any Blob/File), and then send the FormData object to your XHR object. It will automatically be converted into a multipart/form-data message and uploaded to the server using the browsers existing mechanism for serializing and uploading a form. If I have a complaint, it’s that in Chrome at least, even if you’re not uploading a file, it will encode the message as multipart instead of a normal POST message, which seems a bit wasteful to me, and hints that the form data isn’t being passed through the same code path as a normal form submission.

It is at this point that YUI3’s io module fails me. Let me start by saying that io is great for probably 99% of what people want to use it for. It can do Cross-Domain requests, passing off to a Flash shim if necessary. It can do form serialization automatically. It can do file uploads using a iframe-shim. While it was designed reasonably modular and it only loads these additional features at your request, this apparent ‘modularity’ from a user perspective is actually hard coded into the method call. For instance, for the form handling, we currently have this:

if (c.form) {
    if (c.form.upload) {
        // This is a file upload transaction, calling
        // upload() in io-upload-iframe.
        return Y.io.upload(o, uri, c);
    }
    else {
        // Serialize HTML form data into a key-value string.
        f = Y.io._serialize(c.form, c.data);
        if (m === 'POST' || m === 'PUT') {
            c.data = f;
        }
        else if (m === 'GET') {
            uri = _concat(uri, f);
        }
    }
}

This code example is used purely to suggest that io is currently designed in a way that is a bit inflexible. In fact, 90% of the logic used in io occurs in a single method, and while there are events you can respond to, including a few that occur before data is sent down the wire, you’re unable to modify any data used in this method in your event handlers. So, if this method does anything that is counter to what you’re trying to do, you’re forced to reimplement all of it. And, of course, the method does something counter to my goals.

c.data = (Y.Lang.isObject(c.data) && Y.QueryString) ? Y.QueryString.stringify(c.data) : c.data;

io-base optionally includes querystring-stringify-simple, so there is a very high likelihood that it will be present. And having my FormData object trying to be serialized in this method will result in all of my data magically disappearing. It is unacceptable to me to tell users of my file-upload module that they must turn off optional includes (though for production, you probably should be anyway, but that’s another discussion).

IO being so inflexible makes sense, in some ways. It’s a static method, not a module, so configuration can be difficult, since the only way to add extension points would be to send them in via the configuration object, which complicates things in other ways. The io module, it seems, requires a reimaging.

And we’ve got something. Luke Smith has put together a code sketch of a potential future for IO⁹, which breaks things out in an exciting fashion. For my file upload, I can declare a Y.Resource to my endpoint, set some basic options when declaring the resource, and post multiple messages to the resource. It actually shortens my code quite a bit, and while I still need to look at a shim of some sort for those browsers which lack an implementation of the File API and XHR Level 2 before I push it into the gallery, since I would want it to work across all A-Grade browsers.

Unfortunately, the code there is just a proposal, it doesn’t actually work. But I’m excited about the proposal, and I’m going to try to get it at least partially functional, but for now I haven’t worked on it just yet, because I wanted to touch base with Luke to see what kind of expectations there were about the API, and there are a few important ones (though I don’t think they’ll impact me getting things sort of working). Hopefully I’ll have this working in Firefox 4 and Chrome very soon, and then I can start working on the shims necessary to support less-capable browsers.

References: 1. http://www.w3.org/TR/FileAPI/ 2. https://github.com/foxxtrot/html5file-yui3uploader 3. http://nodejs.org/ 4. https://github.com/felixge/node-formidable 5. http://tools.ietf.org/html/rfc2616#section-19.4.5 6. https://secure.wikimedia.org/wikipedia/en/wiki/ASCII 7. https://developer.mozilla.org/en/usingfilesfromwebapplications 8. http://www.w3.org/TR/XMLHttpRequest2 9. https://github.com/lsmith/yui3/tree/master/sandbox/io

Open Government

2011-03-29T16:32:18Z

The prospect of government transparency is very important to me. I firmly believe that best way to protect the integrity of our union is by the populace taking a more active role in their own governance. This is why I have always been such a supporter of GovTrack, though I have had to become increasingly selective on what events I track (I limit to activity of my own legislators, a few classes of issues, and the occasional specific Bill). It was the key part of Obama’s electoral platform that I supported, though there was plenty with his candidacy that I did not. We, the people, require more data to be able to participate meaningfully with the government.

Incidentally, the Open Government book, a collection of essays from a wide variety of people trying to better the interaction of the public with government, was driven in large part by the promises made by Obama during his campaign, and the efforts begun shortly after his election, like data.gov. The book was published just over 1 year ago, and at that time, nearly every single contributor felt that the efforts to date were disappointing. I doubt many people’s minds have changed much in this regard.

Now, I work for a state institution, and I’ve made it a goal to make our data more accessible. I understand that it’s hard, but the data that I expose is unquestionably public, has been available in the past (I’m just trying to make it better), and I have a lot fewer roadblocks to the work I’m doing than I suspect most people do. But the federal transparency efforts have been wrought with delays and missed deadlines. Part of this is the fact that much of this data has been behind a paywall in the past, since it required people to physically copy and mail the information, with the new directives, that income, which I suspect had become something of a profit center for many department since transcripts are for us, will be drying up.

A great many of the essays in this book are from people associated with projects like GovTrack, which take government information (either that freely available, or sometimes behind paywalls which they then digitize), and often do analysis of the data to show connections that may not have been directly obvious. Sites like FollowTheMoney.org, or MAPLight.org, both of which show voting records and campaign contributions, and how they may be related. Both sites do a reasonable job of not editorializing on what they’re presenting.

My favorite technology that I read about was RECAP, which is a Firefox Plugin (I’ve considered porting it to Chrome) which detects when you’re browsing PACER, an online database of all US Federal Court decisions used when researching case law. PACER costs about 8 cents per page, with a max cost of $2.40 for a document. While not an exceptional amount of money, with the number of documents that someone may need to pull can really add up, particularly for a non-profit legal defense firm. Using RECAP, when you request a PACER document, it checks the RECAP database, serving it for free if it exists, and if it doesn’t, if you buy the document, it will be uploaded to RECAP automatically. Even for-profit legal firms can benefit from this, by reducing their research costs (and hopefully passing that on to clients).

This is a really interesting book, but like others of it’s ilk (collections of essays on a similar topic, Beautiful Code being one example), this is not a book meant to be read from cover to cover without breaks as far as I’m concerned. As someone who wants to write a review, this puts me in an awkward position. By the end, I was bored, and not inclined to say much nice about the book. Hell, the only reason the tone of this review is so positive is because I finished reading this book almost two months ago and have had time to reflect on it.

The reason the book got boring by the end was because everyone contributing to it had similar ideas on why openness in government is important, so I kept reading the same points repeated time and again in almost every single essay, and not just in single sentences, but often whole paragraphs felt paraphrased and redundant. To be clear, I don’t know how one would ‘fix’ this issue in a compilation book such as this, but when reading straight through, I know it’s detrimental to my experience.

Still, I think the work these people are doing is interesting and important, and there are plenty of resources I’m now aware of that I wasn’t before, and a lot of great disucssion about the challenges in the data and the way it’s collected that I hadn’t been aware of. It’s absolutely worth a read, but it’s absolutely unnecessary (and I’d say unadvisable) to read from cover to cover.

The Productive Programmer

2011-03-16T04:32:58Z

Neal Ford’s The Production Programmer¹, published by O’Reilly Media, claims to teach you the tricks of the best programmers in the industry. The book proceeds to meet this goal by, in the first half, giving specific tips and tricks for various applications and tools across Windows, Mac, and Linux. The second half discusses the techniques that one can use to learn or familiarize yourself with a new tool in order to ultimately improve your productivity.

I’ll be frank. I don’t think this book is worth the value.

The first half talks about too many tools, too many platforms. It’s not able to cover any one tool terribly well, and it’s attempts to cover a given class of tools felt unfocused and messy.

The advice overall is sound. Learn the tools you’re using. Focus on tools that eliminate the need to move your hands from the keyboard. Automate tasks, often by building scripts.

Maybe it’s just because I’ve been developing on Unix for the past decade, and grew up on the DOS command line before that, but most everything in this book just felt obvious. Hell, I was playing around with Beagle² (now defunct) for desktop search in it’s very earliest days, well before Google had their own desktop search product³. I live at the command line, even when I’m on Windows as I am at my current day job.

I am not trying to brag. It’s just a familiarity that I’ve gained that even in college I recognized as grossly missing in a lot of my class mates who were bound to their GUI’s and their mouse. For me, the advice was all obvious.

Part two, which covers more of software best practice, like not over developing, or doing proper testing, learning multiple languages, using continuous integration, is also great advice, but I can’t help but think that other books cover the topics better. Admittedly, this book isn’t trying to be comprehensive or definitive on any of these topics, but in it’s general coverage, it seems to fall short.

In retrospect, I am not the target for this book. I read plenty of blogs and other books on the subject of software development and general computing. Ben Collins-Sussman claims that there are two kinds of programmers, the 80% who just plug away, and the 20% who really excel and care⁴, while Jeff Atwood makes the claim that the 20% are largely the people who actually take the time to read this sort of material⁵. I am not so proud as to claim myself to be in that 20%. I think I could be some day, and I know that I am above 50%, but I am reluctant to accept the idea of a hard 80/20 split.

This book isn’t bad. It’s fairly entertaining, and it’s got a lot of good information. But it’s almost too introductory. If you feel like you’re trying to get your feet underneath you, and that you really, truly want to be at the top of your field (as you should), then by all means pick up this book. It’s worth it. But if you’ve spent a bunch of time studying the material, reading the blogs, and most importantly, working on techniques to make you faster and better at your day-to-day tasks, then odds are you won’t get too much out of this book.

http://oreilly.com/catalog/9780596519544/
http://beagle-project.org/Main_Page
http://desktop.google.com/
http://blog.red-bean.com/sussman/?p=79
http://www.codinghorror.com/blog/2007/11/the-two-types-of-programmers.html

Boise Code Camp 2011

2011-03-09T17:45:19Z

A few weekends ago, on February 26th was the fifth Boise Code Camp¹ held at the Boise State University campus. It is the third Code Camp in Boise I have attended, and sadly it was reduced to a single day because they felt they didn’t have enough submissions. As I didn’t submit a talk this year, I suppose I’m at least partially to blame for that, but either way it was still a solid event.

There was substantially less JavaScript talk this year than in years past, the only talk being strictly on the subject was an introductory jQuery talk. Of course, had I submitted a talk, it would have been an introductory talk on YUI3, meaning that we wouldn’t have had much in the way of advanced JavaScript topics. On the one hand, there is still definitely a big audience for introductory JavaScript concepts in the greater developer community, but I’d love to do more advanced talks at this sort of event.

But in spite of the small number of JavaScript talks, there was still plenty of web talks at the conference this year, though the first talk I went to was one attempting to show the basics of what a ‘Monad’ is in Functional Programming². I say attempted because I had a hard time drawing much from the talk, though that might be because it was the first talk of the day, though I think it’s more that describing Monads is generally made more difficult than it ought to be.

It did however occur to me, that, at the most basic level, a Monad can be described as a collection containing a homogeneous collection of data, whose methods are designed to support chaining commands together into a pipeline. Incidentally, this is very much how working with DOM nodes in jQuery or YUI3 works, though I’m pretty sure either library wouldn’t describe themselves as ‘Monadic’, and it’s probably not wholly accurate, but I think it provides a working definition to help get someone started on investigating this concept.

Second hour, I attended Glenn Block’s³ talk on WCF and REST, which was really interesting. I had used WCF in .NET 3.5, and it was an improvement over the older web-service mechanisms that .NET provided for building web services. However, the new WCF is amazingly customizable. Content Negotiation is nearly trivial, Glenn showing off an easy way to generate vCard files based on the Accept headers sent from the client. Luckily there is a reasonable parallel of this talk at MVC Conf⁴ this year⁵. But having recently done up a simple RESTful service in ASP.NET MVC, the tooling that WCF provides is really interesting to me, plus it’s Open Source and available now⁶.

After lunch, I attended a talk about F# on the Web given by Ryan Riley⁷. Ryan has built a Sinatra⁸ from Ruby clone in F#, which reminded me a bit of Express.js⁹ from Node.js, in that the app is it’s own server and it’s based on routing paths to commands. F#, particularly with it’s asynchronous processing, allows for very clean code for spec’ing out a web service. It’s still a work in progress, but definitely something to at least watch. Implied callbacks in async processing is pretty cool.

I attended Ole Dam’s Leadership talk, which was really inspiring in, but the slides don’t seem to be posted (unfortunately), and it’s hard to describe. The short version is that becoming a good leader requires work and care, and most of the leadership advice available is pretty terrible. I won’t say much more about it, but Ole apparently gives these talks all over the place and for a relatively low cash outlay, so if given the opportunity to hear him talk, I’d suggest taking advantage.

Finally, I attended a talk on web performance measurements, talking about the metrics that Google uses. They have some JS on their homepage that measures how long it takes for things like image or script loading to being and end and reports that back to the server. It was interesting, but I think I preferred what the Flickr guys mention in their YUIConf 2011 talk¹⁰, in that they measure only what they care about, which in Flickr’s case is when the image is loaded and when the Scripts are loaded. They just don’t care about the rest of the stuff. I was expecting more out of this talk that I got, since it was a really high-level look at Google’s JavaScript without even much of a discussion about how to improve those numbers or anything else. I am, however, excited about the web timing specification¹¹ in from the of W3C and implemented in Internet Explorer 9. That should be really interesting to have.

Overall, the event wasn’t as valuable to me this year as in years past, but it was still an excellent event, particularly for one that is free to attendees. If nothing else, it’s a great opportunity to meet up with people that I only see once a year or so.

http://boisecodecamp.org/
https://secure.wikimedia.org/wikipedia/en/wiki/Monad%28functionalprogramming%29
http://blogs.msdn.com/b/gblock/
http://www.mvcconf.com/
http://channel9.msdn.com/Series/mvcConf/mvcConf-2-Glenn-Block-Take-some-REST-with-WCF
http://wcf.codeplex.com/
http://wizardsofsmart.net/
https://secure.wikimedia.org/wikipedia/en/wiki/Sinatra_%28software%29
http://expressjs.com/
http://developer.yahoo.com/yui/theater/video.php?v=yuiconf2010-harmes
http://dev.w3.org/2006/webapi/WebTiming/

Adding a Column to a YUI3 DataTable

2011-02-24T18:30:00Z

In December, I wrote instructions on how to add additional columns to a YUI2 DataTable while using YUI2in3. Since then, YUI3.3.0 was released, and in it the first Beta release of YUI3’s DataTable. Recently, I decided to take some time to upgrade my implementation to use the new datatable, knowing that my needs were relatively simple and straight-forward, I figured it would be a good opportunity to test the new API and make it fit my needs.

The following implementation follows exactly my YUI2 implementation. It takes an existing HTML Table, converts it to a DataSource, and then plugs that into a new DataTable instance. YUI3, with it’s plugin architecture, makes certain aspects of this easier than it was in YUI2. In spite of that though, it’s clear that the YUI3 DataTable API is still pretty fresh and probably needs some work. This post is to serve both as documentation for the current state of YUI3 DataTable, as well as an examination of where potential improvements that could be made to the API.

Let’s start with the module list:

Y.use('datatable', 'datasource-xmlschema', 'datasource-local', function(Y) {

To begin, I’m not going to be using any of the datatable plugins, like datatable-sort, but I will address them later. First, we need to set up the DataSource. In YUI2, there was a datasource type for HTML tables, which would parse the columns out in order. Currently, no such datasource exists for YUI3, but it can be easily mapped to an XML Schema:

var dataSource = new Y.DataSource.Local({
    source: Y.Node.getDOMNode(Y.one('#tableId tbody')),
    plugins: [
        {
            fn: Y.Plugin.DataSourceXMLSchema,
            cfg: {
                schema: {
                    resultListLocator: "tr",
                    resultFields: [
                        { key: "abbr", locator: "td[1]" },
                        { key: "name", locator: "td[2]" },
                        { key: "loc", locator: "td[3]" },
                        { key: "loc_href", locator: "td[3]/a/@href" }
                    ]
                }
            }
        }
    ]
});
Y.one('#tableId').remove();

This will remove the table from the DOM, while still keeping it in memory for manipulation, which will be important shortly. This code is taken almost directly from the YUI3 examples for DataSource, but it also marks one of the first places I ran into a caveat with YUI3 DataSource.

In my data, the third column was optionally a link. In the YUI2 DataSource for HTML Tables, the column was read as essentially being the innerHTML of that third column, however, the way that DataSourceXMLSchema is currently implemented, it will always take the textContent of the Node before it takes the XML representation. As such, I had to grab the href attribute off the link (if it exists), which I’ll be able to use with a custom formatter next, when we build the datatable.

var table = new Y.DataTable.Base({
    columnset: [
        { key: "abbr", label: "Abbreviation", sortable: true },
        { key: "name", label: "Building Name", sortable: true },
        { key: "loc", label: "Location", sortable: true, formatter: function (obj) {
                var data = obj.data;
                if (data.loc_href) {
                    return Y.Lang.sub("{loc}", data);
                } else {
                    return obj.value;
                }
            }
        }
    ],
    plugins: [
        { fn: Y.Plugin.DataTableDataSource, cfg: { datasource: source} }
    ]
}).render('#datatable');
table.datasource.load();

This is pretty straightforward, and most of the definition should be familiar to anyone who has used YUI2’s DataTable. The only thing of note is the formatter function on the columnset definition list. It is able to take advantage of the fact that any unmatched fields in the datasource will be undefined, allowing me to use it as a condition for selectively formatting. In an implementation of DataSourceHTMLTableSchema, which I may do for the Gallery, the value will be the innerHTML, and not it’s text.

So far, so good. The API is different only in it’s use of plugins, and aside from the fact that DataSource clearly prefers to deal with data returned off of IO calls instead of HTML, which is a fairly minor inconvenience in my case. The work done here by Tilo Mitra and Jenny Han Donnely was, up to this point excellent. However, there are some caveats to come in the adding of the additional column.

The first step to adding the column is to add it to the datasource. This can be accessed through the table.datasource.get('datasource').get('source'). This returns the TBODY DOM Node, which is the other problem with the DataSourceXMLSchema method that I’m using. I’ve broken the YUI3 abstraction by being provided with a raw DOM Node, something which doesn’t seem correct with the rest of the library. Again, something a DataSchema designed for HTML Tables will be able to handle more appropriately.

var schema = table.datasource.get('datasource').schema.get('schema');
schema.resultFields.push({ key: "new", locator: "td[4]" });
table.datasource.get('datasource').schema.set('schema', schema);

Y.Array.each(table.datasource.get('datasource').get('source').rows, function (row) {
    var node = new Y.Node(row);
    node.append("New Column!");
});

Aside from the brief cognitave dissonance of getting a raw DOM Node, and the verbosity of getting properties off of nested plugins, this is not a difficult process, but actually adding that column to the DataTable is where things get more difficult. In YUI2, it was as simple as calling table.insertColumn() with a new column definition. In YUI3, doing this required me reading a lot of the internals of DataTable, which was not completely pleasant. But let’s start with the code.

var columnset = table.get('columnset'), columns = columnset.get('definitions');
columns.push({ key: "new", label: "This is a new column", sortable: true});
columnset.set('definitions', columns);
columnset.initializer();
table.set('columnset', columnset);
table.datasource.load();

Looking at it now, it’s a bit anti-climactic. It isn’t that much code, but writing it took a deep understanding. But I’m going to hilight a few of the most important calls. The value pushed onto the columns array is the same as the column definition when building the datatable, however, what is actually done internally is YUI builds up a ColumnSet and a collection of Y.Columns. It would, no doubt, be possible to create the Y.Column instance directly and append it to the ColumnSet, but by modifying the definitions and reinitializing the set, I don’t need to know the details of how it’s set up. Finally, setting the columnset Attribute on the table, forces the table to generate the table headers for the new columns, and finally we need to reload the data.

The above code came from a desire to do as little work as possible in the backend, as adding a new column seems that it should be fairly easy. However, in truth, while I’m not creating a new ColumnSet object, I’m doing all the work except creating a new object. I could just as easily take the definitions array, and pass that into table.set('columnset', columns);. This does a bit more work, but probably not much, and saves a couple of lines of code in my source, looking perhaps a bit simpler.

var columns = table.get('columnset').get('definitions');
columns.push({ key: "new", label: "This is a new column", sortable: true});
table.set('columnset', columns);
table.datasource.load();

Either way, this is not an ideal API. To start, an ‘addColumn’ or ‘insertColumn’ method is required, either on the DataTable, or the ColumnSet. Putting it on the ColumnSet makes a lot of sense, but it would still require the set('columnset' method on the DataTable in order to ensure that the DataTable was updated with the new column definitions, or the columnset should fire a changed event that the DataTable can respond to.

There is, however, one potentially big issue with DataTable in it’s current implementation. When table.datasource.load() is called, it creates a new RecordSet object which is what is the actual underlying data for the DataTable. This is, in part, because DataSource is mostly stateless. It’s designed to be pointed at a collection of data, and to return that collection to a callback function. RecordSet actually contains that data. The problem with the current implementation of the DataSource plugin for DataTable, is that it completely replaces the RecordSet in use, which can actually break other plugins on DataTable.

For instance, I was using the DataTableSort plugin to allow my headers to be clickable to sort by any column. This plugin works by augmenting the DataTable’s recordset with the RecordsetSort plugin to help with it’s implementation. However, since DataSource replaces the Recordset, instead of modifying it’s data (even if that were to be by emptying it and reloading it), every call to table.datasource.load() must be followed by replugging the RecordsetSort into the Recordset, which is not expected behaviour. I also found DataTableSort to be painfully slow, but that might be an implementation detail on my end, I haven’t determined yet.

YUI3 DataTable doesn’t support editing yet. Row Clicks are not explicitly supported (though easily delegated). In short, if you just need to display data and want to allow some interaction, such as sorting and filtering, then YUI3 DataTable will probably meet your needs, but it has a ways to go, and the API is liable to change going forward. Still, it’s a solid core to work from, and I look forward to seeing it moving forward, especially if these API issues I ran across can be fixed.

Book Review: RESTful Web Services Cookbook

2011-02-16T18:30:00Z

O’Reilly Media is really fond of the ‘Cookbook’ format of technical guide. I’m guessing it sells well for them, and I’m glad because I happen to enjoy it. They’re good for reference for solving specific kinds of problems, and if you’re like me, they provide a solid ‘learn-by-example’ mechanism to learn how to work with a new technology or technique. It’s not ideal for everything, but it’s good. I’ve had web-service stuff hanging in the back of my head for a while, so I decided to pull the “RESTful Web Services Cookbook” by Subbu Allamaraju¹ to stimulate my thinking on the matter.

And it worked greatly. This particular cookbook is of the ‘Patterns’ variety, as there is absolutely zero code anywhere in the book (XML and HTTP Headers don’t count). This never really feels like a weakness, since HTTP and XML (and even JSON) are all well supported across virtually every programming environment you could imagine.

While these cookbooks are not necessary to read from cover to cover, and this is no exception, I found it to be comfortable to read from that perspective. The chapters are organized such that they start with the simplest things that you need to know to work in REST, and then slowly layers on top of that with more detail and nuance that help to make a really stellar web service. And I suppose that you could be building a service while reading this book, though I found it short enough I’d probably suggest reading at least any potentially relevant recipes while in the design phase, since I think some recipes are easier to integrate into an existing design than others, though depending on your tolerance for changing the API, or the ease of your ability to keep multiple version endpoints, this may be more or less of a problem you.

I started reading this very shortly after I had put together an API for Washington State University Schedule data², and I found this really inspiring. Our heaviest API user is using a few hundred megabytes per day, and the Queries chapter hinted me toward the way that I can reduce that usage dramatically by giving them more querying options, which incidentally also supports my plans for a Mobile site that can be completely API driven in the near future, and saves me from creating a ton of specialized endpoints when query parameters just make more sense. A lot of this book isn’t relevant to what I’m doing at this moment, since I’m working with GET only resources right now, but the more I play around with REST, the more I like it as a mechanism for web services, there is an elegance there that I don’t see in other web-based API standards, and it’s not really even a standard on it’s own, it just leverages other existing standards.

I think this is a great book for anyone working in REST. I have a few others on my reading list that I need to get to, but this one has definitely pushed my thinking on this technology in a way that has me really interested in how I can best design my services. I can’t say that it’s definitive, but I was really happy with having read it, and I look forward to reading more on the topic.

http://oreilly.com/catalog/9780596801694
http://schedules.wsu.edu/API/

When a Keyword....isn't.

2011-02-09T14:00:00Z

JavaScript is a beautiful language, but it has some bad parts, like any language. JavaScript’s Bad Parts can be basically be broken down into two categories: Those that are confusing, and those that are dangerous. This post is specifically about one of those dangerous features.

Recently, I found a subtle bug¹ in YUI where the array methods defined in NodeList would fail with edge-case values, such as 0. The bug was on a line of source reading as follows:

while (arg = arguments[i++]) {

This is a common trick, one often used in C/C++, where you can perform an assignment from the array, and test for the values ‘truthiness’ all in a single statement. The problem is that JavaScript has a broad definition of what is ‘false’ in the language. The list of ‘false’ values, when used in conditionalys, include: boolean false, 0, ”, NaN, null, and undefined. However, there are situations where one of these ‘false’ values would be perfectly valid.

I ran into this bug, when working on my gallery-node-extras² module, specifically the nextAll and prevAll methods, get all the siblings preceeding or following a node that match a given selector (or any valid argument to filter, which includes functions if you use my gallery-nodelist-extras³). In prevAll, I get all the siblings that precede the current node with the following snippet:

    var list = this.ancestor().get('children');
    list = list.slice(0, list.indexOf(this));

In my tests, this failed, because the call to slice was returning all the children, because the first argument, 0, is falsy, and therefore the while loop doesn’t execute at all, resulting in slice being called on the NodeList with no arguments, returning a shallow copy. This bug has been fixed in git master⁴, but Matt Sweeney didn’t take my suggested fix.

// My fix:
while ((arg = arguments[i++]) !== undefined) {

// What Matt Committed
while (typeof(arg = arguments[i++]) !== 'undefined') {

I was a bit confused. My version was more compact, and it seemed that it should be the better solution. However, I respect Matt a lot, so I knew that there must have been something I was missing. It turns out, that in JavaScript, the value of the keyword undefined can be…redefined. So, if you set undefined to something ridiculous, say the value of Pi multiplied by the Avagadro’s number⁵, my code would find itself in an infinite loop!

So, why is undefined allowed to be variable? I have no idea. If anyone can tell me how this came about, I’d LOVE to hear it. The ‘null’ keyword can’t have it’s value changed, so this just seems like a ridiculous oversight. If you’re using this oddity of the language, stop. You’re a bad person. And I don’t mind saying so.

Firefox 4 has made undefined read-only, as demonstrated by the following script.

undefined = true;

console.log(undefined);
console.log(typeof(undefined));

test2 = function() {
    undefined = true;
    console.log(undefined);
}

test = function() {
    "use strict";
    console.log(undefined);
}

console.log(undefined);
test();
console.log(undefined);
test2();
console.log(undefined);

/// Outputs in Firefox 4 Beta 10
/// undefined
/// undefined
/// undefined
/// undefined
/// undefined
/// undefined
/// undefined

/// Outputs in Chrome 9.0.597.86
/// true
/// boolean
/// true
/// true
/// true
/// true
/// true

The console.log statement inside of my ‘strict’ block is still tainted by the non-strict reassingment of undefined in Chrome. Even if you get rid of the first reassignment, calling test2() above redefines it for test(). Hopefully, when Chrome supports ES5 strict mode, it will protect undefined as agressively at Firefox 4. Incidentally, when in strict mode, trying to assign to undefined raises a TypeError due to it’s read only nature.

Why ECMAScript 5 did not promote undefined from what appears to be a variable into a protected keyword, like null is (Chrome balks at attempts to assign to null), I have no idea. Protection of undefined should not be down to setting the read-only flag on the undefined property of the window object, especially when null is already appropriately protected. This is the kind of language change that I think should have been made, because any code it may have broken isn’t just bad, it is dangerous. There are already people calling for the death of JavaScript due to such poor security decisions⁶, a sentiment I don’t necessarily agree wholesale with, but I think that the language can be made better by fixing this sort of thing. Even at the risk of breaking code.

At the very least, it would have been lovely to see this addressed in EcmaScript 5’s new ‘strict mode⁷.’ Using strict mode is simple, though it’s opt-in. Basically the first instruction in your JavaScript file, or a given method, must be the string "use strict"; For YUI3, which is built on the module pattern, this allows you to enable strict mode for your module easily, since your module code is wrapped either in an add function call or a use call. Unfortunately, strict mode is only enforced at the moment in Firefox 4, which is still in Beta.

Had this been enforced in strict mode, then we could be ensured that undefined was actually, well, undefined in all strict-mode scripts. We could have expected a reference error if, by some accidental idiocy we attempted to change the value of undefined. And, there would have been reason to begin enforcement of ES5 strict mode through a page-level meta tag which could have set a flag requiring all code on the page be strict. That way, strict mode is still opt-in, but I can be notified if I’ve either written, or am trying to use, shit code that I should probably have some concerns with.

Runtime-Built LINQ Clauses Building Expression Trees

2011-02-03T00:16:13Z

Microsoft’s Language Integrated Queries (LINQ) introduced in .NET is an amazing tool, however, there are several types of requests that can be…difficult to build in the default engines. The problem I encountered was where I had a string of data which was being treated as an array of single-character codes that I would want to query for a subset. The data in my database could appear as ‘ABCDE’, and I’d want to match it if it contains A or D, or if it contains neither.

This implementation is being used on Washington State University’s Schedules of Classes Search (pictured above), to handle the footnote search listed near the bottom of the search form. My constraints are as follows:

Footnotes is stored in the database as a CHAR(5), but is to be treated more as an Array than a String
I want the records that match either ANY of the user selected options, or NONE of the user selected options, depending on whether or not they choose the ‘exclude footnotes’ option
This should be convertable to SQL and run on the SQL Server

Point 3 is important. Using LINQ-to-SQL I had the option to pull back more records from the database and do the additional filtering on the client, but in circumstances where a user was only searching by footnotes, this would result in an enormous delay as we transfer back way too much data from the database to be filtered in a much slower .NET layer. SQL is designed for data searching, and we should let it do the work.

My initial implementation worked in the .NET layer, since I wanted something that ‘just worked’ and looked something like this:

foreach (char c in footnotes) {
    queryLocal = queryLocal.Where((s => s.Footnotes.IndexOf(c) != 1) == !excludeFootnotes);
}

Unfortunately, this didn’t even work. For one, it was slow, especially when footnotes were the only search term. For another, it only represents AND relationships, when my requirement was for OR relationships. Oh, and the s.Footnotes.IndexOf thing is because LINQ-to-SQL can’t translate the Contains method, but this is a minor issue.

For many people, unfortunately, this probably would probably be a non-starter. However, what LINQ does internally is convert the Lambda expression you provide it, into an Expression Tree, which allows the C# (or VB.NET) code to be translated to SQL. With that in mind, what’s stopping you from building your own custom expression tree? Nothing…except the willingness to step just a little ways down the rabbit hole.

Let’s first build the function we want to add to our WHERE clause out using the following prose, which does not include the ‘exclusion’ flag from the requirements:

For each character in the user input, we want to test that character to see if it is contained in a string of input from the database. If any character in the user input is found in the database input, the row should be returned. For user input ‘BD’, the following boolean expression should be generated: (‘B’ in databaseInput) || (‘D’ in databaseInput).

If the exclusion flag is set, the expression will be: (‘B’ not in databaseInput) && (‘D’ not in databaseInput), which can be more easily represented by the homomorphism: false && (‘B’ in databaseInput || ‘D’ in databaseInput). This homomorphism is important, because now we have two forms of the same boolean expression. These can essentially be represent as follows: !exclude && (‘B’ in databaseInput || ‘D’ in databaseInput), which allows me to build a single expression regardless of the value of my exclusion flag.

This can not be directly plugged in LINQ because I don’t even know the length of the user input at compile time. But let’s start with an input of 1, and look what it takes to build that simple conditional. First, some book-keeping.

var param = Expression.Parameter(typeof(Database.SectionInfo), "sectionInfo");
var indexOfMethod = typeof(string).GetMethod("IndexOf", newType[] { typeof(char) });

Now, the boolean expressions we have above are useful, but what we’re really getting ready to do is build an expression tree. Let’s begin with a single footnote conditional and see what that would look like. As a boolean expression, we have (‘B’ in dataBaseinput). In C#/LINQ-to-SQL, this will look like si.Footnotes.indexOf('B') != -1.

This gets me a parameter I can use which represents the SectionInfo table in my database that I will be querying, as well as a reference to the IndexOf method that I require for my test. You’ll need an Expression.Parameter for each argument you’ll need to add at runtime, and a Reflective reference to any methods. There is no way around this. Next, let’s build the first request. Keep in mind that Expressions can only operate on other Expressions, so we need to use the System.Linq.Expressions.Expression factory methods to generate our expressions.

// Step 1. Lookup the Footnote Properties
// The param value is the one declared in the 'book-keeping' section above.
var footnotesProperty = Expression.Property(param, "Footnotes");

// Step 2. Call IndexOfMethod on Property
// indexOfMethod was also declared above
// Expression.Constant converts a static value into an Expression
var methodCall = Expression.Call(footnotesProperty, indexOfMethod, Expression.Constant(inputCharacter));

// Step 3. Test return of method
Expression.NotEqual(methodCall, Expression.Constant(-1));

// All together now:
Expression.NotEqual(
        Expression.Constant(-1),
        Expression.Call(
                Expression.Property(param, "Footnotes"),
                indexOfMethod,
                Expression.Constant(inputCharacter)));

Next, we want to expand this to include a list of elements, which can be wrapped easily in a ForEach loop:

var optionsString = "BDF";
// Build the first request
var builtExpression = 
        Expression.NotEqual(
                Expression.Constant(-1),
                Expression.Call(
                        Expression.Property(param, "Footnotes"),
                        indexOfMethod,
                        Expression.Constant(optionsString[0])));

// Build the subsequent OR parts
foreach(char option in optionsString.substring(1)) {
    builtExpression = Expression.Or(
            builtExpression,
            Expression.NotEqual(
                    Expression.Constant(-1),
                    Expression.Call(
                            Expression.Property(param, "Footnotes"),
                            indexOfMethod,
                            Expression.Constant(optionsString[0]))));
}

Of course, this should be rewritten according the DRY-principle:

private Expression checkForFootnote(ParameterExpression param, char footnoteCode) {
    static MethodInfo indexOfMethod = typeof(string).GetMethod("IndexOf", newType[] { typeof(char) });

    return Expression.NotEqual(
                        Expression.Constant(-1),
                        Expression.Call(
                                Expression.Property(param, "Footnotes"),
                                indexOfMethod,
                                Expression.Constant(footnoteCode)));
}

var optionsString = "BDF";
// Build the first request
var builtExpression = checkForFootnote(param, optionsString[0]);

// Build the subsequent OR parts
foreach(char option in optionsString.substring(1)) {
    builtExpression = Expression.Or(
            builtExpression,
            checkForFootnote(param, option);
}

This gets us 90% of the way to where I need to be. The only thing we’re missing is the exclude option. In my implementation, exclude is equal to true when I want it active, and false otherwise, which means it must be negated in order to meet the requirement above. Remember in this case, the boolean expression we want is: !exclude && (footnoteCheck) where footnoteCheck is the expression constructed above.

builtExpression = Expression.And(
        builtExpression,
        Expression.Not(Expression.Constant(exclude)));

At this point, we’ve built the entire expression in a way that can be sent to LINQ for conversion into dynamically generated SQL. After this was done, I saw a dramatic increase in speed on footnote-only searches via our search page as I’m letting my SQL Server do the work it was designed to do. More than that, I’ve been able to deconstruct a method to show that nearly ANY data processing problem can be solved in LINQ, only requiring a bit more thought.

http://msdn.microsoft.com/en-us/library/system.linq.expressions.expression.aspx

Book Review: Cooking for Geeks

2011-01-26T14:00:00Z

Jeff Potter’s book, Cooking for Geeks, was published last year and received quite a bit of praise over the web. As a Geek who is into food, I was interested in picking this one up and reading through it.

My taste is cookbooks is a bit specialized. I don’t tend to care much for books that are simply collections of recipes. My favorite cookbooks have been Michael Ruhlman’s Ratio and Mark Bittman’s How to Cook Everything. But these cookbooks, which do both contain recipes, are more about the process of food. Of what differentiates classes of recipes from one another.

Cooking for Geeks falls solidly in this class of cookbooks. Yes, there are plenty of recipes, as there are in those others, but it’s much more about imparting a deep understanding of the science of food. And it’s done through interviews with known chefs and scientists and geeks, including Tim O’Reilly and Adam Savage.

Potter effectively shares the language used in the food world. In the chapter on Wine, there is significant discussion with a sommelier about the reality of wine pairings and descriptive characters that are used inside of the industry. He goes into detail on ‘molecular gastronomy’, both in the sense that it’s used today to mean ‘using chemicals in cooking’ (by which metric, the Twinkie is probably the pinnacle of food), but also in the sense of hard science that’s being done on food to better understand the reactions that take place inside of food.

I kid a bit about molecular gastronomists, and while I respect the skill of chef’s like Wylie Dufresne, there have been people practicing molecular gastronomy for far longer that aren’t chefs, and approach the problems from a purely research perspective. However, the techniques of both are discussed, and presented in ways that left me interested in them, and thinking critically about what I’m doing while cooking.

That is where Cooking for Geeks really excels. If you’re into geek, and by that I mean you eat, drink and breathe geek, and you’re also into food (or want to be), then I think you’re likely to enjoy this. It goes into enough detail to sate your desire for information, while driving it home time and again that just like whatever else you’re doing, it’s okay to experiment, and even fail. After all, like Potter points out, even if you’ve created an inedible atrocity (and believe me, I have), pizza is only a phone call away.

Links: