Past Midnight

Debugging mobile phones

2012-02-19T19:30:00.003+02:00

Last week I posted an update on the progress we are making towards a functional JavaScript debugger in Firefox. We still have a ways to go before we get there, but the foundation this is based on, namely the Remote Debugging Protocol, provides a solid basis for debugging mobile browsers, and as it turns out, even mobile operating systems. Supporting these use cases is still farther down our list, but I decided to spend a few hours the other day trying to estimate how far we are from this goal. It turns out we are not that far, and I have a screencast to prove it.

I had done some prototyping work for Firefox for Android a while back, which was rather straightforward, so I was curious to know what would it take to tackle a more ambitious project: debugging Boot to Gecko. After a few hours of hacking and orienting myself around the B2G and Gaia code base, I got this far:

Not too bad I think for a few hours work. Don't expect this to be ready of course, before we have a working debugger in desktop Firefox. If, however, you'd like to follow along and maybe even lend a hand, you are always welcome!

Debugging JavaScript

2012-02-13T10:08:00.000+02:00

tl;dr: Firefox nightlies now ship with an experimental JavaScript debugger. It's not ready for end-users yet, but we are feverishly working on filling in the missing bits.

Debugging is always hard, especially when debugging other people's code. Adding a bunch of console.log() statements, becomes tedious after a while and that's assuming that you are familiar enough with the code in question to know where to place them. Tools like Firebug are a tremendous help in this case. Having the ability to pause the execution at any point or inspect variables and stack frames, provides valuable insights into the runtime behavior of a program.

Firebug depends on the traditional JSD API in SpiderMonkey to perform its magic, but nowadays JSD is limiting in a number of ways. In order to overcome these limitations, the new JSDBG2 API was designed and implemented by Jim Blandy and Jason Orendorff, and the first version of this work landed in Firefox 8. Jim, being the kind of guy that has forgotten more about debuggers than I will ever know, designed a remote debugging protocol to separate the debugger from the browser, allowing for debugging desktop browsers, mobile browsers, mail readers and even more exotic software in the future. Dave Camp got a great protocol implementation up and running, but was assigned managerial duties (poor sap!) before having a chance to finish the job, so Victor Porof, Mihai Șucan and yours truly lined up to carry the torch. The culmination of this work landed in mozilla-central last week, including the remote protocol implementation and a prototype UI (disabled by default). It is now available in nightlies for adventurous souls to play with.

Let me be clear about that last point: what's in there right now is not useful for day-to-day work, even for experienced developers. There is no UI to add breakpoints besides the experimental Graphical Command Line Interface (whose awesomeness deserves a separate post), no stepping, no variable inspection besides call parameters and 'this', and numerous other limitations. In short: it's not ready, yet. In the course of the next weeks we will be furiously working to add the missing bits and improve the UI, so that it becomes a worthy addition to our existing suite of developer tools.

If, however, you are the kind of person who doesn't take no for an answer, here is what you need to do to play with it: type about:config in the awesomebar and set the preference devtools.debugger.enabled to true. Also, set devtools.gcli.enable to true in order to be able to set breakpoints. Restart your browser and select Tools -> Web Developer -> Script Debugger to open it. You will need to reload the page for the debugger to become aware of the scripts in the page (see? It's not ready!). Browse through the source scripts to find a place where you would like to add a breakpoint, then open the web console and type break add line <url> <line>. Don't worry about memorizing it, GCLI will guide you along the way, suggesting among the available options at each step. When the code hits the breakpoint, the debugger will display the stack frames and the variables in scope (not all of them, it's not ready, remember?) for inspection.

If you have better things to do with your time than play with experimental, uninished debuggers, but would still like to take a peek into what's in store already, here is a short screencast that should give you a broad idea. Note that the stepping functionality in the demo is not in the nightlies yet, because, you guessed it, it's not quite ready.

Ready or not, we would love to have any feedback you might have. Drop by #devtools for a chat, or file bugs to ~~torture~~help us get things fixed. Rest assured, when it's done, debugging web apps will never be the same again.

Next stop: Mozilla

2011-04-16T21:00:00.002+02:00

Six months ago I left my previous job, in order to join a product company and experience new things. The move was rewarding in many ways, but most importantly, because of the people I've met. We've accomplished a lot together and laid out the foundations for much more. As I've established before, this is not the right time to be moving on, but while in theory, theory and practice are the same, in practice, they are not. When good fortune knocks on your door, it's a fool who does not answer.

So I am seizing the opportunity to work with some of the best hackers in the world, on a product that I adore, with technologies that until now I had to stay up late at night to use. In a few days I’ll be joining Mozilla to work on Firefox Developer Tools, a passion of mine for quite some time.

I am thrilled with this opportunity to learn from the best, and have an impact on the lives of millions. And to do this with a non-profit organization, whose mission is to promote openness, innovation and opportunity on the web, for the public good.

I'm almost wishing I gave-up this long-needed vacation.

Moving On

2010-10-31T01:46:00.002+02:00

About 10 years ago, I was invited to become employee No1 of a new software startup. I didn’t know much about startups back then, besides that they were supposedly cool, and employees weren’t treated like small cogs in a giant machine. Indeed, it felt really fun to build our first product in my 5 year-old laptop, running FreeBSD, of all things. That, and getting praised by our users, too.

Things got even better as time went by: interesting projects to build, exciting technologies to explore, new colleagues to work with, new challenges to face. I felt my work was making a real impact, getting the team to follow new and exciting technological paths, often far away from the mainstream. I became partner, pushed for launching our first large consumer product, coded, mentored, planned and dreamed about where to go next. I had a 27-inch Mac, a big desk and a great view from my office window.

And then I left.

When is it a good time to leave? When everything is over, it’s obviously too late. When things are just starting to take shape, it’s definitely too soon. I used to say, ‘when you are done’. But it turns out that, in real life, you are never done. There are always more things to do, unfinished businesses to attend to, and existing plans to follow through. Perhaps a better answer is: ‘when you are too comfortable, almost on the brink of getting bored’.

A friend asked me why would I leave this company in what appears, by all accounts, to be the pinnacle of our accomplishments, ‘now that your work is being recognised and respected’, as he put it. Another said that we’re all going through some pretty hard times, and it wouldn’t be wise to take risks. A rather unsettled family member asked: ‘aren’t you afraid?’

Of course I was.

Afraid of leaving behind the tried-and-true for something new and unknown; afraid that the new gig might not live up to the initial impressions; afraid that my work-life balance might deteriorate. However, I’ve learned long ago that fear isn’t something to be afraid of. As Eleanor Roosevelt said once: ‘You gain strength, courage, and confidence by every experience in which you really stop to look fear in the face. You must do the thing which you think you cannot do.’ What I wanted to do, is to become even better at my craft. To have an impact on even more people’s lives through my work. To inspire and be inspired. To be afraid and excited again.

So, as of a few days ago I’ve taken up an offer to work for Golden Deal. It’s a product company, something I’ve always been keen on, with a great track record and an awesome crew that is rapidly expanding (are you a senior dev or admin? we’re hiring!). I’m still getting to know the place around here, but it feels good, and I think we have a shot at becoming one of the greatest Internet companies in Greece.

I know that my former teammates will continue to rock on, and I’ll be cheering from the sidelines for their future accomplishments. As for me, I just might find more time to blog again.

P.S.: For folks contemplating a career move, here are some tips for choosing your next gig. This is not how I approached my own case, since I wasn’t really on the market for a job. I was more like seduced, from some pretty persuasive folks. However, had I been looking for a new employer, this is what I would have looked out for:

Product company: this is where you have a better chance at learning the ropes of modern technologies, without doing pure academic exercises that have little to no importance for the business.
Startup mentality: you want to be a part of a group that fights as a whole for the success of the company, not a bunch of bureaucrats isolated in their silos.
Experienced team with a good track record: so you can learn from people better, smarter or wiser than you, and improve your skills.
Teams not afraid to experiment with new and unproven technologies: because innovation is the name of the game, at least for startups and product companies.
A ‘getting things done’ attitude: because success apparently only comes for the right kind of people.
Honest presentation: this is rather obvious.

Bear in mind that people have a tendency to present a rosy picture of their situation, because, if nothing else, it makes them feel better about themselves. Be a little pessimist and assume things are a tad worse, and you’ll be closer to the truth most of the time. Also, do some research on what you’re being told, don’t trust their sales pitch blindly.

I know I did.

GSS/Pithos update: iPhone, Android, Mongo & getting to 2.0

2010-10-12T22:55:00.000+02:00

It has been a while since my last post on GSS/Pithos, and there have been quite a few important announcements that I have not blogged about, only tweeted:

a shiny native iPhone client
a brand new Android client
a back-office application for statistics and admin tasks
public folders for hosting static web sites
a slew of performance, usability and bug fixes for the web & desktop clients
new user registration and quota upgrade workflows

And since the code base has been adopted by other European NRNs, we have also improved our release engineering process and documentation. This however was just the tip of the iceberg. What we are now close to completing, is nothing short of an almost complete rewrite of the core GSS server, aiming at even greater performance and scalability. We are about to release GSS 2.0.

The initial plan was far more modest. We had identified the PostgreSQL installation that stores all of the the system’s data besides the actual files, as a future bottleneck when user adoption would start to accelerate. In such an event there were a few low-to-medium-cost solutions that we could pursue, like replication and manual sharding, but since the problem seemed best suited for a NoSQL solution, we decided to bite the bullet and attempt a transition.

Our field search included lots of products, like Mongo, Couch, Cassandra, Riak, Redis, HBase, MemcacheDB and others. The basic requirements were high write throughput, sharding, replication and easy migration. Most of the products boasted high performance for writes and lots supported replication. In the end we discarded the ones that didn’t support sharding, which limited the options a lot, and we investigated the various APIs, trying to figure out what it would mean to rewrite GSS for one of these. In the end we chose Mongo for its rich query support, since it was the best fit for our problem at hand, making the transition of the existing code base easier. Although we’d love to have had the opportunity to try different ports for each of those datastores, it is most likely that we would have had to rewrite the entire server from scratch, which was not an option.

The initial plan was to try to isolate the changes to the data access layer, essentially building a new adapter for the middle tier that would appear almost identical to the existing code. This effort was fruitless however, since we could not find satisfactory solutions for various issues, like transactions, data mapping, entity lifecycle, etc. We came to the conclusion that even if we managed to build a not-too-leaky abstraction for Mongo, it would have probably killed the performance, making the switch from PostgreSQL pointless.

So we ditched JPA and container-managed transactions and decided to make a simpler POJO mapping to Mongo using Morphia, a Java data mapper for Mongo that takes care of the nitty-gritty details and repetitive coding, for only a minor performance hit. Since Mongo only supports atomicity for a single query, we reverted to manual transaction handling and decided to ditch the EJBs that formed the backbone of GSS altogether, since they didn’t buy us anything at that point. The new GSS server would be a set of servlets, service and domain objects, wired together via Google Guice. Since we could host the new server in a plain servlet container, we went from JBoss to Jetty, which vastly reduced the server startup time from 60 seconds down to 2. As a consequence, development round-trip times went down, and coding was fast again. Lots of object copying and conversions from entities to DTOs and back again were now unnecessary, since there was no more a transaction boundary at the session bean layer. This kept memory usage low, increased cache locality and boosted performance. There are lots of interesting details about our approach, but they would be better discussed in separate blog posts. If you are interested, you can already check out the code in the gss2 branch of the GSS repository.

The next generation GSS server is not production ready yet, but we'll be getting there soon, and so far it has been a great ride!

The Cloud Desktop

2010-02-19T22:23:00.006+02:00

This was originally posted in the mynetworkfolders.com blog.

We live in a browser these days. E-mail, word-processing, spreadsheets, chat, are all occupying a browser window in front of me. Maybe not for everyone yet, but the trend is clear, and has been clear for quite some time now. This is a good thing. The Web of today bears little similarity to the Web of the millennium. It's fast, it's powerful and it's ubiquitous. A consequence of that evolution is the simplicity with which I can continue my work when leaving my Linux workstation at work and sit in front of my Mac at home. Since my applications are web-based, they are OS- and browser-agnostic. More importantly, my data are always there when I need them. Readily accessible from my desktop, laptop or mobile phone. I may keep local backups for archival purposes, but my master copies are always online.

Assuming this trend is indeed inescapable, one can't help but wonder: what are the applications that have not migrated to the cloud, yet? What kind of things would I like to do online, seamlessly, from every device I own? What are the barriers that make our current online experience less powerful than the desktop one?

One thing that we noticed was missing from the online desktop, was the... desktop! Yes, my spreadsheets are online. Sure, my pictures are online. Yes, I can upload arbitrary files in various storage services. But, I couldn't find a way to use a familiar file manager interface to work with my data in the cloud, as I do on my computer. This was the need we decided to fulfill with MyNetworkFolders.

MyNetworkFolders attempts to provide an answer to the question: how would my desktop look in the cloud? And even though we are nowhere near done yet, the results so far are exciting and very promising.

MyNetworkFolders offers a familiar file manager interface for my cloud storage needs, with advanced functionality and ease of use. Documents can be viewed, modified, searched or sent to the trash. They can be shared with other users or among the members of a group using flexible ACLs, or even be publicly accessible from the whole internet. They can be versioned for tracking changes more effectively. The service can be accessed from a variety of ways, including a web browser, a desktop client, a WebDAV network share or (really soon) a mobile phone. But most importantly, there is no vendor lock-in. There is a REST-like API available, battle-tested from all the above client implementations, giving everyone the opportunity to access his data in his own particular way. Or even get them out of our service altogether.

Oh, and one other thing. It's all based on open-source software. The core infrastructure, the various clients we have built, even API helpers in various programming languages, are all available for anyone and everyone to see, use and modify. Or maybe used to setup and run YourNetworkFolders, if you feel really ambitious.

If however you just want a true desktop environment, with unparalleled flexibility, and maybe also see how far the cloud desktop rabbit-hole goes, then come along. It's going to be an exciting ride.

Lessons from Startup Weekend

2009-12-07T23:50:00.002+02:00

I had an exhausting but fun weekend at the Athens Startup Weekend a few days ago. Along with Christos I joined Yannis Kekatos, Panagiotis Christakos and Babis Makrinikolas on the Newspeek project. When Yannis pitched the idea on Friday night, the main concept was to create a mobile phone application that would provide a better way to view news on the go. I don't believe it was very clear in his mind then, what would constitute a "better" experience, but after some chatting about it we all defined a few key aspects, which we refined later with lots of useful feedback and help from George. Surprisingly, for me at least, in only two days we managed to design, build and present a working prototype in front of the judges and the other teams. And even though the demo wasn't exactly on par with our accomplishments, I'm still amazed at what can be created in such a short time frame.

Newspeek, our product, had a server-side component that periodically collected news items from various news feeds, stored them and provided them to clients through a simple REST API. It also had an iPhone client that fetched the news items and presented them to the user in a way that respected the UI requirements and established UX norms for that device.

So, in the interest of informing future participants about what works and what doesn't work in Startup Weekend, here are the lessons I learned:

If you plan to win, work on the business aspect, not on the technology. Personally, I didn't go to ASW with plans to create a startup, so I didn't care that much about winning. I mostly considered the event as a hackathon, and tried my best to end up with a working prototype. Other teams focused more on the business side of things, which is understandable, given the prize. Investors fund teams that have a good chance to return a profit, not the ones with cool technology and (mostly) working demos. Still, the small number of actual working prototypes was a disappointment for me. Even though the developers were the majority in the event, you obviously can't have too many of them in a Startup Weekend.
For quick server-side prototyping and hosting, Google App Engine is your friend. Since everyone in the team had Java experience, we could have gone with a JavaEE solution and set up a dedicated server to host the site. But, since I've always wanted to try App Engine for Java and the service architecture mapped nicely to it, we tried a short experiment to see if it could fly. We built a stub service in just a few minutes, so we decided it was well worth it. Building our RESTful service was really fast, scalability was never a concern and the deployment solution was a godsend, since the hosting service provided for free by the event sponsors was evidently overloaded. We're definitely going to use it again for other projects.
jQTouch rocks! Since our main deliverable would be an iPhone application, and there were only two of us who had ever built an iPhone application (of the Hello World variety), we knew we had a problem. Fortunately, I had followed the jQTouch development from a reasonable distance and had witnessed the good things people had to say, so I pitched the idea of a web application to the team and it was well received. iPhone applications built with web technologies and jQTouch can be almost indistinguishable from native ones. We all had some experience in building web applications, so the prospect of having a working prototype in only two days seemed within the realm of possibility again. The future option of packaging the application with PhoneGap and selling it in the App Store was also a bonus point for our modest business plan.
For ad-hoc collaboration, Mercurial wins. Without time to set up central repositories, a DVCS was the obvious choice, and Mercurial has both bundles and a standalone server that make collaborative coding a breeze. If we had zeroconf/bonjour set up in all of our laptops, we would have used the zeroconf extension for dead easy machine lookup, but even without it things worked flawlessly.
You can write code with a netbook. Since I haven't owned a laptop for the last three years, my only portable computer is an Asus EEE PC 901 running Linux. Its original purpose was to allow me to browse the web from the comfort of my couch. Lately however, I'm finding myself using it to write software more than anything else. During the Startup Weekend it had constantly open Eclipse (for server-side code), Firefox (for JavaScript debugging), Chrome (for webkit rendering), gedit (for client-side code) and a terminal, without breaking a sweat.
When demoing an iPhone application, whatever you do, don't sweat. Half-way through our presentation, tapping the buttons didn't work reliably all the time, so anxiety ensued. Since we couldn't make a proper presentation due to a missing cable, we opted for a live demo, wherein Yannis held the mic and made the presentation, and I posed as the bimbo that holds the product and clicks around. After a while we ended up both touching the screen, trying to make the bloody buttons click, which ensured the opposite effect. In retrospect, using a cloth occasionally would have made for a smoother demo, plus we could have slipped a joke in there, to keep the spirit up.

All in all it was an awesome experience, where we learned how far we can stretch ourselves, made new friends and caught up with old ones. Next year, I'll make sure I have a napkin, too.

Spell-checking in JavaScript

2009-08-20T14:06:00.002+03:00

I just heard of Atwood's Law a few days ago. Apparently Jeff Atwood first published his discovery a couple of years ago, which is like a century in Internet time, but I can't keep up with everything. The Law states that "any application that can be written in JavaScript, will eventually be written in JavaScript". As Atwood explains, it is based on Tim Berners-Lee's Principle Of Least Power:

Computer Science spent the last forty years making languages which were as powerful as possible. Nowadays we have to appreciate the reasons for picking not the most powerful solution but the least powerful. The less powerful the language, the more you can do with the data stored in that language.

I think there is a common theme between Atwood's Law, Zawinski's Law ("Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can") and Greenspun's Tenth Rule ("Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp"). The inevitability of the specified outcome makes the world seem like a much simpler place. There is no need to argue. Believe and ye shall be redeemed.

I might have been in this state of mind, when I came across Peter Norvig's "How to Write a Spelling Corrector", because I knew instantly I had to port the algorithm to JavaScript. The algorithm is quite simple and has already been ported to many different languages, so I seized the opportunity to study the differences in expressiveness, performance and style, between these languages and my latest affection, JavaScript.

After a few night's work I have it working and free (as in beer and speech) for anyone to use. Check out the speller project on GitHub. If you want to understand how the algorithm works, you should go and read Norvig's article, although you might get some hints from the comments in my code. The modestly condensed version of the code is 53 lines:


var speller = {};
speller.train = function (text) {
  var m;
  while ((m = /[a-z]+/g.exec(text.toLowerCase()))) {
    speller.nWords[m[0]] = speller.nWords.hasOwnProperty(m[0]) ? speller.nWords[m[0]] + 1 : 1;
  }
};
speller.correct = function (word) {
  if (speller.nWords.hasOwnProperty(word)) return word;
  var candidates = {}, list = speller.edits(word);
  list.forEach(function (edit) {
    if (speller.nWords.hasOwnProperty(edit)) candidates[speller.nWords[edit]] = edit;
  });
  if (speller.countKeys(candidates) > 0) return candidates[speller.max(candidates)];
  list.forEach(function (edit) {
    speller.edits(edit).forEach(function (w) {
      if (speller.nWords.hasOwnProperty(w)) candidates[speller.nWords[w]] = w;
    });
  });
  return speller.countKeys(candidates) > 0 ? candidates[speller.max(candidates)] : word;
};
speller.nWords = {};
speller.countKeys = function (object) {
  var attr, count = 0;
  for (attr in object)
    if (object.hasOwnProperty(attr))
      count++;
  return count;
};
speller.max = function (candidates) {
  var candidate, arr = [];
  for (candidate in candidates)
    if (candidates.hasOwnProperty(candidate))
      arr.push(candidate);
  return Math.max.apply(null, arr);
};
speller.letters = "abcdefghijklmnopqrstuvwxyz".split("");
speller.edits = function (word) {
  var i, results = [];
  for (i=0; i < word.length; i++)
    results.push(word.slice(0, i) + word.slice(i+1));
  for (i=0; i < word.length-1; i++)
    results.push(word.slice(0, i) + word.slice(i+1, i+2) + word.slice(i, i+1) + word.slice(i+2));
  for (i=0; i < word.length; i++)
    speller.letters.forEach(function (l) {
      results.push(word.slice(0, i) + l + word.slice(i+1));
    });
  for (i=0; i <= word.length; i++)
    speller.letters.forEach(function (l) {
      results.push(word.slice(0, i) + l + word.slice(i));
    });
  return results;
};

It may not be as succinct as Norvig's Python version (21 lines), or the record-holding Awk and F# versions (15 lines), but is much better than C (184 lines), C++, Perl, PHP, Rebol and Erlang. There is even a Java version with 372 lines. It must contain some sort of spell-checking framework in there, or something. The condensed version above, although correct, has terrible performance in most JavaScript engines, however. For real-world use you should pick the regular version which may be slightly longer, but performs much better.

Since this was a toy project of mine, I wanted to play with ServerJS modules as well, in order to run it as a shell script. I turned the code into a securable module, so you can run it from the command line, using narwhal. I have a couple of scripts to that end. Of course since this is JavaScript, you can try it from your browser, by visiting the demo page, courtesy of GitHub Pages. Be sure to use a modern browser, like Firefox 3.5, Safari 4 or Chrome 3 (beta), otherwise you won't be able to run the test suite, since I used the brand-new Web Workers to make the long-running tasks run in the background.

Norvig's Python implementation took 16 seconds for test 1 and the best I got was 25 seconds with Safari 4 on my Mac. Narwhal is using Rhino by default, so it is definitely not competitive in such tests (139 seconds), but I'm planning to fix support for v8cgi and give that a try.

And it goes without saying that I'd love to hear about ways to improve the performance or the conciseness of the code. If you have any ideas, don't be shy, leave a comment or even better fork the code and send me a pull request on GitHub.

GSS authentication

2009-07-07T16:23:00.001+03:00

As I've described before, the GSS API is a REST-like interface for interacting with a GSS-based service, like Pithos. Using regular HTTP commands a client can upload, download, browse and modify files and folders stored in the GSS server. These interactions have two important properties: they store no client state to the server and they are not encrypted. I have already mentioned the most important benefits from the stateless architecture of the GSS protocol, namely scalability and loose coupling along the client-server boundary. In the same line of thought, SSL/TLS encryption of the transport was avoided for scalability reasons. Consequently, these two properties of the communication protocol, lead to another requirement: message authentication for each API call.

Since no session state is stored in the server, the client must authenticate each request as if it were the first one. Traditional web applications use an initial authentication interaction between client and server, that creates a unique session ID, which is transmitted with each subsequent request in that same session. While using this ID, the client does not need to authenticate again to the server. Stateless protocols, like WebDAV for instance, cannot rely on such an ID and have to transmit authentication data in each call. Ultimately the tradeoff is a minor increase in the message payload, in return for a big boost in scalability. The HTTP specification defines an Authorization header for use in such cases and WebDAV uses the HTTP Digest Access Authentication scheme. The GSS API uses a slight variation in that theme, blended with some ideas from OAuth request signing.

Essentially the standard HTTP Authorization header is populated with a concatenation of the username (for identifying the user making the request) and a HMAC-SHA1 signature of the request. The request signature is obtained by applying a secret authentication token to a concatenated string of the HTTP method name, the date and time of the request and the actual requested path. The GSS API page has all the details. The inclusion of the date and time helps thwart replay attacks using a previously sniffed signature. Furthermore, a separate header with timestamp information, X-GSS-Date, is used to thwart replay attacks with a full copy of the payload. The authentication token is issued securely by the server to the client and is the time-limited secret that is shared between the client and server. Since it is not the user password that is used as a shared secret, were the authentication token to be compromised, any ill effects would have been limited to the time period the token was valid. There are two ways for client applications to obtain the user's authentication token, and they are both described in detail in the API documentation.

In my experience, protocol descriptions are one thing and working code is a totally different one. In that spirit, I'm closing this post with a few tested implementations of the GSS API signature calculation, for client applications in different languages. Here is the method for signing a GSS API request in Java (full code here):


public static String sign(String httpMethod, String timestamp,
        String path, String token) {
  String input = httpMethod + timestamp + path;
  String signed = null;

  try {
    System.err.println("Token:" + token);
    // Get an HMAC-SHA1 key from the authentication token.
    System.err.println("Input: " + input);
    SecretKeySpec signingKey = new SecretKeySpec(
        Base64.decodeBase64(token.getBytes()), "HmacSHA1");

    // Get an HMAC-SHA1 Mac instance and initialize with the signing key.
    Mac hmac = Mac.getInstance("HmacSHA1");
    hmac.init(signingKey);

    // Compute the HMAC on the input data bytes.
    byte[] rawMac = hmac.doFinal(input.getBytes());

    // Do base 64 encoding.
    signed = new String(Base64.encodeBase64(rawMac), "US-ASCII");

  } catch (InvalidKeyException ikex) {
    System.err.println("Fatal key exception: " + ikex.getMessage());
    ikex.printStackTrace();
  } catch (UnsupportedEncodingException ueex) {
    System.err.println("Fatal encoding exception: "
                + ueex.getMessage());
  } catch (NoSuchAlgorithmException nsaex) {
    System.err.println("Fatal algorithm exception: "
                + nsaex.getMessage());
    nsaex.printStackTrace();
  }

  if (signed == null)
    System.exit(-1);
  System.err.println("Signed: " + signed);
  return signed;
}

Here is a method for signing GSS API requests in JavaScript, that uses the SHA-1 JavaScript implementation by Paul Johnston (full code here):


function sign(method, time, resource, token) {
  var q = resource.indexOf('?');
  var res = q == -1? resource: resource.substring(0, q);
  var data = method + time + res;
  // Use strict RFC compliance
  b64pad = "=";
  return b64_hmac_sha1(atob(token), data);
}

Here is a Tcl implementation by Alexios Zavras (full code here):


proc ::rest::_sign {id} {
  variable $id
  upvar 0 $id data
  set str2sign ""
  append str2sign $data(method)
  append str2sign $data(date)
  set idx [string first ? $data(path)]
  if {$idx < 0} {
    set str $data(path)
  } else {
    incr idx -1
    set str [string range $data(path) 0 $idx]
  }
  # append str2sign [::util::url::encode $str]
  append str2sign $str
  puts "SIGN $str2sign"
  set sig [::base64::encode [::sha1::hmac -bin -key $::GG(gsstoken) $str2sign]]
  set data(signature) $sig
}

I'd love to show implementations of the signature calculation for other languages as well, but since my time is scarce these days, I could use some help here. If you've written one yourself and you'd like to share, leave a comment and I'll update this post, with proper attribution of course.

Retrying transactions with exponential backoff

2009-06-17T19:25:00.004+03:00

The method I described in my previous post about retrying transactions could use some improvement. A deficiency that will only become relevant in highly congested servers is the constant retry interval. When two or more transactions try to commit at the same time and fail, with the code from the last post they will retry probably simultaneously again. Random runtime events (like process/thread scheduler decisions, JIT compiler invocations, etc.) might help avoid collisions, but in general the collided transactions may well collide again. And again. And then again, until the specified maximum number of retries is reached.

The general method to alleviate such problems is to randomize the retry intervals. The most well-known algorithm in this category is called exponential backoff. This is one way to implement it for the utility method in my previous post (full code in gss):


public T tryExecute(final Callable<T> command) throws Exception {
  T returnValue = null;
  // Schedule a Future task to call the command after delay milliseconds.
  int delay = 0;
  ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);
  for (int i = 0; i < TRANSACTION_RETRIES; i++) {
    final int retry = i;
    ScheduledFuture<T> future = executor.schedule(new Callable<T>() {

      @Override
      public T call() throws Exception {
        return command.call();
      }
    }, delay, TimeUnit.MILLISECONDS);

    try {
      returnValue = future.get();
      break;
    } catch (ExecutionException e) {
      Throwable cause = e.getCause();
      if (!(cause instanceof EJBTransactionRolledbackException) ||
          retry == TRANSACTION_RETRIES - 1) {
        executor.shutdownNow();
        throw new Exception(cause);
      }
      delay = MIN_TIMEOUT + (int) (MIN_TIMEOUT * Math.random() * (i + 1));
      String origCause = cause.getCause() == null ?
          cause.getClass().getName() :
          cause.getCause().getClass().getName();
      logger.info("Transaction retry #" + (i+1) + " scheduled in " + delay +
          " msec due to " + origCause);
    }

  }
  executor.shutdownNow();
  return returnValue;
}

The notable changes in this version are the delayed invocations of the Callable and the retry interval calculation. The former is accomplished using a ScheduledFuture from the excellent utilities in java.util.concurrent, which gets executed after a random time period. The interval calculation is something that can be implemented in a number of ways, either with monotonically increasing intervals or not. I opted for the formula above since it provided the fastest collision elimination in my tests, much faster than the monotonically increasing interval formulas I tried. The MIN_TIMEOUT constant is more of a black art. It should be tuned to the particular hardware and network setup in order to attain maximum efficiency: the minimum number of retries with the minimum interval between retries.

Another issue raised in a comment in my previous post was that the exception I am using as guard, EJBTransactionRolledbackException, may be too generic for this purpose. This is definitely true, EJBTransactionRolledbackException just wraps an OptimisticLockException as one would expect for this case, which in turn wraps the Hibernate-specific StaleObjectStateException that is thrown initially. However, contention in a database table, index or row might not necessarily result in optimistic locking violations, but in deadlocks as well, when one transaction holds exclusive locks on resources that are needed by the other and vice versa. Deadlock detection is performed by the DBMS, which forces a rollback of both transactions. This time however the initial exception (at least in my testing, might be DBMS or JDBC driver-specific) is GenericJDBCException which gets wrapped in a PersistenceException, which in turn is put inside a EJBTransactionRolledbackException before being received by our call site. Therefore, the generic nature of the guard is not a problem in this case. On the contrary it covers more relevant cases and one might argue even that there is no better service that can be offered to the caller, than retrying in all such occasions.

Retrying transactions in Java

2009-06-15T16:18:00.011+03:00

When building a web-based system, the most often encountered persistence solution is the use of a relational DBMS. This provides many benefits, but probably the most important one is that its behavior and usage are widely understood after so many decades of use. The simplicity of the ACID transactional model, coupled with the nice fit of the HTTP request/response protocol, make for a killer combination.

The simplicity starts to break down however, when the load on the DBMS server increases, causing transactions to start failing. The DBMS can guarantee that no data corruption will occur, but handling such cases is not that simple. The scalable Optimistic Locking model requires transactions to be resubmitted when failure occurs. Since this is supposed to be rather rare (otherwise optimistic locking should probably not be used), developers often cheat by not dealing with such errors, letting them bubble up to the user, who is then responsible for resubmitting the transaction. This leads to simple, elegant codebases and unhappy users.

Adding ORM to the mix only makes things worse. In JavaEE, the JPA API hides much of the complexity of dealing with relational databases, but corner cases become harder. Add the container-managed transaction boundaries provided by session EJBs and one might be tempted to raise his hands up in despair.

However a solution is not as hard as it sounds. Provided that session EJB methods are sufficiently free from side effects, one can replay a transaction for a number of times from the call site (external to the session bean) with a small amount of code and a small added complexity to the call sites. Since I seem to come up against this requirement quite often, I thought I should describe one way to do it. The following tryExecute() helper method, retries any action supplied in a standard Callable interface for a number of times, in the face of optimistic locking violations:


public T tryExecute(Callable<T> command) throws Exception {
  T returnValue = null;
  for (int i = 0; i < TRANSACTION_RETRIES; i++) {
    try {
      returnValue = command.call();
      break;
    } catch(EJBTransactionRolledbackException trbe) {
      if (i == TRANSACTION_RETRIES - 1)
        throw trbe;
      } catch (Exception e) {
        throw e;
      }
      logger.debug("Transaction retry: " + (i+1));
    }
  return returnValue;
}

By abstracting away the session bean method code in a parameterized class we can reuse this method in every place that needs to retry transactions. Using a Callable as a parameter lets us return any values received from the session bean. For supplying bean methods that do not return values, we could create Runnable instances and wrap them to Callable using Executors.callable(). You can see the original code here.

If you use the above code in a JavaSE environment without EJB, you should probably need to catch OptimisticLockException for JPA, or even StaleObjectStateException, if you don't mind depending on a Hibernate API. I use the above helper method like this in my code:


file = new TransactionHelper<FileHeaderDTO>().tryExecute(new Callable<FileHeaderDTO>() {
  @Override
  public FileHeaderDTO call() throws Exception {
    return sessionBean.createFile(user.getId(), folder.getId(), name, mimeType, uploadedFile);
  }
});

Java's verbosity makes this not that easy to understand at first sight, but what happens is very simple indeed: I wrap a one-liner call to the session bean method createFile(), in a Callable and supply it to tryExecute(). The result from the bean method is received in a local variable as usual. One thing to keep in mind though is that since Java doesn't have closures, the above anonymous function can only receive final variables as parameters. In my example user, folder, name, mimeType and uploadedFile are all final. You might have to resort to using temporary local variables for storing final values on some occasions.

Ubuntu on EEE PC

2009-06-04T12:19:00.000+03:00

A few weeks ago I set out to upgrade Ubuntu on my EEE PC 901 from Intrepid to Jaunty. It was an, um, interesting experience, so I'm jotting down these notes for next time. Upgrading from Update Manager was not an option, because my root partition was not big enough to host the upgrade. The 901 has a 4GB SSD that I'm using as root and an 8GB (or 12GB) SSD that I've put /home on. It turns out that 4GB is OK for running a modern operating system, but a little short for updating it.

The usual procedure for upgrading is to use the official ISO images to boot from. As always you need a USB stick, or an external optical drive. I settled for the former, as you can see below.

The good news with Jaunty is that a special netbook edition is available for download. With previous editions you had to start with the desktop edition and do lots of tweaking, to arrive at a satisfactory result. With the netbook edition however things were unbelievably smooth. Everything worked out of the box with my EEE PC. Well, almost everything, but I will get to that in a while.

The first step after downloading the ISO (for the security paranoid at least) is verifying the download:


$ md5sum ubuntu-9.04-netbook-remix-i386.img
8f921e001aebc3e98e8e8e7d29ee1dd4  ubuntu-9.04-netbook-remix-i386.img

Then, we can write the image to the disk. I used Ubuntu ImageWriter (package usb-imagewriter) from another Ubuntu PC:

That went very smooth. Next step was booting the EEE PC with the newly formatted USB disk. Hitting ESC on boot, allows one to change the boot drive sequence. If this is the first time you are replacing the original OS, you probably need to disable Boot Booster from the BIOS first.

The installation process is quite simple. The only important thing was partitioning in my case, since I wanted to preserve my /home partition intact. Last time I installed Ubuntu I went with ext2 filesystems, trying to squeeze as much life from my solid state disks as I could. After doing some more research on the subject, I decided that I'll probably throw away my EEE long before the disks begin to wear out. Either that, or I'll throw an SD card in the expansion slot. Life is full of choices, particularly on the $300 range.

So I selected manual partitioning and chose ext3 for my first SSD. This is a screenshot from my first attempt. No, actually it was my second attempt. I screwed up the first one. Turns out I screwed up this one, too. See if you can spot why.

It may not be that obvious, but swap should not be on the first disk. If you are using swap at all (and I know many suggest that you don't), it would be a better choice to put it on the second disk. With 1GB of RAM, swap should be at least 1GB, which leaves only 3GB for Linux. The default installation needs something north of 2GB, so that leaves less than a gig to install software you use. Unless of course you are planning to play symlink tricks, in which case I should probably offer my condolences. Anyway, I ended up chopping 1GB with gparted from the second disk and redoing the installation again, but fortunately you won't have to.

What you should definitely do however, is put aside 16MB of space in the first disk for enabling Boot Booster in the BIOS again. This is an ASUS feature that persists the BIOS configuration on the disk, shaving a couple of seconds on each boot. It may not seem much, but think about how many times you are going to boot this thing. If you add them up, it starts to matter. Boot Booster needs only 8MB of disk to function, but I found that the installer's partition editor would only allocate one cylinder for 8MB, while we need two. Using 16MB did the right thing.

The regular customization I always do is to put /tmp and /var/tmp on tmpfs, so that many common operations don't touch the disks at all. Put the following lines in fstab to do that:


tmpfs        /tmp        tmpfs    defaults,noexec,nosuid        0       0
tmpfs        /var/tmp    tmpfs    defaults,noexec,nosuid        0       0

After editing /etc/fstab the proper way to remount these file systems is by following these steps:

Log out of the Ubuntu desktop in order to minimize the number of open temp files.
Press CTRL-ALT-F1 on your keyboard. You should see a login prompt. Login with your usual user/pass.
Clear out the files in the existing /tmp and /var/tmp directories.
sudo rm -rf /tmp/* /var/tmp/*
Mount the filesystems using tmpfs.
sudo mount /tmp
sudo mount /var/tmp
Reboot the EEE PC.
sudo shutdown -r now

Next, from the boot-as-fast-as-you-can department comes another tweak. In /boot/grub/menu.lst set "timeout 0" in order to skip the kernel selection delay. If I ever need to boot a different kernel, I'll change it back to something larger, thankyouverymuch.

Log out of the Ubuntu desktop. (Yes.. I said log out. This will go a long way to ensuring the fewest number of temp files are currently open.)
Press CTRL-ALT-F1 on your keyboard. You should see a login prompt. Login with your usual user/pass.
Clear out the files in the existing /tmp and /var/tmp directories.
sudo rm -rf /tmp/* /var/tmp/*
Mount the filesystems using tmpfs.
sudo mount /tmp
sudo mount /var/tmp
Reboot the EEE PC.
sudo shutdown -r now

Now in order to enable back Boot Booster follow these instructions for configuring the small partition we created earlier. When you reboot make sure to enable the option in the BIOS (it will be hidden when the proper partition isn't found).

Now you are all set. At least if you have no use for WPA-secured WiFi. Because if you do, you'll hit a bug with the included rt2860 driver. The details aren't that interesting, but the available options are two: either use the array.org kernel or patch the driver. The former solution used to be necessary with Intrepid, since lots of things were not working with the default kernel. However after trying out both, I don't see much need for Adam's kernel in Jaunty, unless you are terrified when encountering patch, make and their ilk. Since I'm on first name basis with them, I went with the second option, hoping that a future kernel update will bring along the fixes.

All in all, the process this time around was very easy. Almost easy peasy. My EEE PC with Ubuntu Netbook Remix makes me almost forget that I'm not on my beloved Mac.

Almost.

An introduction to the GSS API

2009-04-23T12:10:00.008+02:00

Application Programming Interface or API design is one of my favorite topics in programming, probably because it is both a science and an art. It is a science because there are widely accepted principles about how to design an API, but at the same time applying these principles within the constraints of a given programming language, requires the finesse of an experienced practicioner. Therefore it is with great pleasure that I'll try to explain the ins and outs of the GSS REST-like API, as promised before. As I've already mentioned, GSS is both the name of the source code project in Google Code, as well as the GRNET-sponsored service for the Greek research and academic network (although, it's official name after leaving the beta stage will be Pithos). Since anyone can use the open-source code to setup a GSS service, in this post I'll use generic examples, so anyone writing a client for the GRNET service should modify them accordingly.

When developing an application in a particular programming language, we are used to thinking about the APIs presented to us by the various libraries, which are invariably specified in that same language. For instance, for communicating with an HTTP server from a Java program we might use the HttpClient library API. This library presents a set of Java classes, interfaces and methods for interacting with web servers. These classes hide the underlying complexity of making the low-level HTTP protocol operations, allowing our mental process to remain constantly in a Java world. We could however interact with a web server without such a library, opting to implement the HTTP protocol interactions ourselves instead. Unfortunately, there is no such higher-level library for GSS yet, wrapping the low-level HTTP communications. Therefore this post will present a low-level API, in the sense that one has to make direct HTTP calls in his chosen programming language. The good news is that the following discussion is useful for programmers with any background, since there is support for the ubiquitus HTTP protocol in every modern programming language.

A RESTful API models its entities as resources. Resources are identified by Uniform Resource Identifiers, or URIs. There are four kinds of resources in GSS: files, folders, users & groups. These resources have a number of properties that contain various attributes. The API models these entities and their properties in the JSON format. There is also a fifth entity that is not modeled as a resource, but is important enough to warrant special mention: permissions.

Users · Users are the entities that represent the actual users of the system. They are used to login to the service and separate namespaces of files and folders. User entities have attributes like full name, e-mail, username, authentication token, creation/modification times, groups, etc. The URI of a user with username paul would be:

http://host.domain/gss/rest/paul/

The JSON representation of this user would be something like this:

{
    "name": "Paul Smith",
    "username": "paul",
    "email": "paul@gmail.com",
    "files": "http://hostname.domain/gss/rest/paul/files",
    "trash": "http://hostname.domain/gss/rest/paul/trash",
    "shared": "http://hostname.domain/gss/rest/paul/shared",
    "others": "http://hostname.domain/gss/rest/paul/others",
    "tags": "http://hostname.domain/gss/rest/paul/tags",
    "groups": "http://hostname.domain/gss/rest/paul/groups",
    "creationDate": 1223372769275,
    "modificationDate": 1223372769275,
    "quota": {
        "totalFiles": 7,
        "totalBytes": 429330,
        "bytesRemaining": 10736988910
    }
}

Groups · Groups are entities used to organize users for easier sharing of files and folders among peers. They can be used to facilitate sharing files to multiple users at once. Groups belong to the user who created them and cannot be shared. The URI of a group named work created by the user with username paul would be:

http://host.domain/gss/rest/paul/groups/work

The JSON representation of this group would be something like this:


[
    "http://hostname.domain/gss/rest/paul/groups/work/tom",
    "http://hostname.domain/gss/rest/paul/groups/work/jim",
    "http://hostname.domain/gss/rest/paul/groups/work/mary"
]

Files · Files are the most basic resources in GSS. They represent actual operating system files from the client's computer that have been augmented with extra metadata for storage, retrieval and sharing purposes. Familiar metadata from modern file systems are also maintained in GSS, like file name, creation/modification times, creator, modifier, tags, permissions, etc. Furthermore, files can be versioned in GSS. Updating versioned files retains the previous versions, while updating an unversioned file replaces irrevocably the old file contents. The URI of a file named doc.txt located in the root folder of the user with username paul would be:

http://host.domain/gss/rest/paul/files/doc.txt

The JSON representation of the metadata in this file would be something like this:


{
   "name": "doc.txt",
   "creationDate": 1232449958563,
   "createdBy": "paul",
   "readForAll": true,
   "modifiedBy": "paul",
   "owner": "paul",
   "modificationDate": 1232449944444,
   "deleted": false,
   "versioned": true,
   "version": 1,
   "size": 802,
   "content": "text/plain",
   "uri": "http://hostname.domain/gss/rest/paul/files/doc.txt",
   "folder": {
       "uri": "http://hostname/gss/rest/aaitest@uth.gr/files/",
       "name": "Paul Smith"
   },
   "path": "/",
   "tags": [
       "work",
       "personal"
   ],
   "permissions": [
       {
           "modifyACL": true,
           "write": true,
           "read": true,
           "user": "paul"
       },
       {
           "modifyACL": false,
           "write": true,
           "read": true,
           "group": "work"
       }
   ]
}

Folders · Folders are resources that are used for grouping files. They represent the file system concept of folders or directories and can be used to mirror a client's computer file system on GSS. Familiar metadata from modern file systems are also maintained in GSS, like folder name, creation/modification times, creator, modifier, permissions, etc. The URI of a folder named documents located in the root folder of the user with username paul would be:

http://host.domain/gss/rest/paul/files/documents

The JSON representation of this folder would be something like this:


{
   "name": "documents",
   "owner": "paul",
   "deleted": false,
   "createdBy": "paul",
   "creationDate": 1223372795825,
   "modifiedBy": "paul",
   "modificationDate": 1223372795825,
   "parent": {
       "uri": "http://hostname.domain/gss/rest/paul/files/",
       "name": "Paul Smith"
   },
   "files": [
       {
           "name": "notes.txt",
           "owner": "paul",
           "creationDate": 1233758218866,
           "deleted":false,
           "size":4567,
           "content": "text/plain",
           "version": 1,
           "uri": "http://hostname.domain/gss/rest/paul/files/documents/notes.txt",
           "folder": {
               "uri": "http://hostname.domain/gss/rest/paul/files/documents/",
               "name": "documents"
           },
           "path": "/documents/"
       }
   ],
   "folders": [],
   "permissions": [
       {
           "modifyACL": true,
           "write": true,
           "read": true,
           "user": "paul"
       },
       {
           "modifyACL": false,
           "write": true,
           "read": true,
           "group": "work"
       }
   ]
}

Working with these resources is accomplished by sending HTTP protocol requests to the resource URI with GET, HEAD, DELETE, POST, PUT methods. GET requests retrieve the resource representation, either the file contents, or the JSON representations for the resources specified above. HEAD requests for files return just the metadata of the file and DELETE requests remove the resource from the system. PUT requests upload files to the system from the client, while POST requests perform various modifications to the resources, like renaming, moving, copying, moving files to the trash, restoring them from the trash, creating folders and more. The operations are numerous and I hope to cover them in more detail in a future post.

One important aspect of every RESTful API is the use of URIs to allow the client to maintain a stateful conversation. For example, fetching the user URI would provide the files URI for fetching the available files and folders. Fetching the files URI would in turn return the URIs for the particular files and folders contained in the root folder (along with other folder properties). Returning to the parent of the current folder would entail following the URI contained in the parent property. This mechanism removes the state handling from the server and puts the burden on the client, providing excellent scalability for the service. Furthermore, since the URIs are treated opaquely by the client, the API allows client reuse across server deployments. A client can target multiple GSS services, as long as they speak the same RESTful API. Moreover, links from service A can refer to resources in service B without a problem (in the same authentication domain, e.g. the same Shibboleth federation). This is the same as using a single web browser to communicate with multiple web servers, by following links among them.

Reconciling Apache Commons Configuration with JBoss 5

2009-04-20T21:01:00.008+02:00

There are many ways to configure a JavaEE application. Among the available solutions are DBMS tables, JNDI, JMX MBeans and even plain old files in a variety of formats. While we have used most of the above in various occasions, I find that plain files resonate with all types of sysadmins, when no other administrative interface is available. For such scenarios, Apache Commons Configuration is undoubtedly the best tool for the job. Recently, I came across an undocumented incompatibility when using Commons Configuration with JBoss 5 and I thought I should describe our solution for the benefit of others.

Usually we are storing our configuration files in the standard place for JBoss, which is JBOSS_HOME/server/default/conf, for the default server configuration. This has the disadvantage that is not as easy to remember as /etc in UNIX/Linux or \Program Files and \Windows in Windows systems, but it has the important advantage of being specified as a relative path in our code, making it more cross-platform without cluttering it with platform-specific if/else path resolution checks.

Commons Configuration can reload the configuration files automatically when changed in the file system, which helps prolong the server uptime. However JBoss 5 has introduced the concept of a virtual file system that caches all file system accesses performed through the context class loaders using relative paths. Unfortunately this generates resource URLs in the form vfsfile:foo.properties that Commons Configuration does not know how to deal with. Fixing this requires extending FileChangedReloadingStrategy, like we do in gss. Alternatively, one could patch Commons Configuration with the following change and use the standard FileChangedReloadingStrategy unchanged:


Index: FileChangedReloadingStrategy.java
===================================================================
--- FileChangedReloadingStrategy.java    (revision 764760)
+++ FileChangedReloadingStrategy.java    (working copy)
@@ -46,6 +46,9 @@
  /** Constant for the jar URL protocol.*/
  private static final String JAR_PROTOCOL = "jar";

+    /** Constant for the JBoss MC VFSFile URL protocol.*/
+    private static final String VFSFILE_PROTOCOL = "vfsfile";
+
  /** Constant for the default refresh delay.*/
  private static final int DEFAULT_REFRESH_DELAY = 5000;

@@ -161,7 +164,8 @@

  /**
   * Helper method for transforming a URL into a file object. This method
-     * handles file: and jar: URLs.
+     * handles file: and jar: URLs, as well as JBoss VFS-specific vfsfile:
+     * URLs.
   *
   * @param url the URL to be converted
   * @return the resulting file or null 
@@ -181,6 +185,18 @@
              return null;
          }
      }
+        else if (VFSFILE_PROTOCOL.equals(url.getProtocol()))
+        {
+            String path = url.getPath();
+            try
+            {
+                return ConfigurationUtils.fileFromURL(new URL("file:" + path));
+            }
+            catch (MalformedURLException mex)
+            {
+                return null;
+            }
+        }
      else
      {
          return ConfigurationUtils.fileFromURL(url);

Neither of these solutions covers the case of storing configuration files in zip or jar containers, but since it is something I haven't found a use for yet, I can't test a fix for it. If anyone is interested in such a use case, I'd advise extending FileChangedReloadingStrategy, combining the logic in jar: and vfsfile: URL handling.

GSS architecture

2009-04-13T22:38:00.006+02:00

Handling a large number of concurrent connections requires many servers. Not only because scaling vertically (throwing bigger hardware at the problem) is very costly, but also because even if an application can be designed to scale vertically, the underlying stack probably can not. Java applications for instance, like GSS, run on the JVM and although the latter is an excellent piece of engineering, using huge amounts of heap is not something it's tuned for. Big Iron servers with many cores and 20+ GB of RAM are usually running more than one JVM, since garbage collection is not all that efficient with huge heaps. And since running application instances with a 4-8 GB heap size can be done with cheap off-the-shelf hardware, why spend big bucks on Big Iron?

So having a large number of servers is a sane choice, but brings it's own set of problems. Unless one partitions users to servers (having all requests a particular user makes be delivered to the same server), all servers must have a consistent view of the system data, in order to deliver meaningful results. Assigning user requests to particular servers, usually requires expensive application layer load-balancers or customized application code on each server, so it would rarely be your first option. Having all servers work on the same data is a more tractable problem, since it can be solved by having the application state being replicated among server nodes. Usually, only a small part of the application state needs to be replicated, for each user, that is the part which concerns his current session. But even though session clustering solutions have been a well studied field and implementations abound, having no session to replicate is an even better option.

For GSS we have implemented a stateless architecture for the core server, that should provide us with good scalability in a very cost-effective manner. The most important part in this architecture is the REST-like API that moves part of the responsibility for session state maintenance to the client applications, effectively distributing the system load to more systems than the available server pool. Furthermore, client requests can be authenticated without requiring an SSL/TLS transport layer (even though it can be used if extra privacy is required), which would entail higher load on the application servers or require expensive load balancers. In the server side, API requests are being handled by servlets that enlist the services of stateless session beans, for easier transaction management. Our persistence story so far is JPA with a DBMS backing, plus high-speed SAN storage for the file contents. If or when this becomes a bottleneck, we have various contingency plans, depending on the actual characteristics of the load that will be observed.

The above image depicts the path user requests will travel along, while various nodes interact in order to serve them. A key element in this diagram is the number of different servers that can be found, effectively specializing in their own particular domain. Although the system can be deployed on a single physical server (and regularly is, for development and testing), consisting of a number of standalone sub-services instead of a big monolithic service is a boon to scalability.

This high-level overview of the GSS architecture should help those interested to find their way around the open-source codebase and explain some of the design decisions. But the most interesting part from a user's point of view would be the REST-like API, that allows one to take advantage of the service for scratching his own itch.

So that will be the subject of my next post.

Introducing GSS

2009-04-08T16:40:00.000+02:00

During my recent work-induced blog hiatus, I've been working on a new software system, called GSS. I've been more than enjoying the ride so far and since we have released the code as open-source, I thought discussing some of the experience I've gained might be interesting to others as well.

GSS is a network, er grid, er I mean cloud service, for providing access to a file system on a remote storage space. It is the name of both a service (currently in beta) for the Greek research and academic community and the open source software used for it, that can also be used by others for deploying such services. It is similar in some ways to services like iDrive, DropBox and drop.io, but it can also be regarded as a more high-level Amazon S3. Its purpose is to let the desktop computer's file system meet the cloud. The familiar file system metaphors of files and folders are used to store information in a remote storage space, that can be accessed from a variety of user and system interfaces, from any place in the world that has an Internet connection. All usual file manager operations are supported and users can share their files with selected other users or groups, or even make them public. Currently there are four user interfaces available, a web-based application, a desktop client, a WebDAV interface and an iPhone web application, in various stages of development. Underlying these user interfaces is a common REST-like API that can be used to extend the service in new, unanticipated ways.

The main focus of this service was to provide the users of the Greek research and academic community with a free, large storage space that can be used to store, access, backup and share their work, from as many computer systems as they want. Since the available user base is close to half a million (although the expected users of the service are projected to the low ten thousands), we needed a scalable system, that would be able to accommodate high network traffic and a high storage capacity at the same time. A Java Enterprise Edition server coupled with a GWT-based web client and a stateless architecture were our solution. In future posts I will describe the system architecture with all the gory details. The exposed virtual file system features file versioning, trash bin support, access control lists, tagging, full text search and more.

All of these features are presented through an API for third party developers to create scripts, applications or even full blown services that will fulfill their own particular needs or serve other niches. This API has a REST-like design and though it will probably fail a formal RESTful definition, it sports many of the advantages of such architectures:

system resources such as users, groups, files and folders are represented by URIs

GET, HEAD, POST, PUT and DELETE methods on resources have the expected semantics

HTTP caching is explicitly supported via Last-Modified, ETag & If-* headers

resource formats for everything besides files are simple JSON representations

only authenticated requests are allowed, except for public resources

Users are authenticated through the GRNET Shibboleth infrastructure. User passwords are never transmitted to the GSS service. Instead GSS-issued authentication tokens are used by both client and server to sign the API requests after the initial user login. SSL transport can provide even stronger privacy guarantees, but it is not required, nor enabled by default.

The GSS code base is GPL-licensed and therefore anyone can use it as a starting point to implement his own file storage service. We have yet to provide binary downloads, due to the various dependencies, but the build instructions should be enough to get someone started. We are always interested in source or documentation patches, of course (did I mention it's open source?). Most importantly, the REST API will ensure that clients developed for one such service can be reused for every other one.

I will have much more to say about the API in a future post. In the meantime you can peruse the code and documentation, or even try it out yourself. I'd be very interested in any comments you might have.

Applications meme

2009-03-14T15:00:00.001+02:00

It's been a while since my last post, but I've been wanting to write about the interesting stuff that I've been working on. Unfortunately it took longer than anticipated, but now we're almost done and I'll be posting soon with the gory details. However, since I got tagged, I suppose I can throw in a not-so-interesting post, about my Linux application preferences.

Which desktop manager do you use more often ?
Gnome. It's the only one that reminds me of OS X somewhat (appearance- and functionality-wise).
Which desktop application you would not like to see implemented again on linux? And why?
A desktop manager. If you are going to provide yet another incarnation of a 30-year old idea, please don't bother. The concept is pretty well understood: icons/workspace/file manager/drag'n'drop/mouse/menus/launcher/whatever, it's been done before. Like a million times. If you want to be useful, try to fix some of the shortcomings of the existing ones (notification mechanisms, translucent effects, faster access patterns, application integration, etc.).
Which desktop application you definitely would like to see implemented on linux? Describe it briefly or point out to a similar application.
Adobe Premiere, or iMovie, or any other semi-decent video editing software. This was the main reason I bought a Mac.
Write the name of the last project (not the very best, the last!) that made you wish to thank their developers so you can thank them now!
Git. You guys rock! (And if you improve on the UI, you will jazz!)
(Optional) Link the blogs of 1-3 people you’d like to take part to this meme. (no more than three). you can skip this question if you like.
Let's hear from chstath, ckotso and keramida (who may s/Linux/FreeBSD/ at his discretion).

Programming for fun

2008-10-07T23:30:00.000+02:00

I've been in a hiatus from blogging lately, mainly because free time has been in short supply. It's not just the very busy days at work, nights have been even busier. Instead of churning out long blog posts, I've been banging on code instead, day and night. Unfortunately I only get paid for the code I write during the day, but surprisingly (or not) I enjoy the code I write at night the most.

I have written code for banks, multinational corporations, the kind of places where the phrase "professional dress code" is not used strictly in jokes. I've helped build software that was resilient, scalable, secure and intuitive, although granted, not all those at the same time. This was mainly Custom Enterprise Software, or to put it more plainly, Software That Runs On Computers At Work. So I thought I'd try a different angle in my spare time: write software that people use at work, or at home. Or at a coffee shop. Or at the airport. Or wherever they feel like using a computer, anyway. This is sometimes called End-User Software, or to put it in layman's terms, Software People May Actually Like. After trying out some ideas that didn't seem worthwhile, I discovered that some of my buddies had similar thoughts. This is how FireStatus was born.

FireStatus is a Firefox extension that aims to be a swiss army knife for dealing with various social networks, right from your browser, without visiting any particular website. Facebook, Twitter and FriendFeed are currently supported, but more are in the pipeline. For starters, it allows you to simultaneously update your status to all or some of these services, so that all your friends see it, no matter what they are using. The notion of a status that is occasionally updated is familiar to Twitter and Facebook users, since the text field that asks 'What are you doing?' is prominent in their user pages. FriendFeed does not have the notion of a status, but its users can post short (or long) messages, just like Twitter's. FriendFeed also allows posting links to web pages, accompanied with a short description, something that many Twitter and Facebook users have been doing by constructing status updates that start with the description text and are followed by an appended link, usually shortened.

FireStatus can ease the task of posting these messages or status updates, by being always available, instead of needing to have the service pages open and without a large memory footprint, like other similar applications, since it takes advantage of the fact that most people nowadays always have one browser open. I know I do. Come to think of it, I don't know anyone who uses computers that does not keep a browser running most of the time. At least untile the darned thing crashes, which fortunately is something I haven't experienced in Firefox for quite some time.

Posting a message or status update

Clicking on the FireStatus icon, pops up a small toolbar window just above the status bar and below the window document. It is similar to the Firefox findbar that pops up when one searches for text in a page, albeit slightly larger. That was a deliberate design decision that aimed to imitate the success in the usability of the findbar. Lots of little details like this one have been carefully thought out and occasionally debated at length among the team and our beta testers:

Enabling the spell checker in the status message field, for catching those typos when hastily typing a message.
Having the URL inclusion unchecked by default, while the shortening checked, since most posts do not include URLs (and it might be embarrassing if done inadvertently), but those that do usually want them short.
Adding a character counter to help guard against the maximum message length imposed by some services (e.g. Twitter).
Tightly placing the available services in a way that allows for an unambiguous selection.
Using the Escape and Enter keys as shortcuts for canceling and sending the update respectively.

As one reviewer in addons.mozilla.org so eloquently put it, "the simplicity & minimal design are the key components here".

Posting is not the whole story, though. These services provide continuous streams of updates from friends that we need to monitor. Sifting through the updates in every service, while very useful, becomes tedious after a while. It is like making a mental note to check your e-mail every once in a while for new messages. It's one way to do it for sure, but mail notifiers have been around for ages and provide more of a "live" experience.

FireStatus imitates their success in making e-mail conversations "alive", for conversations in the social network space. Every time a new incoming update is received, a notification popup appears, so I can continue using whatever application I was using, but being instantly aware of the news. Twitter notifications contain the name and the picture of the message author as well as the message sent. Low-priority or uninteresting messages can be just glanced at and then automatically dismissed, while important ones can be clicked on, so that the message can be viewed in the browser. FriendFeed notifications can be of various flavors, since FriendFeed aggregates updates from a large variety of online services.

Therefore FriendFeed updates display the service icon along with the author's name for quick identification of the type of update. Clicking on an interesting update opens a browser window to the link contained in the message. Facebook notifications currently contain new message counts, pokes and shares, but work for getting more data is underway.

Although the notifications appear instantaneous, as with e-mail notifications there is a polling process involved underneath. The polling frequency for each service can be separately tuned in the preferences window. The preference window can be opened by right-clicking on the FireStatus icon and selecting the appropriate option in the popup menu. Along with the time intervals between polling for updates, one can enable posting and/or receiving updates for the various services, as well as the authentication credentials where appropriate. For Facebook the user logs in in a Facebook-supplied browser window that pops up, similar to regular Facebook use. No credentials need to be stored separately by FireStatus. For FriendFeed the username and remote key have to be stored locally in the extension preferences. The remote key is not the same as the user password, but is provided by FriendFeed specifically for use by third-party applications, like ours. It can be found by clicking on the link displayed in the explanatory note. Twitter can work either way. If a username and password are entered in the preference window, they will be stored locally in the extension preferences and used for subsequent authentication. If, on the other hand, the fields are blank, FireStatus will consult the browser's Password Manager for any stored Twitter credentials. This will probably cause the master password dialog to pop up once, but the credentials will remain stored only in the Password Manager.

FireStatus preferences

Since we built this thing for our own personal use, pleasure and self-punishment, and as we don't plan to ever get rich out of it, we have released it from the beginning as open-source software through the permissive BSD license. You are most welcome to come by the project's home at Google Code, study the code, fix a bug (or two), help with the (embarrassingly outdated) documentation, contribute a review (otherwise it will remain an experimental extension forever) or just chat with us at the mailing list. Heck, if you feel like it, you can grab the whole code and use it in your own lucrative business and you don't owe us a dime.

With the shape the world economy currently is in, you probably couldn't afford it, anyway.

Murdered by Numbers

2008-07-23T01:24:00.000+03:00

"JavaScript has a single number type. Internally, it is represented as 64-bit floating point, the same as Java's double. Unlike most other programming languages, there is no separate integer type, so 1 and 1.0 are the same value. This is significant convenience because problems of overflow in short integers are completely avoided, and all you need to know about a number is that it is a number. A large class of numeric type errors is avoided."

Douglas Crockford, JavaScript: The Good Parts

When I was twenty-something, I liked watching The Young Indiana Jones Chronicles. Besides the regular stuff that your average Indy fan loves, I was particularly fond of Indy's apparently insatiable appetite for traveling and learning different languages. So while I was contemplating what it would take for me to follow in his footsteps, I figured I'd better start with learning foreign languages. By that time I had already got my English certificate and I had a few years of learning French under my belt that had at least provided me with some means to court women (with little success, regrettably). So I started learning Italian, which everyone said was easy to pick up, especially if you had already mastered some other foreign language. That turned out to be true, and I managed to get a degree in Italian after two years of intensive studies. What made that period frustrating (and occasionally funny) however, were the times that I would inadvertently mix all three languages in the same sentence, creating my own version of Esperanto. Due to the similarities among them, I might be trying to speak Italian, but use an English noun while constructing the past tense of a verb in French. Sometimes I would even be oblivious to my mistake until someone else pointed it out to me. It all seemed very natural as I was doing it.

Lately I'm having a déjà vu, when I find myself coding in Java, JavaScript and C++, often in the same day. More than once I tried to initialize a Java object using a JavaScript object literal. Sadly, the compiler was not very accommodating. While writing a GWT-based front-end, I often transmitted POJOs without being aware that a long value was silently converted to a JavaScript Number, which essentially amounts to a Java double. Reading David Flanagan's "Rhino" book had already left me with the impression that JavaScript was rather flawed for not having separate types for integers, bytes, chars and all the other goodies that languages like C/C++ and Java spoil us with. But after getting a copy of Douglas Crockford recent, highly opinionated, "Good parts" book, his argument resonated with me: "A large class of numeric type errors is avoided." It's Richard Gabriel's "Worse Is Better" principle, or alternatively "Less Is More", in new clothes. Recent events made sure that the message was permanently bolted in my brain.

I've been writing a desktop application in Java that communicates with various Bluetooth and USB devices. The communication protocol is some sort of terminal-like commands and responses that initiate at the Java application and travel through a thin JNI layer down to the C/C++ device driver, and then to the actual device. The protocol documentation describes in... er, broad terms, the sequences of bytes that constitute the various requests and responses. Suffice it to say that my system administrator's experience in sniffing and deciphering network packets proved invaluable.

Sending the command was easy (or so I thought): fill a byte[] array with the right numbers and flush it through the JNI layer. There we get the jbyteArray and put it inside an unsigned char array, which is later send through the device driver to the actual device. When receiving responses the sequence was reversed. It all seemed to work fine for quite some time, until suddenly I discovered that one particular command caused the device to misbehave. I couldn't be sure if the device was faulty or my code was buggy, but since I had zero chances of proving the former, I focused on investigating the latter. A couple of days of debugging later I was still on square one, since as far as I could tell the command reached the device unscathed. Logic says that if a fine command reaches a fine device, then one would be entitled to a fine response. Since I wasn't getting it, I began questioning my assumptions.

I resisted the urge to blame the device, since I couldn't prove it conclusively, and started blaming the command. There was definitely something fishy about it, and to be honest, I had a bad feeling all along. The other commands were simple sequences, like:

0x14, 0x18, 0x1a, 0x3e

or

0x13, 0x17, 0x19, 0x26

This particular one however, was icky:

0x11, 0x15, 0x17, 0xf0

If you can't see the ickiness (and I won't blame you), let me help you.

Java's byte type is a signed 8-bit integer. C++'s unsigned char type is an unsigned 8-bit integer (at least in 32-bit Windows). Therefore we can represent values from -128 to 127 in Java and values from 0 to 255 in C. So, if you have a value between 128 and 255, ickiness ensues. 0xf0 is, you guessed it, between 128 and 255. It is 240 to be precise, if Windows Calculator is to be trusted.

Now, of course I am not that dumb. I knew that you can't assign 0xf0 to a byte in Java, so I had already made the conversion. You see, what is actually transmitted is a sequence of 8 bits. If you get the sequence right, it will reach it's destination no matter what. When you convert 0xf0 to bits you get 11110000. The first bit is the sign bit, which is causing all the trouble. If it was zero instead, we would be dealing with 1110000, or 0x70, or 112 if you're into decimal numbers.

So that's what I had done. I'd constructed the negative version of 112 and used that to fill my command buffer:

0x11, 0x15, 0x17, -112

Looking at it made me feel a bit uneasy, without being able to explain why. I used to think it was the mixed hex and decimal numbers. Yeah, I'm weird like that. However, the zillionth time I reread the Java's byte type definition, a lightbulb lit up over my head. I actually paid attention to the words in front of me: "The byte data type is an 8-bit signed two's complement integer".

Sure, it's 8 bits wide and yes it's signed, using the most common representation for negative binary numbers, two's complement. What's new here? I know how to negate in two's complem...

Whoa! Wait a minute. What I just described above isn't how you negate a number in two's complement. It's actually how you do it in the sign-and-magnitude variant. In two's complement you invert the bits and add one to the result:

11110000 -> 00001111 -> 00010000 or 16, or 0x10

Yep, 16, not 112. So the proper command sequence becomes

0x11, 0x15, 0x17, -16

and, yes, the device seems quite happy with that. As happy as devices get, that is.

So, basically, I wasted many many hours and suffered innumerable hair-pulling episodes for falling prey to a numeric type error. The kind Doug Crockford alludes to in his book. Now, don't get me wrong, I like mucking with bits as much as the next guy. But had I been living solely in JavaScript-land, with the single number type, I'd feel less tired and probably not hate my work as much. Though, granted, I might be needing a haircut right now.

Why you should go to conferences

2008-06-01T23:57:00.000+03:00

One thing I like about working in a large corporation is that you get to meet all sorts of interesting people. You actually have to meet them, since you can't get anything done without coordinating with a whole bunch of them. But since this tends to get nothing done most of the time, I chose to work in a small company instead. Here if you want to do anything, you probably have to do it yourself (there are some interesting repercussions of this that I will leave for a future post). Now, if you work in a small company and want to meet lots of interesting people, you basically have ~~three~~ two options:

Use the net - blogs, mailing lists, newsgroups, twitter, friendfeed, etc. (done that)
Attend conferences.
~~Go to parties.~~

Now as you can personally attest to, I've got No.1 covered. So the last couple of days I worked on number 2 instead. The nice guys at ellak.gr organized the 3rd FLOSS conference and since it took place very close to where I work, I took the opportunity to ~~avoid working~~ meet some old friends and make some new ones.

Number One of course, is Wietse Venema. Since his work was an inspiration for me to pursue a PhD in computer security, I just had to see him in person once. He was more fun that I expected, to be honest. Security guys have a reputation of being a wee bit paranoid and itsy-bitsy abrupt on normal people, not to mention complete strangers. Wietse however was neither. This probably has to do with his academic background or maybe the fact that he is Dutch, who knows. He proved to be a warm, funny guy instead, who runs FreeBSD on his laptop (that always shows character in my book).

His talk was about Open Source and Security, both matters dear to my heart. The discussion of Postfix was a walk down memory lane for me. I can still remember the excitement around its initial release: "Hooray! No more sendmail exploits!". Wietse briefly discussed the architecture of postfix and how it relates to its security. He also showed some statistics comparing postfix to sendmail and qmail that I had seen in the past, but was curious to where everyone currently stands. Apparently postfix holds the second place in number of deployments (good), the first in number of lines of code (bad), but that is caused by having reached sendmail's feature list, without compromising the original architectural goals (good). For the problem of security at large, the most provocative suggestion was to make software development too hard for the laymen, so only experts could do it. Heh. As John Wayne so eloquently used to put it: "that'll be the day".

Lots of comparisons were presented in the talk by Diomidis Spinellis, titled A Tale of Four Kernels. The talk covered a code quality comparison of four major commercial and open-source operating systems, Windows, Solaris, Linux and FreeBSD. Diomidis is a well known author and FreeBSD commiter, and that was evident in the quality of his work and presentation. The room was packed, and the Q&A session lasted almost as long as the presentation itself. I suppose the sensitivity of the subject had something to do with it. Diomidis went through heaps of information so fast you could barely comprehend each slide, unless you had the forethought to study his paper before the conference, like your truly did. Although he was very delicate in his conclusions, stating that all systems had comparable quality, there were some members of the audience that seemed to have already made up their minds and didn't care for diplomacy.

Another great talk was by Jim Blandy. The talk was about Open Source Infrastructure for Software Development, which covered the evolution of version control systems (VCS), from SCCS, to the recent crop of distributed VCS. Being one of the people who gave us CVS and Subversion, his recollection of the history of this area was very exciting. Besides sharing some colorful historical tidbits, he also gave a meaningful comparison of the algorithms and data structures used by the various systems. He gave high marks to Subversion for being a fine choice in a centralized organization structure, but eloquently presented the fundamental paradigm shift that distributed VCS present. He said (as best as I can remember) that controlling access to the commit step (what all centralized VCS do) is the wrong (worst?) place to do it, since adding a change is not that big a deal. Instead the point of merging is what should be guarded, since that is when another changeset gets incorporated in our code. The fact that distributed VCS replicate whole repositories, not just working copies, makes allowing commits a natural thing, while making merge selection part of the release engineering process. I've read many pro-distributed-VCS arguments, but I found Jim's focus on the social aspect of the merge process a refreshing one. Luckily, most of my cohorts were present in this talk, so switching our team to Mercurial might not be just a dream after all.

Jim currently works on Mozilla, on the new ActionMonkey JavaScript engine. Since that work was outside the scope of this talk, I managed to corner Jim afterwards to find out more about its current status. The project attempts to integrate the current Mozilla JavaScript engine, SpiderMonkey, with the Tamarin engine contributed by Adobe. The plan was to replace various parts of the old engine with the new ones, like the garbage collector, the JIT compiler, etc. However doing so leads to performance regressions in some areas and Jim said that one lesson they have learned is that if you ship with worse performance, no matter how rare the circumstances that expose it, you will pay for it at some point. String manipulation appears to be one of those areas, since the old engine had been carefully optimized during all this time and achieving the same performance with the new one, requires work. I have been wondering for some time now whether SpiderMonkey uses some variation of the Ropes data structure for String representation, but Jim explained to me that was not the case.

He was also very excited about the experimental tracing JIT that they are working on, that appears to be a perfect fit for dynamic languages and is expected to boost JavaScript performance on future versions of the browser. I told him that I had already heard of it from Steve Yegge, and I've been meaning to read the literature Real Soon Now(TM), which I swear is the truth, the whole truth and nothing but the truth. After I finish this belated post of course. Jim said that Steve is a great story-teller, which means that Steve, if you ever find yourself in Athens and want a free dinner, just let me know. Explaining the benefits of tracing JIT led us to a detour into LISP-land and while Jim has written a Scheme compiler, I've only done LISP programming for a class as a student. Which is another way of saying I didn't understand a word he said, alas. Anyway, the cool things they are working on will probably come to us in firefox 4, so I still have time to get up to speed.

All in all, the conference was quite an enjoyable experience. Old friends, new friends, free food, stimulating discussions, what's more to ask?

What? To be on TV? Hey, that's easy. You just have to hang outside the main conference room chatting with random people, until you notice a cameraman and a journalist taking interviews from conference organizers and speakers. Then you make sure to be in the camera frame and voilà! You've got your 3 seconds of fame.

JavaScript on the server with Helma

2008-04-03T00:30:00.001+02:00

Rumor has it that the most widely deployed programming language is JavaScript, due to being embedded in every contemporary browser. This extraordinary feat has obscured the fact that the language itself is not only suitable for client programming, but for server-side development as well. Dynamic, loosely-typed languages offer shorter development cycles, agility in meeting changing customer requirements, particularly for enterprises without well-specified business processes, or rapidly expanding enterprises that constantly adapt their core business. JavaScript on the browser is a de-facto standard; using it on the server provides the benefit of reusing a single code base and data model, and debugging it once.

It is also relatively unknown that there exist already more than one frameworks for programming on the server with JavaScript. The most mature among them is Helma, an open-source framework that has been used in production environments for years, like the Austrian Broadcasting Corporation. I've been playing with it for quite some time and I'd like to show how easy it is to get up and running, in order to make any adventurous souls less afraid to give it a try.

Setting up Helma is fairly straightforward. First you have to download the latest version from http://helma.org/download/. Follow the proper link for your platform and the zip or tar archive will be downloaded in your system. After extracting the archive it will create a folder with the latest version of Helma (1.6.1 as of this writing) in it. The only prerequisite is that you have Java installed on your system, since Helma uses the Rhino JavaScript engine underneath. This arrangement allows the reuse of existing Java code through the Rhino APIs. If your system did not come with Java preinstalled, a visit to java.com should suffice. You should get the latest version available (Java 6 as of this writing), but anything after 1.4 should be good. If you are set, go to the Helma directory and execute start.bat or start.sh depending on your platform and you should see some brief startup messages:

Mac:~/helma-1.6.1 panagiotisastithas$ ./start.sh
Starting Helma in directory /Users/

panagiotisastithas

/helma-1.6.1
Starting HTTP server on port 8080
Starting Helma 1.6.1 (January 8 2008) on Java 1.5.0_13

The server is now ready to accept requests. Helma embeds Jetty for its HTTP engine, something that makes the startup process a snap and provides for some interesting architectural choices with its support for continuations. Point your browser to http://localhost:8080/ and you should see the welcome page:

This is an actual application, the “welcome” application, that you are interacting with. You might want to navigate around a bit to skim through the available on-line documentation. When you've had enough, you can stop the server by getting back to the console window and hitting Control-C:


^CShutting down Helma - please stand by...
Mac:~/helma-1.6.1

panagiotisastithas

Just to recap:
Download Helma from http://helma.org/download/
Unzip
Go to the unzipped directory
Make sure you have Java 1.4+ installed
Execute start.bat/start.sh
Point the browser to http://localhost:8080/

Creating a simple application with Helma involves only a few easy steps. First of all create a directory called 'hello' inside the subdirectory 'apps' of the main Helma installation directory. Inside 'hello' create a directory named 'Root'. Inside 'Root' create a file called actions.js with the following content:


function main_action() {
   res.write("Hello, World!")
}

The following screenshot illustrates our progress so far:

Then open the file apps.properties in the main Helma installation directory and add a single line at the end of the file with the following content:


hello

Now start (or restart if it is already running) the server and point your browser to http://localhost:8080/hello. This is what you should see in your browser:

That's it! Your first server-side JavaScript application is ready. In future posts I intend to discuss how Helma wires the various pieces together and explore some of its more interesting features. In the meantime, I suggest that you download it and experiment. You might be pleasantly surprised!

(Grease) Monkey Business

2008-03-21T00:41:00.003+02:00

I've been reading David Byrne's journal (of the Talking Heads fame) since before I heard about blogs and blogging. In an ocean of software blogs that flood my Google Reader, it's one of the few that remind me that I have other interests besides computers. From this journal I found out that he also hosts an internet radio station that I've been often enjoying while at work or sometimes at home.

For this task I've been using iTunes on my Mac at home and Rhythmbox on Linux at work. Besides the easier subscription to the station, the one thing I miss from iTunes while at work is seeing the currently playing song on Rhythmbox. I don't know how iTunes does it, but it's always showing the correct song title and artist, while on Rhythmbox all I see is the station name.

Therefore I had resorted to having a tab open in Firefox all day that displayed the radio home page that shows the playlist and refreshes each time a new song begins. I still had to switch to the browser from whatever I was doing when a new song started, in order to glance on its title and performer. A few days ago I thought to myself: "wouldn't it be nice to have the Firefox title bar display that information and only take my eyes off what I'm currently doing and not my fingers too?".

It turns out that I'm very persuasive when I'm talking to myself, so I set off to make my wish come true. I've been also trying to find an excuse to learn more about Greasemonkey for some time and it seemed like the right time to do it. So after I spent some time reading up stuff from the web, I came up with something that made me happy:


(function() {
  window.addEventListener("load", function(e) {

   var PLAYLIST_URL = 'http://www.live365.com/pls/front?handler=playlist&cmd=view&viewType=xml&handle=todomundo&maxEntries=1';

   (function fetchCurrentSong() {
      GM_xmlhttpRequest({
          method: "GET",
          url: PLAYLIST_URL,
          onload: function(details) {
            var xmlString = details.responseText;
            var parser = new DOMParser();
            var xmlDoc = parser.parseFromString(xmlString, "application/xml");
            var refresh = xmlDoc.getElementsByTagName('Refresh')[0].textContent;
            var artist = xmlDoc.getElementsByTagName('Artist')[0].textContent;
            var album = xmlDoc.getElementsByTagName('Album')[0].textContent;
            var title = xmlDoc.getElementsByTagName('Title')[0].textContent;
            var label = title+' by '+artist+' from '+album;
            document.title = label;
            setTimeout(fetchCurrentSong, refresh);
          }
      });
   })();

  }, false);
})();

It's amazing what 20 lines of JavaScript can do. You can install this user script by clicking here, assuming you have Greasemonkey installed and enabled. If you don't have Greasemonkey or prefer to install a firefox extension that does not require it, click this link instead. It was automagically created with the very nifty User Script Compiler.

This is what I see now when I'm listening to David's playlist (although many times I just glance at the title in the minimized button on the bottom panel):

Remixing the web, indeed.

Re: I wish I could...

2008-03-19T23:20:00.001+02:00

I can tell that spring is here, when my friend George tags me with another blog meme. Apparently it's like a ritual to him, but it came a few weeks earlier this year, so I figured I could delay posting at least until last year's anniversary. With the frequency I blog, he doesn't have much to expect anyway.

This year's theme is to write about crazy or stupid things you'd like to do, but can't, or stupid things that you do anyway. Since I'm running an old-fashioned, respectable blog here, I'll restrict my answer to appropriate stuff, namely computers and software. And with that I expect to have lost 50% of my audience, so I guess that leaves me and you George. So without further ado, here are 10 things (in no particular order) I wish I could do, but probably won't:

...stop reading FreeBSD mailing lists.
...convince my colleagues at work to switch to Mercurial for version control.
...transfer a call at work (in a Linksys IP Phone) without loosing it.
...build the zero-administration appliance I was planning.
...use and contribute to JNode, an operating system written in Java.
...write a book about a computer-related subject.
...implement class data sharing for 64 bit VMs.
...write Objective-C code using Xcode.
...stop subscribing to more blogs and even ditch a few altogether.
...blog more often.

I guess I'm not that ambitious after all...

How to create a WebDAV interface in 10 easy steps

2008-03-15T01:59:00.000+02:00

Test Driven Development (TDD) is all the rage these days. And rightly so, if you ask me. Compare developing software against constantly changing client requirements and developing against a specification in the form of tests that your code must pass. I've done my share of the former for the better part of the last 9 years and I've got the gray hair to prove it. However, I only got a chance to practice the latter very recently and I must say I'm hooked.

My TDD endeavor concerns a small project to add a WebDAV interface to an existing server-side code base. The system already featured other interfaces for manipulating the resources stored in it, but a cross-platform, standards-based, nicely integrated, native client was deemed necessary. Enter WebDAV, a.k.a. HTTP on steroids. Most desktop systems nowadays have a WebDAV client installed by default (Windows calls it Web Folders, others prefer not to rename it), so testing a WebDAV server is not that complicated: if you can mount, browse, store and retrieve files and folders using your desktop client, the server implementation is fine. Nevertheless, we can do better than that. Using litmus, a "WebDAV server protocol compliance suite", we can make sure that our implementation respects the protocol, even in its more obscure corner cases and more importantly, perform the testing automatically, without painstaking clickety-click.

Implementing a WebDAV interface from scratch by reading the protocol specification is not the easiest nor the smartest path. You will have to read it in order to figure out what goes where, but you could also reuse some parts from an existing implementation. If you are extending an existing Java codebase, like me, your best bet is reusing the Tomcat implementation. You can run the litmus suite against a vanilla Tomcat installation, just to verify its WebDAV functionality. Not all tests pass, since the implementation is incomplete, but don't worry, most clients don't implement the spec completely either. However take a note of the results, since this is your target outcome.

For your own convenience, I have described the whole development process in 10 easy steps, ready to print, free of charge, batteries not included:

Download the litmus test suite.
Download the sources for Tomcat, particularly WebdavServlet.java and DefaultServlet.java.
Create your own version of WebdavServlet that will delegate the actual functionality to your own backend and fill it with empty stub methods that correspond to the ones in Tomcat.
Build & deploy your server.
Run the litmus test. All tests fail the first time, but when it eventually succeeds (i.e. matches the result of vanilla tomcat), go to step 9.
Fill one method at a time from Tomcat's sources, substituting the JNDI communications with your backend logic.
Add any other helper methods from other Tomcat classes as necessary.
Go to step 4.
Most tests succeeded, so you are done!
For extra credit: fix the code to pass some more tests and send your changes upstream.

You are bound to come across some hair-splitting problems when modifying the codebase to fit in you own backend, but it beats writing it all from scratch hands down, unless you are Greg Stein or Jim Whitehead. In my case, after two weeks and a submitted patch, I was done and had the tests to prove it. Furthermore, when the need for a code refactoring arises, I can rest assured that my changes won't break the protocol functionality if the litmus tests still pass. Not to mention that I have now contributed to Tomcat.

Man, I haven't felt that good in ages.

Job hunting is like dating

2007-12-17T01:59:00.000+02:00

Paul Buchheit posted an interesting analysis of social networking products, their similarities and differences. In there he has hidden a rare gem:

10. Jobs. Job hunting and hiring are essentially the "professional" analog of dating and seem to work in somewhat similar ways.

Shaking Hands, by Aidan Jones

If you ponder it a bit, it sounds about right, doesn't it? People look for jobs they like and hiring managers look for candidates they like. When there is a match, there is a hire. The parallels are many:

Dumping/seduction: an employee will dump his current job for a better one, often due to seducing promises from their hiring manager.
Bachelorhood/marriage: some people change jobs like crazy (especially in the States), while others stick to one for life (quite common in Japan).
Stagnation: employment relationships, like sexual ones, tend to stagnate after a long period of time, when people lose interest in it.
We might as well admit it, women and work are the two dominant discussion subjects with one's friends, after midnight.
When one gets caught cheating on his wife/company, they go to court.
Girls have breasts, jobs have salaries. The bigger, the better.
People get jealous of their friends who make more money than they do.

Now, what does that mean for us? Are you happy with your current girlfr..., er, job? If not, could you perhaps find some new project / technology / tool / sexual position to stimulate your interest? Or is it perhaps time to be on the market again?