(The Scheme Way)

Vladimir Sedach on high-performance network servers in Lisp

2009-11-06T19:44:00.000-08:00

At our upcoming meeting of the Montreal Scheme/Lisp User Group (MSLUG) on Tuesday, November 17th, Vladimir Sedach will talk about developing high-performance network servers in Lisp. Here is the abstract:

This talk will cover techniques for developing high-performance network servers in Lisp, with examples and lessons from the TPD2, Antiweb, and the speaker's own soon-to-be-released
Common Lisp HTTP servers. Topics covered will include techniques for efficient input handling and output generation, vectored IO, thread pool design, and asynchronous IO management using continuations and state machines.

See you there!

Dynamically setting the Yaws log level

2009-10-25T18:28:00.000-07:00

Debugging a web-based application is more often than not done by first analyzing log files. Even when you can attach a REPL to a running application. They are an integral part of the developer's tool set.

But production applications usually do not enable tracing by default, because this could degrade performance and waste essential resources (memory, CPU time, disk, etc.). Would you dump the content of every HTTP request received by a web server on disk? Of course, when problems occur, trace level is simply raised, with the hope that the logs will contain some useful debugging info.

With the Yaws Erlang-based web server, the log level can be set programmatically like this:

-module(my_yaws_utils).
-export([set_yaws_trace_level/1]).

set_yaws_trace_level(Level) ->
  {ok, GC, Groups} = yaws_api:getconf(),
  yaws_config:hard_setconf(GC#gconf{trace=Level}, Groups).

with Level being either none, {true, http}, or {true, traffic}. The first value disables tracing, the second enables tracing of the HTTP requests in file trace.http, while the last enables tracing of the whole traffic (requests and responses).

It is certainly possible to attach to the Yaws server from another Erlang node to call this function. But if your application is administered by someone not really used to Erlang, it may be more appropriate to provide a command-line script for that purpose.

Basically, the script would connect to the Yaws node (named yaws_daemon@lelouch) and invoke the function with the appropriate log level:

#!/bin/sh
node=yaws_daemon@lelouch
# we should check the number of arguments to the script here...
case $1 in
  none) level=none;;
  http) level="{true, http}";;
  traffic) level="{true, traffic}"
esac

erl -sname trace_script -noinput \
    -setcookie ErlangCookie \
    -eval "rpc:call($node, my_yaws_utils, set_yaws_trace_level, [$level])." \
    -s init stop

The call to rpc:call does the actual RPC call to set the log level on the Yaws server.

The cool thing is the fact that this technique can be applied to a whole bunch of other utility tools, like scripts to stop the server, to reload the application configuration, etc. And this applies to all Erlang applications as well, this is in no way specific to Yaws.

And there are probably other ways to do the same thing. Let me know if you have a better solution.

SchemeScript 1.3.0.alpha9

2009-10-19T16:32:00.000-07:00

On the SchemeWay blog, I posted an entry describing the latest features I have added to SchemeScript. In particular, I have modified the indentation strategy for comments. SchemeScript now behaves more like the Emacs Scheme mode. I have also added some support for the Clojure maps syntax. I wrote some Clojure code lately and desperately needed that feature.

Accessing HTTP/JSON services with JVM-based languages

2009-09-11T17:34:00.000-07:00

Lately, I have been working on a number of client APIs for a REST-like service Nu Echo offers for managing dynamic speech recognition grammars. (The APIs will soon be available on github.) This experience made me realize how difficult it is to provide an API in different programming languages using only the core language (i.e. without having to depend on third-party libraries).

To put you in context, my goal was to provide the same API for accessing an web-based, REST-like service in Java, JavaScript (ECMAScript), Python/Jython, Ruby/JRuby, and eventually Groovy and Clojure.

First problem: Base64

Since the web service uses Basic Authorization on most HTTP requests, the username:password string must be encoded using the Base64 algorithm before being added to the HTTP headers. Believe it or not, there is no standard public class in Java to encode/decode Base64 strings. Fortunately, most scripting languages provide one. Except JavaScript (rhino in my case). So I had to include an implementation of the Base64 algorithm in both the Java API and the JavaScript API.

Second problem: JSON

For simplicity, and the best integration possible with JavaScript, the web service can encode its responses in JSON format instead of XML. (The service was first intended to be used from VoiceXML applications, whose scripting language is ECMAScript.) I thought it would be relatively easy to encode/decode data structures in JSON in all the languages I wanted to support. WRONG!

Of course, Java does not have native JSON support. But I knew that from the start. So no surprise there. And JavaScript, through the eval function, supports JSON natively. Again, no surprise.

The first real surprise came from Python. There is no default JSON library that comes with Python 2.5/2.6 (I haven't tried Python 3). I had to install the simplejson library (which is very nice btw). Unfortunately, it cannot run on Jython 2.2, only on Jython 2.5. Since one of my goals was to run the APIs on Tropo, which only supports Jython 2.2, I had a difficult choice to make. I even tried to simply convert Python dictionaries to strings. But although the Python syntax for constant values is very close to JSON, it uses single quotes instead of double quotes for encoding dictionary keys. (The Python constant {'a': 1, 'b': 2} is written as {"a":1, "b": 2} in JSON.)

In the end, I decided to stick with simplejson for greater portability. (The Tropo guys will probably upgrade to Jython 2.2 one of these days.)

On the Ruby side, there is no standard JSON library. You have to install the 'json' Ruby gem. But it is really easy to install in both Ruby and JRuby. My main complaint is that it is not installed by default with [J]Ruby. And services like Tropo do not necessarily provide all the Ruby gems. (They do provide the 'json' gem, to my greatest surprise.)

Conclusion

When designing NuGram Hosted Server's web service, I thought it would really straightforward to provide APIs in most (scripting) languages running on the JVM. HTTP + Basic Authentication + JSON seemed so en vogue... But clearly, it was harder than expected and the code had to depend on classes/modules that don't come with the core language or the standard library.

I strongly believe that JSON should be more widely supported (natively) by all the major scripting languages, much as XML is. Their own syntax for constant data structures (maps, strings, arrays) is so close to JSON that they should encourage people to use JSON instead of XML. Or at least not discourage its use.

Disclaimer: I am fairly new to most JVM-based scripting languages: Python, Ruby, Groovy. I may have missed something trivial. If so, please let me know.

Back to our regular program

2009-09-02T18:57:00.000-07:00

Wow! I'm back from vacation, summer is (almost) over, SpeechTEK 2009 is over, kids are back to school. Hope I'll finally have some spare time for small pet projects.

In particulary, I'll try to continue working on a small Erlang framework for developing communication applications that can be used across several channels: phone, IM, web. I used it for developing demos I gave at SpeechTEK.

The framework offers a simple synchronous API to write dialog-based applications, à la Tropo. In a matter of days (thanks to Erlang, Yaws, and exmpp), I have been able to make the framework

generate VoiceXML for the Voxeo platform,
interact with IMified and ejabberd,
do some outbound calling by interfacing to the Asterisk gateway interface, and
use NuGram Hosted Server for dynamic grammar generation and semantic interpretation of text-based interactions.

And a few more things. (I'll blog about this framework on our corporate blog in the upcoming weeks.)

I used the framework to develop a complete application for monitoring web sites. When a web site fails to answer, the application tries to reach me. It first tries to reach me on my preferred IM account. If I'm not available, it then tries to call me. If I don't answer, it sends me an SMS message. I also have a web interface that mimics an IM client.

This is, IMO, a very cool project. Unfortunately, most of the code I wrote has been completely obsoleted by some of the recent Voxeo announcements (here and here): their Prophecy platform and Tropo can now be used for text-based channels... Unless you want to run everything on your own server.

SchemeScript: git and wordpress

2009-07-07T04:37:00.000-07:00

In the last few days, I migrated the CVS repository of the SchemeScript plugin to git. We have started experimenting with git at work and I thought it would be a good idea to use it for my personal projects too.

Also, the web page for the project was old, ugly, and clearly not up-to-date. So I switched to Wordpress. This will ease the maintenance of the site. This is where I will post news announcements, describe features, try to create some buzz (!), etc.

I hope this is all for the best.

My first Tropo JavaScript application

2009-05-23T04:56:00.000-07:00

As promised in a previous post, I will show you the code of my first Tropo application, a simple voice application that gives me the status of the web sites I maintain and monitor.

Why such an application? Well, I have a Python script that pings my web sites every half hour (a simple HTTP GET request to the site's entry point). But sometimes the latency is so high that the script believes the site is down. So I receive an SMS on my cell phone. If I'm outside the office or home, I cannot check whether there is a real problem or not. So I decided to write a very simple voice application I could call from anywhere.

The VUI (voice user interface) of the application is simple: it asks me for a site name and, on a valid choice, makes an HTTP request to ping the site again. If the response code is 200, it tells me that the site is up and running, otherwise it tells me the site is down. The application loops and asks me for a site name again. At any time, I can ask for help by saying "what are my options" or "help me", or I can quit by either hanging up or say "quit".

The main application

The main part of the application begins with the initialization of a number of global variables:

var sitesUrl = "http://schemeway.dyndns.org/voiceapps/sites.json";
var sites = JSON.parse(http_request(sitesUrl, 'GET'));

var grammar = buildGrammar(sites);
var helpOption = sites.length;
var quitOption = sites.length + 1;

var helpPrompt = buildHelpPrompt(sites);

The sitesUrl variable holds the URL of a JSON document giving the names and URLs of the sites I can ping. The grammar variable holds a string representation of the grammar used for speech recognition (in a Tropo-specific syntax). Finally, helpPrompt holds a TTS (text-to-speech) string for the help message.

Once the variables are initialized, the application answers the call and play a welcome message (calls to the Tropo API are in bold):

answer();                                                                                                                                                                 
say("Hi! Welcome to the web site monitoring application.");

It then enters a loop that repeatedly asks for a site name and handles the outcome of the interaction:

var exit = false;
while (!exit) {
 ask("Choose a site.", {
   choices: grammar,
   maxTime: 10,
   onChoice: function(choice) {
     if (choice.value == helpOption) {
        say (helpPrompt);
        return;
     }
     else if (choice.value == quitOption)  {
        say("Thanks for calling!");
        exit = true;
        return;
     }

     var site = new Site(sites[choice.value]);
     say("The application " + site.name + " is ");

     var responseCode = site.test().status;
     if (responseCode == 200) {
       say("up and running.");
     }
     else {
       say("not reachable. The response code is " + responseCode + ".");
     }
   },

   onBadChoice: function() {
       say("Please try again.");
       say(helpPrompt);
   },

   onTimeout: function() {
       say(helpPrompt);
   }
 });
}

The ask function takes as its second argument a set of options, like the grammar, various timeouts, and callback functions. For example, when a valid choice has been made, the onChoice property is called with a single argument, the selected choice. The onBadChoice callback is called when the caller says something that is not covered by the grammar (a "no match"), while the onTimeout callback is called when the caller says nothing (what we usually call a "no input").

Some helper functions

The functions used to create the grammar and the help prompt are given here:

function buildGrammar(sites) {
  var grammar = "";
  var index = 0;
  for(index in sites) {
    var site = sites[index];
    grammar += (", " + index + "(" + index + ", " + site.name + ")");
  }
  grammar += (", " + (++index) + "(what are my options, help, help me)");
  grammar += (", " + (++index) + "(bye, quit)");
  grammar = grammar.substring(1);
  return grammar;
}

function buildHelpPrompt(sites) {
  var prompt = "Your options are: ";
  var index = 0;
  for(index in sites) {
    var site = sites[index];
    prompt += (site.name + ", ");
  }
  prompt += "help me, or quit.";
  return prompt;
}

Finally, the Site constructor function and the test property are given here:

function Site(obj) {
  this.name = obj.name;
  this.url = obj.url;
}

Site.prototype.test = function () {
  var connection = (new java.net.URL(this.url)).openConnection();
  connection.setConnectTimeout(3000);
  var responseCode = 500;
  try {
    responseCode = connection.getResponseCode();
    connection.disconnect();
  }
  catch (e) {
  }
  return {status: responseCode};
}

The whole source code can be accessed here. Feel free to comment, suggest enhancements, or even steal my code!

(2006-05-26 11AM EST) Update: My home internet access seems flacky today, so the above link may not work. Stay tuned, I am working on getting it back to normal.

(2006-05-26 1PM EST) Update: The link should be working now.

A new module system for Gambit-C

2009-05-12T06:12:00.000-07:00

Andrew Whaley commented on my post titled "Erlang or Gambit-C: A practitioner's perspective"

As a slight update to the situation, we now have the fantastic Black Hole module system for Gambit that includes both a continuations web server and a web client that addresses your point 8. Also I have recently released a pure Scheme mySQL interface for Gambit that addresses point 5.
So whilst it may not be Erlang yet - it's edging in the right direction.

Sounds promising!

Update: Black Hole is now on github: http://github.com/pereckerdal/blackhole/tree/master and the documentation can be found on the Gambit-C wiki.

VoicePHP and Tropo - first impressions

2009-05-11T22:00:00.000-07:00

In a previous post, I discussed a number of issues regarding the scripting approach taken by a number of new telephony platforms. But my opinions were somewhat uninformed, as they were based solely on the provided documentation and not on real experience with those platforms.

So I decided to try two of them: VoicePHP and Tropo. Here are my first impressions. (These are not necessarily profound thoughts, since I played with both platforms for a few hours each.)

VoicePHP

VoicePHP is a voice platform provided by TringMe, an Indian company. To use it you have to register for a 7 day beta period.

As its name implies, voice applications are written in PHP. You configure the script URL in your TringMe account and call the application. You have to host your scripts on your own web server.

To call the application, TringMe offers several choices:

Call a provided phone number in the US. (I reside in Canada, so it's a no go for me. I don't want to make long distance calls just to test small script changes.)
Call using a standard SIP phone. That's what I did. Works quite well.
Call the application using a Flash-based widget on a web page. That's really cool. Makes it very easy to develop a voice application and make it available to a large audience.

I tried to implement a small sample application making use of voice recognition. VoicePHP is supposed to support ABNF grammars. Unfortunately, all I got were "no match" events (or were they?). So I didn't get very far. Since there is no debugger or log files I can inspect, I quickly abandoned without knowing whether the fault was mine or not. The platform may be great for DTMF applications, but support for speech recognition did not convince me. And documentation is very light.

At least, their support team was quick at answering my questions. So I'm sure VoicePHP will get better over time. But for now, they did not convince me that the approach is superior to VoiceXML.

Tropo

Tropo is a voice platform in the cloud, provided by Voxeo, a Florida-based company. It's kind of a GoogleApp Engine for voice applications. It offers support for five scripting languages: PHP, Python, Ruby, JavaScript, and Groovy. Your applications can be hosted on their servers or you can serve them from your own web/application server.

You can add any number of applications to your account. Each application is assigned a separate Skype/SIP/INum phone number. As there is a local number for INum in Montreal, I can easily test my applications using either the PSTN or a SIP phone. Both for free.

To test the platform, I implemented a simple application that tells me the status of the web sites I maintain (I get SMS messages once in a while from a ping-like program telling me that one of my applications is down, but its usually due to unusual latency in the network. When this happens, I want a quick way to check if the web application is up and running.) The application asks me for the name of one of my web sites using speech recognition, and fetches the main page of the selected site. If the status code is 200, it tells me that the site is up and running, otherwise it tells me that the site is down. (I'll show you the code in my next post.)

I implemented the application in JavaScript, since it is the language among the supported ones I know best. Tropo uses Rhino, the Mozilla Java-based ECMAScript interpreter underneath, with support for E4X (ECMAScript for XML, an extension to ECMAScript that adds native support for XML).

In contrast to VoicePHP, Tropo provides a good web-based debugger and log viewer. That's an essential part of any development environment. It makes it easy to spot bugs and problems in your application. (I have not been able to stop the execution of an application, as advertised, though. But I have not contacted their support team yet to get help on this issue.)

Speech recognition works great, but my main complain is that the syntax for grammars is quite limited when you're used to developing grammars in ABNF or GrXML format. You cannot ask for dates or numbers, for example. Once they support those formats, sites like NuGram Hosted Server (shameless plug) will become usable to make great mashups with robust dynamic grammars. (I already have a working JavaScript API to generate dynamic grammars using NuGram Hosted Server that I have tested on Tropo. It's just that Tropo cannot use the generated grammars yet.)

Overall, developing a small application on Tropo was really fun!

Conclusion

Although both platforms are quite promising, I found Tropo to be much more mature than VoicePHP. Their website is well designed, effective, without too many bell and whistles. And they provide the right balance of features and tools to help test and debug applications.

So these were very first impressions on VoicePHP and Tropo. I know there are lot of issues I did not mention. I may address them in future posts.

So what is your experience with these platforms? Please share your thoughts in the comments below!

[Disclaimer: although Voxeo owns VoiceObjects, a Nu Echo partner, I based this very informal review solely on my experience using both platforms for a couple of hours each.]

Tropo on your local machine

2009-05-11T06:59:00.000-07:00

In my previous post (about scripting alternatives to VoiceXML), I wrote:

So I think the best of both worlds is to have a scripting language-based API that sits on top of a VoiceXML platform, with good support for speech recognition.

It seems that the Voxeo guys are already planning that with Tropo:

Yes, we fully plan on making the Tropo platform available for local installation. Unfortunately I do not have even a ball park guess to give you. There's a good bit of work for us to do in Tropo for a while to flesh it out properly.

VoiceXML vs the scripting approach

2009-05-05T07:47:00.000-07:00

[Disclaimer: I am partner at Nu Echo, but the opinions expressed here are mine only and do not necessarily reflect the opinions of my co-workers and partners.]

Fact of life: developing good IVR (interactive voice response) applications is hard. If it was easy, the GetHuman project would not have gotten so much attention in the past years. And more often than not, VoiceXML is blamed (not without some good reasons) for this. Just look at the number of new XML languages/scripting APIs out there that try to simplify the development of these applications: VoicePHP, Tropo, IfByPhone, Twilio, and many more.

But as Mark Headd pointed out, the problem is not so much with the language itself (VoiceXML), but rather with the fact that developing good interfaces (speech-enabled or visual) is difficult. Providing simpler interfaces will not automatically give us better applications. In fact, they make it more difficult to implement high performance voice user interfaces (VUIs) that take advantage of the speech-recognition technologies.

For example, most of these new APIs/languages, although they provide speech recognition, will not return multiple recognition results, with confidence scores and structured semantic results. That's a unfortunate because very clever dialog strategies can be implemented when you have such information:

confirming with the second hypothesis when the first has been rejected by the caller (unless its score is under a certain threshold, of course);
deciding whether the application accepts the answer, confirms it, or rejects it based on the confidence scores;
etc.

(But I may not get it, plain and simple. Are these new platforms targeting the large call centers or only the mass of web developers so they can experiment with telephony applications? And this is only the first generation of such APIs, they will certainly evolve over time and offer some more complex features.)

But although I think VoiceXML is here to stay in the call-center industry (too much investments to displace it anytime soon), I think a programmatic approach to IVR development is superior to a meta-language one:

The barrier to entry is lower in terms of development tools. One can reuse his preferred tools: editor, debugger, etc.
It is much easier to provide different implementations of the API, for unit testing purposes, integration with existing platforms.
The language's abstraction mechanisms (classes, higher-order functions, procedures, etc.) can be used to develop libraries of reusable dialog components.
There is a single language involved, instead of dealing with VoiceXML + ECMAScript + (PHP | JSP | ...).

So I think the best of both worlds is to have a scripting language-based API that sits on top of a VoiceXML platform, with good support for speech recognition. But that's a framework, right?

Or use a graphical VoiceXML service creation environment... VoiceObjects, OpenVXML, xMP Studio, Avaya Dialog Designer, Cisco Unified Call Studio, etc. etc. You'll certainly find one that meets your needs.

Gamerizon and QuantZ

2009-03-13T18:27:00.000-07:00

This morning, I paid a visit to the guys at Gamerizon (they have a number of jobs openings for seasoned Schemers). They gave me a demo of QuantZ, their flagship product. Wow! That is an amazing game! I'm not sure how much I can tell you, but just go to their website and look at the screencast. (In fact,

So if you are looking for a challenging Scheme job, send your CV. It's a strong, talented team (they have a lot of experience in game development), and they are highly passionate about their work and Scheme in general. I have a lot of respect for what they do. And it's nice to see places where Scheme is endorsed first and foremost by the founders/company executives.

Scheme job openings at Gamerizon

2009-03-11T12:46:00.000-07:00

Just seen on the MSLUG mailing list, a message from Robert Lizée about 2 openings at Gamerizon:

We are currently finishing the development of a game called "QuantZ" that has been mostly written in Scheme using Gambit-C. A video of the game can be seen at www.gamerizon.com. The current version of the game runs on Windows and Mac OS.

In the short term, we are looking for 2 developers: a game programmer with good optimization skills and a Scheme programmer with good understanding of compiling techniques to work on the iPhone version and smaller versions of the game in Flash, Java and BREW. For these versions, the objective would be to reuse/adapt parts of the current code base, in order to gradually implement a system on which we could develop games on many platforms at once using the same code base.

Tropo - a new platform for developing speech applications

2009-03-11T03:58:00.001-07:00

Last week, at eComm, Voxeo has launched Tropo, a platform for developing speech applications in a variety of dynamic programming languages. They provide a (synchronous) API to interact with the callers of your application. You can do things like playing prompts, asking questions (DTMF/speech recognition), transfering calls, and so on. All this in your language of choice, without having to mess with VoiceXML, CCXML, low level telephone APIS, and the like.

Tropo currently supports JavaScript, Groovy, Ruby, Python, and PHP. They expect to add support for new languages in the future. And if you follow my Twitter feed, you noticed that they are open to supporting Scheme/Lisp. I even volunteered to help them. What cool guys they are! (It seems I am not the only one who asked about Lisp...)

Now, which Lisp dialect would make most sense on the Tropo platform? Given the state of the Scheme community and the variety of implementations, the answer may not be as simple as you would expect. Here are a few possibilities, with their respective pros and cons.

JVM-based systems

First, there are a number of Lisp implementations on the Java platform: SISC, Kawa, and Clojure. With these implementations, their foreign interface to Java compensates for their lack of libraries.

SISC is a fully R5RS-compliant implementation of Scheme. It can interface to Java, optimizes all tail-calls, fully supports call/cc, etc. It is a mature implementation, but there has not been any new release since the end of 2007. Unfortunately, it does not implement optional and keyword arguments, and does not have a compact syntax for hash maps. I am not sure how elegant code will be in SISC compared to what we can do in JavaScript or the other scripting languages.

Kawa is a less compliant Scheme implementation than SISC and does not seem to have a lot of community support. It does have DSSSL-based optional and key arguments, which is nice, but it does not provide a clean, compact syntax for hash maps natively. But since the reader can be extended, this is not really a big deal.

Clojure, the newest Lisp on the block, may be a good choice. I am not really familiar with it, but from what I saw in the documentation, it supports objects, multi-methods, special syntax for hash maps, optional and keyword arguments, and much more. It does not optimizes tail-calls but, hey, it's a Lisp, not a Scheme. And there seems to have a strong community behind it.

Native implementations

Now, if we now turn to native implementations of Scheme (i.e. not running on the JVM), there are many more choices. I won't list them all, but if we consider the size of their respective community, the clear choices (from my point of view) are PLT-Scheme, Gambit-C, and Chicken. As you know, I have a preference for Gambit-C. It is a robust, fast implementation of Scheme (R5RS). It has very good debugging tools, supports DSSSL-based optional and keyword arguments. The readtable can be modified. Etc. Etc. Add Termite and you have a strong platform for developing amazing next-generation multi-modal voice-enabled applications. But on the down side, it cruelly lacks libraries.

Of course, PLT-Scheme and Chicken do have a lot of libraries, with a language support for using them. Chicken has most of the features of Gambit-C, except good debugging tools (and Termite, of course, which only runs on Gambit-C). PLT-Scheme, on the other hand, does not have runtime debugging tools as good as Gambit-C, but it comes with a great development environment, DrScheme.

And the winner is...

Well, I don't know. From a programmer point of view, I would hesitate between Clojure and Gambit-C. But there may be some other issues that I did not consider, like the architecture of the Tropo platform, the interpreter start time, memory footprint, etc. that would affect how the application could scale. Is there a new interpreter launched for each new call?

So tell me, what would be your ideal Scheme/Lisp implementation on the Tropo platform?

On the use of parser/lexer generators

2009-02-22T17:43:00.000-08:00

Today, I came across a blog post on Advogato about the use of parser and lexer generators. The author makes a case about writing hand-crafted lexers (in his case, a lexer for XML). As the maintainer of a LALR(1) parser generator for Scheme, I could not resist and wanted to share my thoughts on this subject.

First and foremost, let me say I have mixed feelings about the article. Although it addresses important issues, it concludes with a very simple lexing problem, not a challenging one. I think this alone greatly diminishes the strength of his argument.

Now let me give my own list of pros and cons of parser/lexer generators.

Pros

I am a big fan of DSLs. And parser/lexer generators obviously fall in this category of tools. They operate on a high level, declarative description of the language we wish to parse. This results in more maintainable software, with less bugs (most popular generators have been used extensively, so you can already be confident on the correctness of generated code).

At Nu Echo, I wrote two parsers for a language called ABNF (it's a language for defining grammars used in speech recognition applications), one using an LALR(1) parser generator and the other one being an hand-crafted recursive-descent parser. The latter is certainly faster and give better error diagnostics (I'll get back to this in a moment), but from a maintenance point of view, the former wins.

Another important aspect of parser generators is that they often support different parsing technologies/drivers. For instance, bison can generate LALR(1) or GLR parser (lalr-scm can, too). And ANTLR supports LL(1) but also LL(k) grammars. Without having to rewrite the source grammar. You can't do that with an hand-crafted parser. (Of course, you must stay in the same "family" of parsing technologies. A grammars written for an LR parser generator may not be appropriate for an LL parser generator, and vice versa.)

A corollary to this is that you can concentrate on offering the best interface to your users: the language itself. You don't have to adapt the language to a parsing technique. This is very important from a design point of view. Usability is (or should be?) one of the driving goals when designing a new language.

Cons

Unfortunately, they are more pragmatic issues that can preclude the use of parser/lexer generators. Here are a few:

My experience with parser generators showed me that they are nice for command-line tools, but they do not produce good parsers for use in interactive environments (like an editor). (I must confess that I have almost exclusively used LR parser generators a la yacc/bison, so this argument may be a bit stretched. And I know that there have been quite a lot of work on interactive programming environments and incremental parsing.) Error recovery mechanisms are often limited and tend to produce cryptic error messages.

Hand-crafted parsers are often much easier to debug. Ever tried to understand what a shift/reduce conflict is? And tried to figure out how to rewrite your grammar to resolve it? You have to understand the parsing technology. Talk about an abstraction leak!

Generated parsers are derived resources in a development project (i.e. resources obtained from other resources). This means they are usually not kept in the CVS/git/SVN/... repository. This can complicate the build process. This may not be a big deal, but in some projects this can be trickier.

In some languages other than C/C++/Java, like Scheme, this may not be a problem since the parser generator can be called at macro-expansion time.

Et voilà! I'm surely missing a lot of other equally important aspects, but it's getting late and I am tired. And you? What are you reasons for using or not using a parser generator/lexer generator?

Nils Holm's textbooks for free

2009-02-22T15:00:00.000-08:00

Nils Holm is offering most of his text books online free of charge. I have not read any of them yet, but judging from their title, I'm definitely missing something!

Reflection can be costly

2009-01-28T06:15:00.000-08:00

At least in Java.

A few days ago, I wanted to optimize a piece of code written in Kawa Scheme. You know how that works: you write something that seems to be good enough in most cases in terms of performance. Then you try it on a large input (a file several several megabytes long). And your code chokes. It does not crash, but takes way too much time to be usable in practice. But that's ok. Premature optimization is the root of all evil.

So I ran the profiler on my code to find out that it was using the Java reflection routines like hell. I was astonished, to say the least. My code was carefully written, with lots of type declarations (which are optional in Kawa) to prevent calls to the reflection API by the Kawa runtime. It seems that I was not careful enough. Or that I was expecting more from the Kawa compiler (I use a very old release, btw).

The problem was this. In a single static Kawa module, I was defining some classes and a few helper functions. Something along the lines of:


(module-name <mymodule>)
(module-static 'init-run)

(define-simple-class <ClassA> (<Object>)
  (some-field))

(define (helper-function (an-a :: <ClassA>))
  (do-something-with (field an-a 'some-field)))

Surprisingly, the reference to the some-field must be done through reflection! I'm not sure why, I don't master the inwards of the Kawa compiler.

Is there a solution to this problem? Yes. Put the class definitions in a module an the helper functions in a separate one. By doing this, the run time of my code dropped by 50% (a speedup of 2). There are many other possible improvements to speedup my code even further (data structures, algorithms, etc). But at least my code is now compiled the way I expect it to be.

Extended desktops and Compiz

2009-01-12T06:53:00.000-08:00

This morning, I realized that Compiz was not working anymore. I could not remember when it stopped working, but I remembered having a similar problem a month ago, when using my Wubi-based install of Ubuntu.

After a few minutes of googling around, I found that this probably happened after connecting a projector (or another monitor) on my laptop. If that happens again, I just need to issue the following command:

  % sudo dpkg-reconfigure -phigh xserver-xorg

and restart X (by just logging out and in).

A case for voice test cases

2009-01-07T06:40:00.000-08:00

To me, the future of voice service creation environments (SCEs) is in IDEs that will offer ways to test applications without having to resort on a VoiceXML platform, that will ease the creation of test cases. Testing voice applications manually by calling the application is costly (in both time and money) and the more you can do in your development environment, the better.

Think about how easy it is to run JUnit tests from Eclipse. In a fraction of a second, you can run hundreds of test cases. And you can use the power of Java to factor large portions of your tests and reuse lots of code. Doing the same for voice applications would be a killer app. (Of course, you can certainly do that when using a well-designed custom VoiceXML framework with a strong underlying model. But most SCEs have not been designed with this in mind.)

In a tough economy, environments that will help us deliver better and more robust applications in a more cost-effective way will clearly have a competitive advantage.

At least, some companies understand that.

How I switched to Ubuntu

2008-12-18T03:34:00.000-08:00

Yesterday, I decided to install Ubuntu on it own dedicated, ext3 partition, to get full performance (sometimes, on my Wubi-based install, a process called mount.ntfs takes 00% CPU and freezes most applications).

But prior to installing Ubuntu, I first had to shrink my Windows (ntfs) partition. Here is what I did, only using free software:

Removed unnecessary data to from Windows (I removed about 14 Gb of junk!)
Made a backup of my Windows partition. I first tried ghost but our corporate version is old and I did not succeed in detecting my Samba shares on my home server (CentOS). So I dowloaded and installed Macrium Reflect Free Edition. Great software. Run flawlessly.
I shrunk my Windows partition. To do this, I used EASUS Partition Manager. Again, this worked flawlessly.

At 10h30PM, I finally realized I didn't have any blank CD to write my Ubuntu installation CD! So I decided to use my USB key. To write the ISO file to my USB key, I used UNetbootin. It was very simple to use and I was able to boot the Ubuntu installed just a few minutes later.

That's it. Now I run Ubuntu natively. It's so fast, now!

A new fan of Ubuntu

2008-12-15T19:52:00.000-08:00

I installed Ubuntu (using the Wubi installer) on my Dell Latitude last week. It just rocks!

I was still under the impression that installing and configuring a Linux system was not an easy task (I have been a Linux user since the early '90s, I installed CentOS 5 just a few weeks ago on a server at home). I was just plain wrong. Installing Ubuntu 8.10 took just matter of minutes. Configuring the internet (wireless access + openvpn) is as intuitive as it can be.

Moreover, In the past couple of years, it took me hours to configure my HP printer on Windows XP. With Ubuntu, all I had to do was inserting the USB cable in my laptop et voilà!.

I'm sold to Ubuntu!

Now if I can take the time to ghost my machine and install Ubuntu on a native ext3 filesystem... I would unleash the full potential of my laptop (Wubi installs Ubuntu on a virtual drive, a file on an ntfs partition.).

VoiceObjects acquired by Voxeo

2008-12-10T09:41:00.000-08:00

Voxeo, a company I mentioned a few times in the past, announced yesterday the acquisition of VoiceObjects, a company located in Cologne, Germany, with which Nu Echo has developed a great relationship in the past eighteen months.

I'm very excited by this acquisition. This will certainly create new opportunities for VoiceObjects in the US, and Voxeo will extend its reach on the old continent.

But more importantly, I'm very happy for the all the people I know at VoiceObjects. I have a lot of respect for their work, their attitude, and their professionalism. Voxeo not only acquired a great technology and platform, they have joined their forces to a great team.

Jazz night at the MSLUG

2008-11-12T16:19:00.000-08:00

I just added a new meeting announcement to the next MSLUG website, to be held on November 27th. Here is an excerpt:

As some of you know, Guillaume Cartier has been working for the past 12 years on his Jazz Scheme system. It is a powerful and complete development environment for Scheme applications. You can read about it here: www.jazzscheme.org.

A couple of years ago, he decided to port the whole Jazz kernel to Gambit-C Scheme and to eventually release Jazz under an Open Source License. With the near completion of X11 support, the time to do a full public release is coming very close.

In order to help spot and iron out the remaining issues for a full public release, Guillaume will be doing a "Closed Beta" release to the participants of the MSLUG. The release will take place at the MSLUG meeting on November 27th. You are encouraged to bring your own laptop, as Guillaume will walk us through the installation of Jazz and the creation of a simple application. Note that Jazz should run on Windows *and* on systems featuring an X11 server.

Thanks to Marc Feeley, WiFi Internet access will be available at the meeting, which should help with the whole process.

Don't miss it!

Erlang impressions (in French)

2008-09-24T18:53:00.000-07:00

François Pinard, a colleague of mine, reveals his first impressions on Erlang. François has worked on Erlang stuff for the past 4 months and explored a lot of dark corners of both Erlang and Yaws. The only caveat: it's in French.

Erlang @ Yahoo

2008-09-17T16:23:00.000-07:00

Too bad I cannot make it... Mike Gerakines will talk about the use of Erlang at Yahoo at the BayFP. Hope the slides will be available afterward...