(The Scheme Way)

Lager is cool!

2012-10-01T18:20:00.002-07:00

No, not that type of lager that is so refreshing on a hot summer day... Although I like beer a lot, I rather meant the Erlang logging library developed by Basho (the authors of the Riak NoSQL database).

Logging is that not so sexy thing that every long running application needs. Especially the kind of application that runs in large data centers with operations people monitoring every aspects of it 24 hours a days, 7 days a week . Log files are central to their work as they have to analyze the behaviour of complex systems when a problem arises. You cannot simply stop those applications or put breakpoints in them. So you rely on log files.

The fact that the Erlang system already provides a default logging infrastructure (error_logger) really proves that Erlang was meant for long-running applications, servers, etc. But frankly, being used to logging frameworks in other languages (like the good old log4j), I always feel error_logger way too verbose. Even a simple logging instruction like:

error_logger:info_msg("Hello!~n", []).

generates a verbose trace like:

=INFO REPORT==== 1-Oct-2012::20:45:39 ===

Hello!

So when I stumbled on Lager, I liked it from the start. I am now using it in all my pet projects and will certainly consider replacing an in-house logging infrastructure we have at Nu Echo. (I know there are other very nice logging frameworks for Erlang, I just didn't take time to review them.) It even wraps the error_logger infrastructure, so your existing code will transparently use lager if properly configured.

Lager can log to the console or in files using the provided log handlers (with log rotation and the like). You can also extend it with custom handlers as well (like putting log entries in a database, for example, or actively monitoring special events).

With lager, the logging line above would instead be written:

lager:info("Hello!").

and would print the following at the console:

20:57:48.802 [info] Hello!

Much better, isn't it?

One interesting feature of Lager is that log entries can be tagged with attribute/value pairs. For example, you can tag log entries with an application ID with something like:

lager:info([{appid, ?APPID}], "Hello!").

and redirect in a file all the entries for a specific app ID.

What's more, thanks to the ability to connect to remote nodes in a distributed Erlang-based system, debugging runtime servers becomes a pleasure. Start a shell on a remote node, and call lager:trace_file like this:

{ok, Trace} = lager:trace_file("logfile.log", [{appid, MyApp}], debug).

This will write to "logfile.log" all log entries for the debug log level or higher, and whose attribute appid is MyApp. Once you have debugged your stuff, you remove the trace by calling:

lager:stop_trace(Trace).

That's it. Neat, isn't it?

And you, what logging facilities do you use in your applications?

xmerl crashes my process!

2012-09-20T13:00:00.000-07:00

To my greatest surprise this morning, I noticed that the XML parsing library in Erlang, xmerl, crashes the process that launches the parse when the XML document is not valid. That's right. The parser does not return in that case, it simply calls erlang:exit/1. This was not documented and I had to dig the code to understand why one of my production servers crashed.

Quick notes on Erlang

2012-09-20T10:20:00.000-07:00

In the upcoming weeks, I will try hard to get back to blogging on a more regular basis. As I'm experimenting a bit with Erlang for distributed apps, I will log my thoughts and results as they come in.

Stay tuned!

Functional programming job in Montreal

2012-06-19T18:24:00.001-07:00

Wow! A job opening for functional programmers in Montreal at Genetec. Very nice! Not clear what language is used as the job description only mentions C#, though. Maybe it's F# (that would make sense, given that they seem to use the .NET platform).

I hope there would more such jobs on the market.

MSLUG meeting next week

2011-11-17T16:49:00.001-08:00

The MSLUG (Montreal Scheme/Lisp User Group) is rising from the ashes! Next week, Marc Feeley will talk about Gambit REPL, a version of Gambit-C for iOS and Android.

More details can be found here.

Session handling in NuGram Hosted Server

2011-05-25T06:57:00.000-07:00

Some time ago, I wrote a description of how session handling works in NuGram Hosted Server, which is implemented in Erlang. It's also briefly described on slide 15 of my talk at the Erlang Montreal meetup. Over on the Nu Echo blog, I wrote a short post explaining how the session timeout mechanism can be implemented using a plain Erlang/OTP gen_server behaviour. It's not a very profound post, but it's one of those areas where OTP shines (a very simple solution to a very common problem, and no need to reinvent the wheel).

Slides from my talk at the first Erlang Montreal meetup

2011-05-20T04:19:00.000-07:00

Over on our corporate blog, I've put the slides from the talk I gave at the first Erlang Montreal meetup. The talk was a summary of some lessons we've learned at Nu Echo building NuGram Hosted Server in Erlang.

First Erlang Montreal meeting tomorrow

2011-05-11T19:29:00.000-07:00

Tomorrow, May 12th, 2011, the first meeting of the Erlang Montreal user group will take place at the Notman House from 6:30PM to 8PM. I will have the pleasure to give the first talk, in which I will share some of the lessons we learned at Nu Echo on building real applications in Erlang.

Securing a couchdb database

2011-02-11T13:30:00.000-08:00

Last week, for a post on our corporate blog, I created and populated a hosted CouchDB database (on CouchOne, now CouchBase). Since I put the code on github and it contains references to my publicly available database, I had to secure it a bit. Here is what I did.

Note: this is not a tutorial on CouchDB security issues, only a list of steps I did to secure my database. For a comprehensive guide on CouchDB security, there are excellent online sources.

1. Creating the user/password

The first thing to do is to create a new user with an associated password on the instance.

% curl -X PUT 'https://dboucher.couchone.com/_config/admins/username' -d '"password"'

2. Configuring the user

The next step is to configure the user account by setting its type and roles. In my case, I wanted to add the editor role to my account so I could restrict the creation of new documents to this role (the validator below is kept in a document on the database itself, so it is readable by everyone and I don't want other people to know which user account to hack).

The user account configuration is done using another simple HTTP request:

% curl -X PUT 'https://username:password@dboucher.couchone.com/_users/org.couchdb.user:username' -d@userdata.json

where the file userdata.json contains the following text:

{
  "name": "newuser",
  "type": "user",
  "roles": ["editor"]
}

3. Validator

Finally, to prevent creation of new documents (and modifications or deletion as well) from unauthorized users, I created a design document for my database that only contains a validate_doc_update function. The validation function checks that the user that tries to create/modify/delete a document has the role editor. The code goes as follows:

{
  "_id": "_design/address",
  "validate_doc_update": "function (newdoc, olddoc, userCtx) { if (userCtx.roles.indexOf(\"editor\") == -1) throw({unauthorized: \"illegal access\"});}"
}

To create the design document, I simply entered the following at the shell prompt:

% curl -X PUT 'https://username:password@dboucher.couchone.com/streets/_design/address' -d@designdoc.json

That's it. I could certainly have done something fancier, but that worked and that was really easy to setup.

My resolutions for 2011

2011-01-06T19:09:00.000-08:00

First of all, I wish all of you a very happy new year 2011!

My regular readers may have noticed that I did not publish a lot of posts in the last months. I've been quite busy at work and wrote a number of articles on Nu Echo's blog instead. And most of all, I have not done as much Scheme or Erlang programming as I would have liked to do.

But I played a bit with a number of different technologies in the past months (Ruby, Groovy, CouchDB, NodeJS, websockets, etc.) and it be would worthwhile to write down my observations on these technologies. So my resolution for this new year (as far as this blog is concerned) is to write on a more regular basis about them. For example, I will try to explain:

what I think of Ruby and Groovy for writing DSLs from a Scheme perspective (you know I'm a big fan of macros and DSLs);
what makes Erlang a great language for writing voice applications;
my (very) personal take on NodeJS

and many other things.

Also, there seems to be a growing interest for SchemeScript in the last 6 months. The number of downloads on SourceForge has surged. For this reason, I will take some time to write tutorials and make some screencasts on how to use SchemeScript effectively.

Happy New Year!

ECMAScript - turning a binary function into a variadic one

2010-08-24T19:43:00.000-07:00

In my previous post, I mentioned that it's easy to take a binary JavaScript function and turn it into a variadic one. Here's the helper function to do that:

function variadic(fun, val0f, val1f) {
  return function() {
    var len = arguments.length;
    if (len == 0) {
       return val0f();
    } else if (len == 1) {
       if (val1f != undefined) {
         return val1f(arguments[0]);
       }
       else {
         return arguments[0];
       }
    } else {
      var tmp = fun(arguments[0], arguments[1]);
      for (index = 2; index < len; index++) {
        tmp = fun(tmp, arguments[index]);
      }
      return tmp;
    }
  };
}

The 'variadic' function takes three parameters:

the binary function;
a function that returns the value for the variadic version when called with zero arguments
a function that returns the value for the variadic version when called on a single. If not specified, the first argument to the variadic function is returned as is when called on a single argument.

For example, the summation operator is defined as

var sum = variadic(function(x,y) {return x+y;},
                   function() { return 0;},
                   function(x) { return x;});

while the equivalent of Scheme's / function is defined as

var div = variadic(function(x,y) {return x/y; },
                   function() { throw "not enough arguments to div"; },
                   function(x) { return 1/x;});

It is now possible to call

div.apply(this, [1,2,3,4,5])

to get 0.008333333333333333.

A small experiment with HTML5 canvas and Scheme to JavaScript conversion

2010-08-24T16:54:00.000-07:00

Lately, I've been playing around with some of the upcoming features of HTML5. Last week, I decided to try the HTML canvas. In order to make a realistic test, I tried to port the parse tree viewer in NuGram IDE, one of Nu Echo's products, to JavaScript and the HTML canvas.

To debug a speech recognition grammar with NuGram IDE, an Eclipse plugin, the developer simply enters a test sentence and get a parse tree in the Interpretation view:

NuGram IDE is written almost entirely in Kawa Scheme. The parse tree viewer does not use any graph layout toolkit, the layout algorithm has been written from scratch in Scheme (about 200 lines of commented code). It was a good candidate for a rewrite in JavaScript. Here is what I ended up with (in Chrome):

The conversion process

I translated the whole layout algorithm by hand. I did not use (or develop) any automatic translation tool. I wanted to see how different the JavaScript code would look like compared to the original Scheme code.

I knew that JavaScript/ECMAScript is one of Scheme's very close cousins, but it was amazingly easy to convert the Scheme code into very similarly looking JavaScript code. For instance, take this procedure definition:

(define (compute-nodes-size! gc tree-node level)
  (compute-node-size! gc (get-graph-node tree-node))
  (set-level! tree-node level)
  (for-each (lambda (child)
       (compute-nodes-size! gc child (+ level 1)))
            (get-node-children tree-node)))

It translates to the following JavaScript definition:

function computeNodesSize(ctx, treeNode, level) {
    computeNodeSize(ctx, treeNode.graphnode);
    treeNode.level = level;
    treeNode.children.forEach(function(child) { 
          computeNodesSize(ctx, child, level + 1); 
    });
}

Granted, it's a very simple example. There were some pieces of code that demanded a more complex rewrite. Uses of the Scheme apply procedure is an example. I had to translate the call

(apply max nodes-y)

var maxY = 0;
nodesY.forEach(function(y) { if (y > maxY) { maxY = y; } });

In JavaScript, the max function is not variadic function. It accepts exactly two arguments. It would have been possible to write a function that takes a function of two arguments and turns it into a variadic version of it. But it wasn't worth it, since there was only a single usage of apply and max in the whole algorithm.

Update: I was completely mistaken here, as pointed out by Jon-Carlos Rivera in another post. the max function is variadic. I could instead have written:

Math.max.apply(this, nodesY)

My mistake was to not use apply properly. The first argument to apply must be an object, and not the list of arguments. I should have RTFM before writing such nonsense.

Canvas Support

I tried my parse tree viewer on 3 different browsers (Chrome 5.0.375.29 beta, Firefox 3.6.8, Opera 10.61) on Ubuntu 9.04.

How portable is the resulting widget? Not as much as advertised. I know, HTML5 is not a standards yet, but most of the major browsers claim to support HTML5 canvases. There were two majors aspects that differed between browsers: rendering and user experience.

On the rendering side, Firefox is the worst. Here is what I get:

Some lines appear on top of the nodes (boxes), even though they were drawn before. Opera is not bad, but the texts are not always rendered properly for some font sizes.

On the user experience side, Opera seems the fastest of the three. Chrome comes close. In Firefox, however, the canvas is not redrawn while dragging the parse tree. The parse tree is redrawn only when the button is released.

Conclusion

In conclusion, I've been fairly pleased by the overall experience. The conversion from Scheme to JavaScript was painless, and the result was nice looking. However, and even though it's only based on anecdotal evidence or my lack of experience with the HTML5 canvas, portability is clearly not there yet. My guess is that we'll have to devote a lot of our development time fixing portability issues (we're already spending too much time doing exactly that for plain HTML, right?).

Update: I put the canvas demo online here. Try it and let me know if portability can be improved, if I did things the wrong way, etc.

A blog post on SchemeScript in Russian

2010-07-16T06:52:00.000-07:00

Pavel Samolisov wrote a very good blog post on how to get started with SchemeScript. It's in Russian, but Google can translate it to English (the translation is definitely not very good, but it is better than nothing).

Pavel noted that SchemeScript does not work well on Eclipse Helios. I just pushed a fix on the github repository (and the sourceforge git repository as well) that makes the interpreters work. If you find problems with Eclipse Helios and SchemeScript, please let me know.

2010 Workshop on Scheme and Functional Programming

2010-05-25T05:43:00.000-07:00

This year's Workshop on Scheme and Functional Programming will be held in Montreal on August 21-22. The call for papers is out and papers are due in three weeks (June 14th). Even for those of you not in the research community, it is possible to submit a 6-page paper on practical experience on using Scheme or functional programming.

My VoIP setup at home

2010-05-14T21:41:00.000-07:00

Recently, I had to work from home a bit more. Email and IM is great for communicating with my colleagues and friends, but sometimes talking to them is much more effective. So I decided to finalize my VoIP setup.

Skype

Of course, Skype is the obvious choice for calling friends and relatives. It works great for voice and video on both Windows and my Ubuntu 9.04 (yes, I'm still on Jaunty). My webcam (a cheap Microsoft LifeCam VX-100) is not that well supported on Ubuntu, but it's not so bad. And I use a FreeTalk Everyman USB headset (I got it free at ITExpo in January, thanks to Dan York!).

But Skype itself is not the ultimate answer, at least not for me. Not all my colleagues use Skype at work. So why use Skype Out and pay for local calls when I already have a land line? And I don't like holding a (traditional) phone for one hour when I'm on a conference call with customers or other people at the office. I want to be able to put both of my hands on my laptop keyboard to take notes.

SIP at home

The answer to this is SIP, and three components:

A VoIP adapter. Mine is a Linksys SPA 3102 ATA, a VoIP adapter and gateway from Cisco. This little box converts ordinary (land line) calls into VoIP (SIP) calls, and vice-versa.

Asterisk, a free open-source software PBX. A PBX is the kind of system that can play messages to callers, ask for input, route calls, hold conference calls, queue calls, etc. PBXs are usually costly pieces of hardware that you don't want to buy just for you personal needs.

A SIP softphone.

So my land line is connected directly to the VoIP gateway, which is itself connected to my wired network. When I receive a call on my land line, if I do not answer after a predetermined amount of time, the call is converted to a VoIP call and transferred to Asterisk on my Linux server. Asterisk tries to transfer the call to my SIP phone. If I ignore the call or when I'm not available, the call is simply sent to Asterisk's voice mail module, which will send me a notification email with the recorded message attached to it.

When I want to place a call, I simply enter the number on my softphone, press ENTER, et voilà!

Of course, this whole setup requires a bit of configuration and dialplan programming.

Receiving SIP calls

The only thing that I did not explain is how I can receive SIP calls from outside my home network. I sometimes need the ability to receive calls when I test outbound telephony applications on platforms like Tropo or Teleku.

In order to do this, I created an account on Ekiga.net, a free VoIP service provider. I then configured Asterisk to automatically register (login) to my Ekiga.net account and transfer all calls to my SIP phone when I'm available.

SIP phones

I mainly use two different SIP phones, depending on which OS I boot. On Windows, I use X-Lite free edition. It has a lot of features, has video, but it only supports one account. On Linux, I use Twinkle, which is also free. It does not support video, but it is more stable than the Ekiga softphone on my machine, supports two lines, and has some basic voicemail support.

RabbitMQ acquired by SpringSource

2010-04-13T06:12:00.000-07:00

Rabbit Technologies, a spin out of LShift Ltd. and Cohesive FT, has just been acquired by SpringSource, a division of VMWare. Rabbit Technologies is the company behind RabbitMQ, a high-performance open-source enterprise messaging system based on the AMPQ protocol.

For me, the most interesting thing about this acquisition is the fact that RabbitMQ is an Erlang-based product. This shows once again that Erlang is not an esoteric language anymore, that even large companies are not afraid of using it. This is another great success story for Erlang! And I wish the Rabbit Technologies team the best for the upcoming years.

SchemeScript on Github

2010-03-10T18:09:00.000-08:00

In addition to moving all the development of SchemeScript to git a few months ago, I just pushed my development repository to github. The repository is here:


  http://github.com/schemeway/SchemeScript

I did this essentially because github really rocks. It offers a lot of excellent features for tracking parallel development. I also happen to have a number of other projects hosted there.

SchemeScript used in the Scheme4edu project

2010-02-20T20:20:00.000-08:00

Here is a short screencast of Scheme4Edu, a version of Eclipse streamlined specifically for use by university and college students, featuring the SchemeScript plugin.

Best wishes

2009-12-25T15:56:00.001-08:00

I'd like to wish you all a Merry Christmas and a Happy New Year!

1-session-per-process - some further comments on the NuGram architecture

2009-11-18T16:47:00.000-08:00

In response to my previous post on the NuGram architecture, Ben Simon wrote a post in which he focused most exclusively on the idea of using a single process for handling all the requests for a given session. I wanted to add a few comments about this approach.

First, the idea is not new at all. In fact, I'd say that this is a pretty standard approach in Erlang and the language facilitates its implementation. For instance, the open-source ejabberd XMPP server uses this approach.

Also, it is true that it is completely transparent to the system whether the session's process runs in the same Erlang VM or on a remote machine. The syntax is exactly the same: Process ! Message. That's it. In the case of NuGram Server, the database (which replicates the session table on all nodes) holds references to Erlang processes. Once it has obtained the reference to the session's process, it simply sends a message encapsulating the request to that process (using the ! notation).

Of course, there are some complications if the node on which the process runs suddenly crashes or becomes unavailable. Session replication is less trivial to implement. In our case, the system maintains, together with the session table, a table holding all the relevant data to recreate a session if needed. This was fairly easy to do in our server given our requirements and the nature of the API. For more complex APIs, this could be much more challenging.

Although I'm a big fan of the 1-session-per-process approach (it is a very effective one for the implementation of comet-like servers), it has some limitations that the continuation-based approach do not suffer. The back-button/bookmarking problem immediately comes to mind. Serializable continuations can be put in a database for later retrieval. But the question remains whether this is a practical approach. For instance, for how long do you retain continuations in the database? What happens when the code changes?

Another benefit of using the continuation-based approach is the fact that the code handling the request is written in a more direct style. By this, I mean that the application flow is coded in a more sequential way: do this, then do that, etc. You don't have to code using state machines or callbacks.

But this can be achieved in Erlang as well using two processes per session. (Processes are so cheap in Erlang!) This approach can be used to implement dialog-based applications and providing the illusion of a synchronous API, à la Tropo). I'll talk more about this in another post very soon.

The architecture of NuGram Hosted Server

2009-11-11T16:31:00.000-08:00

In the comments section of one of my previous post, I mentioned that NuGram Hosted Server (www.grammarserver.com) is implemented in a mix of Erlang/Yaws, Java and Kawa Scheme. Alexandre Abreu then asked a few questions about some architectural aspects:

how do erlang and kawa scheme interop? http / json ? ffi ?
to follow up on that, why erlang / yaws and not scala / lift to have a better integration with the jvm ?
the server part if hosted on 1 machine ? how do you handle the load (both the connection load and the computation load since grammar and the services provided by NuGram seem pretty heavy computation-wise)? Maybe you don't need to do much given the amount of people that uses your service and I know you don't offer any guarantees with people using this service in a commercial setup but I am just wondering

These questions prompted for a long answer. So in the following sections I describe some aspects of the NuGram Hosted Server architecture at 10,000 feet high.

1. Context

NuGram Hosted Server is a grammar hosting server for use by communication applications (VoiceXML applications, IM bots, etc.). It provides various grammar-related services through a RESTful API, like dynamic grammar generation and semantic interpretation of text-based sentences (and it will soon provide robust parsing capabilities). It also provides a community-based HTML application to management and share grammars, consult application logs, etc., using standard AJAX techniques.

When we started the project, we had some ambitious requirements in mind:

High-performance HTTP server.
Scalability. (We needed the possibility of dynamically adding nodes to our cluster without interruption of service.)
Fault-tolerance.
Hot code swapping.
Distributed database.

Since I already had some experience with Erlang, I pushed the idea to the management team, after getting the buy-in from my development team. We decided to use this project to evaluate Erlang in the context of a real project.

To answer Alexandre's question more specifically, we did not consider Scala/lift mainly for the same reason we did not choose Java itself. Essentially, we could not use the usual session tracking mechanisms found in J2EE environments (jsessionid, cookies) for our RESTful API. Implementing a clustered solution would have required too much work. And using Erlang was a far better (and more elegant) approach.

2. The architecture

At a very high level, grammarserver.com is composed of two subsystems:

An Erlang/Yaws front-end.
This is the part handling the HTTP requests.
A number of Java/Kawa workers.

Let me describe each subsystem separately.

The Erlang/Yaws front-end

This subsystem handles all the HTTP requests. It currently consists of a single Erlang node, but it has been designed to support the clustering of many Erlang nodes. In this case, we would need a load-balancer in front of the Erlang cluster.

The Erlang system follows most OTP principles. It is a real Erlang application and provides a supervisor and a number of gen_servers. The application also starts an embedded yaws serving requests for several virtual hosts (the HTML websites at www.grammarserver.com, nugram.nuecho.com, with and without SSL support, and the RESTful API at www.grammarserver.com:8082).

Most requests to the RESTful API are done in the context of a session. Each session is associated with an Erlang process and the application keeps a mapping between the session ID and the process for the session in the Mnesia database (it is rather cool to store things like process IDs in a database!). So when the application receives a request, it extracts the session ID from the request URI, finds the corresponding process in the database, and simply forwards the request to that process.

When multiple Erlang nodes are running in a cluster, forwarding a request can involve sending the request to a different Erlang node. This is done completely transparently in Erlang.

Another interesting consequence of using processes to represent sessions is the fact that implementing session timeout becomes trivial. Each session process makes an explicit receive ... after. When the timeout is reached, the session is automatically terminated and removed from the database.

The application also uses Mnesia for other purposes:

It holds the user accounts, of course.
All the instantiated grammars are held in the database.
It holds the node IDs of the available Java/Kawa workers. Keeping that information in a persistent (disk-based) table of Mnesia makes it possible to shut down the Erlang application and reconnect it automatically to the Java nodes when we restart it. More on this below.

Java/Kawa workers

The Java/Kawa workers implement the basic NuGram services. They are written in a mix of Java and Kawa Scheme because most of the code is also shared with NuGram IDE, an Eclipse plugin.

The workers use jinterface to interface with Erlang. This has the advantage of exposing the workers as standard nodes to the Erlang application. In other words, the Erlang application does not even know that the workers are implemented in Java. This is completely transparent.

Many workers can be started, independently of the number of Erlang nodes. The first thing they do is try to find an Erlang node and register with it. If they cannot find an Erlang node, they wait for a specified amount of time, then try again. After a number of retries, they simply stop with an error.

Each grammar is assigned to a single worker. To distribute the load as evenly as possible across the workers, the Erlang system uses a round-robin strategy to assign workers to new grammars (if a session uses a grammar already loaded in a worker, requests are sent directly to that worker, of course).

3. Conclusion

Overall, our experience with Erlang has been excessively positive. (I have to confess that my team members already had some prior exposure to functional programming and Prolog, which helped a lot). Of course, we had to learn some things the hard way, we found some bugs in Yaws. But in the end the platform delivered on its promises. We have an architecture that can scale, we can hot swap code, dynamically change the database schema, add nodes dynamically, etc.

Since NuGram Hosted Server is a free service, we do not guarantee any quality of service, but the platform is really robust and fast and that is very important for communication applications (especially telephony applications where latency translates to dead-air during a conversation).

Vladimir Sedach on high-performance network servers in Lisp

2009-11-06T19:44:00.000-08:00

At our upcoming meeting of the Montreal Scheme/Lisp User Group (MSLUG) on Tuesday, November 17th, Vladimir Sedach will talk about developing high-performance network servers in Lisp. Here is the abstract:

This talk will cover techniques for developing high-performance network servers in Lisp, with examples and lessons from the TPD2, Antiweb, and the speaker's own soon-to-be-released
Common Lisp HTTP servers. Topics covered will include techniques for efficient input handling and output generation, vectored IO, thread pool design, and asynchronous IO management using continuations and state machines.

See you there!

Dynamically setting the Yaws log level

2009-10-25T18:28:00.000-07:00

Debugging a web-based application is more often than not done by first analyzing log files. Even when you can attach a REPL to a running application. They are an integral part of the developer's tool set.

But production applications usually do not enable tracing by default, because this could degrade performance and waste essential resources (memory, CPU time, disk, etc.). Would you dump the content of every HTTP request received by a web server on disk? Of course, when problems occur, trace level is simply raised, with the hope that the logs will contain some useful debugging info.

With the Yaws Erlang-based web server, the log level can be set programmatically like this:

-module(my_yaws_utils).
-export([set_yaws_trace_level/1]).

set_yaws_trace_level(Level) ->
  {ok, GC, Groups} = yaws_api:getconf(),
  yaws_config:hard_setconf(GC#gconf{trace=Level}, Groups).

with Level being either none, {true, http}, or {true, traffic}. The first value disables tracing, the second enables tracing of the HTTP requests in file trace.http, while the last enables tracing of the whole traffic (requests and responses).

It is certainly possible to attach to the Yaws server from another Erlang node to call this function. But if your application is administered by someone not really used to Erlang, it may be more appropriate to provide a command-line script for that purpose.

Basically, the script would connect to the Yaws node (named yaws_daemon@lelouch) and invoke the function with the appropriate log level:

#!/bin/sh
node=yaws_daemon@lelouch
# we should check the number of arguments to the script here...
case $1 in
  none) level=none;;
  http) level="{true, http}";;
  traffic) level="{true, traffic}"
esac

erl -sname trace_script -noinput \
    -setcookie ErlangCookie \
    -eval "rpc:call($node, my_yaws_utils, set_yaws_trace_level, [$level])." \
    -s init stop

The call to rpc:call does the actual RPC call to set the log level on the Yaws server.

The cool thing is the fact that this technique can be applied to a whole bunch of other utility tools, like scripts to stop the server, to reload the application configuration, etc. And this applies to all Erlang applications as well, this is in no way specific to Yaws.

And there are probably other ways to do the same thing. Let me know if you have a better solution.

SchemeScript 1.3.0.alpha9

2009-10-19T16:32:00.000-07:00

On the SchemeWay blog, I posted an entry describing the latest features I have added to SchemeScript. In particular, I have modified the indentation strategy for comments. SchemeScript now behaves more like the Emacs Scheme mode. I have also added some support for the Clojure maps syntax. I wrote some Clojure code lately and desperately needed that feature.

Accessing HTTP/JSON services with JVM-based languages

2009-09-11T17:34:00.000-07:00

Lately, I have been working on a number of client APIs for a REST-like service Nu Echo offers for managing dynamic speech recognition grammars. (The APIs will soon be available on github.) This experience made me realize how difficult it is to provide an API in different programming languages using only the core language (i.e. without having to depend on third-party libraries).

To put you in context, my goal was to provide the same API for accessing an web-based, REST-like service in Java, JavaScript (ECMAScript), Python/Jython, Ruby/JRuby, and eventually Groovy and Clojure.

First problem: Base64

Since the web service uses Basic Authorization on most HTTP requests, the username:password string must be encoded using the Base64 algorithm before being added to the HTTP headers. Believe it or not, there is no standard public class in Java to encode/decode Base64 strings. Fortunately, most scripting languages provide one. Except JavaScript (rhino in my case). So I had to include an implementation of the Base64 algorithm in both the Java API and the JavaScript API.

Second problem: JSON

For simplicity, and the best integration possible with JavaScript, the web service can encode its responses in JSON format instead of XML. (The service was first intended to be used from VoiceXML applications, whose scripting language is ECMAScript.) I thought it would be relatively easy to encode/decode data structures in JSON in all the languages I wanted to support. WRONG!

Of course, Java does not have native JSON support. But I knew that from the start. So no surprise there. And JavaScript, through the eval function, supports JSON natively. Again, no surprise.

The first real surprise came from Python. There is no default JSON library that comes with Python 2.5/2.6 (I haven't tried Python 3). I had to install the simplejson library (which is very nice btw). Unfortunately, it cannot run on Jython 2.2, only on Jython 2.5. Since one of my goals was to run the APIs on Tropo, which only supports Jython 2.2, I had a difficult choice to make. I even tried to simply convert Python dictionaries to strings. But although the Python syntax for constant values is very close to JSON, it uses single quotes instead of double quotes for encoding dictionary keys. (The Python constant {'a': 1, 'b': 2} is written as {"a":1, "b": 2} in JSON.)

In the end, I decided to stick with simplejson for greater portability. (The Tropo guys will probably upgrade to Jython 2.2 one of these days.)

On the Ruby side, there is no standard JSON library. You have to install the 'json' Ruby gem. But it is really easy to install in both Ruby and JRuby. My main complaint is that it is not installed by default with [J]Ruby. And services like Tropo do not necessarily provide all the Ruby gems. (They do provide the 'json' gem, to my greatest surprise.)

Conclusion

When designing NuGram Hosted Server's web service, I thought it would really straightforward to provide APIs in most (scripting) languages running on the JVM. HTTP + Basic Authentication + JSON seemed so en vogue... But clearly, it was harder than expected and the code had to depend on classes/modules that don't come with the core language or the standard library.

I strongly believe that JSON should be more widely supported (natively) by all the major scripting languages, much as XML is. Their own syntax for constant data structures (maps, strings, arrays) is so close to JSON that they should encourage people to use JSON instead of XML. Or at least not discourage its use.

Disclaimer: I am fairly new to most JVM-based scripting languages: Python, Ruby, Groovy. I may have missed something trivial. If so, please let me know.