Kvardek Du

Interview about Lisp at SISCOG

2024-09-03T10:52:00.012+01:00

My friend Rui was interviewed about Lisp and how we use it at SISCOG. The original interview is in Portuguese but you can read a translation via DeepL below:

SISCOG Engineering: get to know this cutting-edge Portuguese company

Find out how Lisp continues to drive innovation at SISCOG. Interview with Rui Patrocínio, Scheduling Team Leader at SISCOG

#1 How did you first get to know the Lisp programming language?

When I joined Técnico in 1999, the programming language taught in Introduction to Programming was Scheme, which is in the Lisp family. The curriculum closely followed the MIT 6.001 course (whose lectures from the 80s are on Youtube) and one of the best programming books ever for beginners and beyond: Structure and Interpretation of Computer Programs. The great advantage of learning to program with Scheme is that it is a very simple language, with a very simple syntax. All the effort goes into understanding the logic of what you're implementing, so there's no need to memorize the syntactical nuances of the language.

After that experience, Lisp reappeared as Common Lisp in the Artificial Intelligence course. Common Lisp is the current industrial version of the Lisp family of languages, which is what SISCOG uses on a daily basis. Common Lisp is a multi-paradigm language. Imperative, functional, object-oriented programming with very sophisticated meta-programming mechanisms available. As such, using the language as a whole implies some maturity and it's only natural that it should appear later in the curriculum of a computer engineering course.

I think it was the “power” of the Common Lisp language that made it particularly appealing to me and, despite the standard being from 1994, it is a very modern language. It should be noted that Guy Steele, one of the main figures behind the Common Lisp standard, was also heavily involved in the development of Scheme, C and Java, being hired by Sun at one point to improve Java. Here's a quote from him on a mailing list focused on the discussion of programming languages from the 1990s, talking about Java:

And you're right: we were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp. Aren't you happy? —Guy Steele

#2 How SISCOG uses Lisp in its products

We use Common Lisp in the vast majority of our software. Both desktop applications and backend parts of web applications are implemented in Common Lisp. We also use C++ to develop specific modules, but Common Lisp still has the most lines of code in our repositories. It's a very expressive language that compiles to machine code reasonably efficiently without much effort on the part of the programmer. So it's an easy choice for most scenarios.

It remains to be said that our products have been in operation for more than 30 years in various national and international companies, such as the London and Lisbon Underground, or the railways of the Netherlands or Canada. They are decision support software for the optimized planning of these transport operators' resources, namely time and space, materialized in timetables, vehicles and personnel. These products help to plan and manage these operational resources as quickly and efficiently as possible, providing gains and savings in various areas, as well as, for example, greater satisfaction on the part of workers thanks to shifts and work schedules that better meet their preferences. And all with Lisp as a base!

#3 What are the main advantages of the Lisp syntax compared to other programming languages, such as C++?

The great advantage of the syntax is also the thing you'll find most strange at first: the brackets. The fact that everything uses a prefixed syntax also makes everything quite uniform. This combination of parentheses and prefixed syntax is called s-expressions in Lisp. S-expressions are code and data at the same time, and this is what allows you to have macros (functions that take code as an argument and return code as a value) that extend the language transparently, as if they belonged to the standard language.

#4 Can you explain the concept of “S-expressions” and how they contribute to the clarity of LISP code?

Basically, “S-expressions” are one of two things:

atomic expressions (e.g. the number 2024 or the string “hello world” or the symbol +)
a list, typically in the format (operator arg1 arg2 ... argn)

It's this uniformity, as I said earlier, that makes everything simpler. It also helps a lot to be able to generate code programmatically because it's all about manipulating lists, for which Common Lisp has a very reasonable API. This is where the power of Lisp macros comes from.

At first you find it strange, then you understand it. [nice try, DeepL. Rui wrote "Primeiro estranha-se, depois entranha-se." which is quoting a famous Coca-Cola slogan written by Fernando Pessoa in the 1940s. Richard Zenith, in his biography Pessoa: An Experimental Life, attempts to translate it as "On the first day you drink it slow. On the fifth day you can't say no." —Luís]

#5 How does Lisp allow rapid prototyping using untyped variables?

Static typing in large codebases is widely advocated today because it allows more automatic tools to check for problems in the code. Common Lisp has dynamic typing, but allows optional type declaration. It's quite common to only declare types when it's necessary to “squeeze” more performance out of a code segment. The compiler, via type inference, provides some information about problems encountered when there are enough type declarations to extract relevant information. This allows us to “fight the compiler” only in segments of code where it is very relevant and not be making premature optimizations to the whole code.

#6 How does Lisp facilitate metaprogramming?

There are two basic “tools” for metaprogramming in Common Lisp: the Meta Object Protocol and the macros mentioned earlier.

Meta Object Protocol is basically the mechanism behind the implementation of the Common Lisp object system (called CLOS, Common Lisp Object System). Although it doesn't belong to the standard, it is available in most Common Lisp implementations and is sometimes useful, particularly when implementing utilities to inspect the code or improve our development environment. For example, it's possible to implement something that shows us the relationships between classes, what methods exist, etc. This is not something you do on a day-to-day basis, but it can be done by you without depending on the guts of a specific IDE.

Another interesting mechanism is macros. When CLOS was introduced, it was basically a set of macros on top of the Lisp of the time (historically there were a few more iterations - CLOS wasn't the first object system developed). In other words, with macros it's possible to extend the language to introduce most of the mechanisms and paradigms that exist in other languages (such as object-oriented programming) in a way that feels natural to a Lisp programmer.

This type of mechanism allows SISCOG to implement undo and redo mechanisms in our software in a way that is practically transparent to the programmer. This is done by extending the class declaration. The programmer notes for which attributes of each class a history needs to be kept and the end user is able to undo their operations and see these data variations instantly (with the typical implementation using Command Pattern, multi-level undo is not instantaneous as it is necessary to re-execute operations, which may not be trivial; the tradeoff is, of course, the memory spent).

#7 What is your opinion of Lisp's learning curve compared to other programming languages?

Anyone with experience in a few different paradigm languages can quickly program in Common Lisp without major problems. Most of the concepts are familiar. Our experience at SISCOG, particularly with more recent hires who had no contact with Lisp in college, is that people adapt quickly and within a week are programming and making minor corrections.

Obviously, there are parts of the language that are less common and you need more maturity to use them effectively. These include the Meta-object protocol and the macros mentioned earlier.

In this case, the need for greater maturity comes simply from the fact that we are extending the language. Designing a new language isn't easy and there are people whose career it is to do this (like Guy Steele mentioned earlier). Of course, this is also rare and the most common thing is to introduce macros for small functionalities that make the programmer more productive with Domain Specific Languages.

#8 Why is Lisp a popular choice for specialized areas such as artificial intelligence and natural language processing?

It's essentially for historical reasons. Lisp was born in that context, there have been several “classic” artificial intelligence systems implemented in Lisp and so it has continued to be used. The fact that it's very easy to start a project quickly, it's easy to iterate and progressively improve what's been done, also helps in a research context where we're more concerned with testing ideas than making the software “bulletproof”, which you can do progressively (treating errors, adding static typing, etc.) in Common Lisp.

#9 Can you give some examples of projects at SISCOG where Lisp has proved particularly advantageous?

SISCOG has relied on Lisp from the start. It's a very stable language, with good compilers and good performance, it's compact and has allowed us to go through the history of computer science across various platforms (Lisp machines, Unix, Windows) and maintain a code base with immense domain knowledge over the years. A company with software in production for over 30 years is not very common in the world and Lisp is clearly our little secret weapon.

#10 How does the Lisp development community compare with other programming language communities in terms of support and resources?

The Lisp community isn't very large, but what we lack in programmers, we make up for in enthusiasm. Paul Graham (of Y Combinators) was a great driving force behind the language in the late 1990s, early 2000s and Hacker News still talks about Common Lisp (and variants) on a regular basis. Many years ago there was a strong community on the old newsgroups, but that has largely moved on to Reddit and IRC. There are also a few annual conferences (e.g. European Lisp Symposium) where some of the most important members of the community usually gather. We don't have by far the largest number of libraries available and sometimes we have to implement things 'in house'. The community does, however, have enough to make high-quality software that is sold all over the world, as SISCOG has demonstrated.

#11 What are your predictions for the future of Lisp in the technology industry?

It seems to me that the future of a language depends a lot on fashions and the investments made in it. Google uses Lisp via the purchase of a company a few years ago (ITA Software, for flight search) and, as long as these large companies continue to invest, languages will thrive. At the moment, fashions have moved on to other platforms, but let's see what the future brings. Perhaps our little secret weapon will become less and less secret.

Manuel Simoni on CL's control flow primitives

2024-01-30T14:56:00.002+00:00

Manuel Simoni dusts his Axis of Eval blog off and writes about Common Lisp's BLOCK / RETURN-FROM and UNWIND-PROTECT. A summary for non-Lispers.

Reddit 1.0

2018-04-02T13:03:00.001+01:00

As many Lisp programmers may remember, Reddit was initially written in Common Lisp. Then, in late 2005, it was rewritten in Python. Much discussion ensued. Steve Huffman (aka spez, current CEO of Reddit) wrote about why they switched to Python. The late Aaron Swartz also wrote Rewriting Reddit, focusing on the Python side of things.

Last week, that original, Common Lisp version of Reddit was published on GitHub: reddit-archive/reddit1.0. There's no documentation, but we know from the articles above that it ran on CMUCL. reddit.asd is perhaps a good entry point, where we can see it used TBNL, CL-WHO and CLSQL, a common toolset at the time.

It's missing various bits of static HTML, CSS and JS, so I don't think you can actually run this website without a fair bit of scrapping and stitching from the Wayback Machine. The code is quite readable and clean. It's not terribly interesting, since it's a fairly simple website. Still, it's a website that grew to become the 6th most visited worldwide (with 542 million monthly visitors) and now has 230 employees, says Wikipedia, so it's nice to see what it looked like at the very beginning.

A Lisp REPL in your pocket

2017-12-19T22:37:00.000+00:00

Thanks to Polos Ruetz, you can now play with Common Lisp directly on your Android phone. All you need to do is install CL REPL from the Google Play Store. CL REPL is one of the examples part of EQL5-Android which is built on top of EQL5, a project that marries ECL with Qt.

CL REPL

A Common Lisp REPL with command line and history, plus a simple editor with syntax highlighting, simple visual paren-matching, a file dialog for opening/saving files, and a simple debug dialog. It uses the ECL implementation for the Lisp side, and Qt5/QML for the UI. This is an open source project (see EQL5-Android).

(via @dk_jackdaniel)

Re: Querying plists

2016-07-10T23:39:00.001+01:00

Zach's Querying plists blog post showcases a neat little querying DSL for plists. I couldn't shake the feeling that it looked an awful lot like pattern matching. I've often been impressed by optima, but I barely get to use it, so I thought I should try and see what querying plists looked like using pattern matching.

Here's what I came up with. Zach's example

(query-plists '(:and (:= :first-name "Zach") (:= :state "ME")
                     (:not (:= :last-name "Beane")))
              *people*)

becomes:

(remove-if-not (lambda-match ((plist :first-name "Zach" :state "ME"
                                     :last-name (not "Beane")) t))
               *people*)

It turned out more succinct than I initially expected! Also, it's trivially adaptable to other kinds of objects. E.g., given the following class:

(defclass person ()
  ((first-name :initarg :first-name)
   (last-name  :initarg :last-name)
   (state      :initarg :state)))

all we have to do is swap plist with the class name person and we're all set:

(remove-if-not (lambda-match ((person :first-name "Zach" :state "ME"
                                      :last-name (not "Beane")) t))
               *people*)

We can't quite define something exactly like Zach's query-plists because, AFAICT, optima's patterns are not first-class objects but perhaps we can cheat a little bit.

;; naming things is hard. :-/
(defmacro matchp (pattern) `(lambda-match (,pattern t)))
(defun filter (predicate list) (remove-if-not predicate list))

(filter (matchp (plist :first-name "Zach" :state "ME"
                       :last-name (not "Beane")))
        *people*)

Making this equally succinct when the query criteria are not constant is a challenge for another day and makes it clear that matchp is a lousy abstraction. ;-)

slime-macrostep

2016-02-03T00:51:00.001+00:00

SLIME's just got a new contrib called slime-macrostep, courtesy of Jon Oddie who also wrote the underlying macrostep package.

And what is slime-macrostep? It's an interactive inline macro-expander. Have a look:

In this quick demo, using a CFFI example, I start by expanding the top-level WITH-FOREIGN-OBJECT form, then I expand the WITH-ALIEN form, but regret it and collapse it back. Then I proceed to expand everything else, including compiler macros!

A nice feature of slime-macrostep is that the expansions are annotated to show which forms are further expandable and macrostep-expand will jump automatically to the next expandable form. That's what makes it a stepper: pressing e repeadly will step through the macroexpansion. Plus, it expands macrolets!

If you'd like to try it out, please grab SLIME's bleeding edge (via MELPA or Git). It's enabled by default if you use the slime-fancy meta contrib. Feedback is most welcome.

Common Lisp Recipes: A Problem-Solution Approach

2016-01-04T11:18:00.001+00:00

Edi Weitz, author of things like CL-PPCRE and Hunchentoot (amongst many other pieces of ediware), has just published a new book, “Common Lisp Recipes: A Problem-Solution Approach”. I've ordered mine. Get yours from Apress or a European Amazon store.

Common Lisp Recipes is a collection of solutions to problems and answers to questions you are likely to encounter when writing real-world applications in Common Lisp. Written by an author who has used Common Lisp in many successful commercial projects over more than a decade, this book is the first Common Lisp book which covers areas as diverse as web programming, databases, graphical user interfaces, communication with other programming languages, multi-processing, and mobile devices as well as debugging techniques and optimization, to name just a few. It is organized around specific problems or questions each followed by ready-to-use example solutions and clear explanations of the concepts involved, plus pointers to alternatives and more information. Each recipe can be read independently of the others and thus the book will hopefully earn a special place on your bookshelf as a reference work you always want to have within reach.
Common Lisp Recipes is aimed at programmers who are already familiar with Common Lisp to a certain extent but do not yet have the experience you typically only get from years of hacking in a specific computer language. It is written in a style that mixes hands-on no-frills pragmatism with precise information and prudent mentorship.
If you feel attracted to Common Lisp's mix of breathtaking features and down-to-earth utilitarianism, you'll also like this book.

My answer to the Pretty Printer Puzzle

2015-05-30T17:06:00.000+01:00

As promised, I'll describe my solution to the Pretty Printer Puzzle I proposed last week. To recap, we wish to pretty print a Lisp form to a string and identify the textual positions of arbitrary subforms therein.

First attempt

A couple of folks proposed a clever solution that goes like this: (1) replace the CAR of each subform with some unique token (a gensym should be close enough), (2) pretty-print that, (3) find the token positions and replace them with the original CARs.

Problem with this solution: changing the form affects pretty-printing. In particular, it will no longer be able to properly indent macros and special forms.

Second attempt

Another approach is to pretty-print then read the form back and track positions by either using a custom reader that keeps track of form positions (such as hu.dwim.reader) or instrumenting the standard readtable by wrapping the #\( reader-macro and doing the reading from a so-called form-tracking-stream.

Problem with this solution: it breaks down if the form contains unreadable objects.

Third attempt, getting closer

The pretty printer is customisable through a pprint-disptach-table. It is analogous to the reader's readtable. So, we try and instrument it like in the previous approach. Each time a list is about to be pretty-printed, we store the current position in the output stream.

Problem: we have been defeated by the pretty printer's intermediate buffer. Turns out the pretty printer only writes to the output stream at the very end of the process. Back to the drawing board.

Fourth and final attempt

But these attempts have not been in vain, and my final solution involves elements from all three. It goes like this:

Pretty print the form normally.
Pretty print the form again, this time instrumenting the pprint-dispatch-table to wrap lists with some token identifying the subform being printed. (I decided to use the unicode range U+E000..U+F8FF which is reserved for private-use, which seemed neat.) This messes up the pretty-printing a little bit, but not too much, it turns out.
Cross-reference the token positions in #2 with #1 by taking advantage of the fact these outputs differ by whitespace (and tokens) only!

And that's it!

With this tool in hand, there are some interesting utilities that can be built in SLIME, but that's another blog post. :-)

Pretty printer puzzle

2015-05-24T15:20:00.001+01:00

This past week, I came across a Lisp challenge that turned out to be trickier than one might expect at first. I needed to pretty-print a Lisp form to a string and identify the positions of certain subforms. Here's an example:

CL-USER> (with-output-to-string (*standard-output*)
           (pprint '(defun factorial (x) (if (zerop x) 1 (* x (factorial (1- x)))))))
"
(DEFUN FACTORIAL (X)
  (IF (ZEROP X)
      1
      (* X (FACTORIAL (1- X)))))"

In this, output the bounding indices of the IF form are 24 and 77. In other words:

CL-USER> (subseq * 24 77)
"(IF (ZEROP X)
      1
      (* X (FACTORIAL (1- X))))"

The challenge, then, is to write a function that, given a form and list of subforms, returns a string with the pretty-printed output and a list of bounding indices for each subform. E.g.

CL-USER> (pprinted-bounds '(defun factorial (x) #1=(if (zerop x) 1 (* x #2=(factorial (1- x)))))
                          (list '#1# '#2#))
"
(DEFUN FACTORIAL (X)
  (IF (ZEROP X)
      1
      (* X (FACTORIAL (1- X)))))"
((24 77) (57 75))

I'll post my solution later. Have fun. :-)

LOOP quiz

2014-12-19T18:27:00.000+00:00

Does (loop for i below 10 finally (return i)) return 9 or 10?
Does (loop for i upto 10 finally (return i)) return 10 or 11?
What does (loop for i below 10 for j upto 10 finally (return (list i j))) return?
What about (loop for i below 10 and j upto 10 finally (return (list i j)))?

I stumbled upon the semantics of this last example in a recent bugfix and thought it was worth sharing. (Reminded me of the joke about what's hard in CS, too.)

Turns out LOOP's FOR ... AND not only mimics LET (rather than LET*) in terms of binding visibility, it also influences when the loop termination checks take place. That was new to me. I initially expected examples 3 and 4 to return the same values. What about you? Which ones, if any, did you get wrong? :-)

P.S.: LOOP for Black Belts is my favorite LOOP tutorial.

paredit

2014-12-11T21:20:00.000+00:00

Taylor Campbell's paredit is one of those Emacs extensions that I can't live without. In a nutshell, it forces you to deal with Lisp code exclusively via operations that don't introduce unbalanced parenthesis (or other similar violations of structure). The genius about this approach is that it completely eliminates the step of making sure parentheses are properly balanced after you write or edit a piece of code. After you get used to paredit, performing — or even watching — manual parenthesis balancing becomes painful.

Recently, I've come across these two introductions to paredit:

So, if you're still not using paredit, have a look at those and give it a try. At first you might feel like the karate kid doing frustrating chores — you can always take a break with M-x paredit-mode — but I promise it'll soon pay off!

SLIME officially available via MELPA

2014-12-01T12:04:00.000+00:00

SLIME has been available from MELPA for a while, but only recently did we iron out some annoying bugs. Notably, upgrading the SLIME package no longer results in confusion about where SWANK is.

So, as of SLIME 2.11, once you have the melpa (or melpa stable) repository set up, installing and updating SLIME from within Emacs is pretty easy:

    M-x package-install RET slime RET

Enjoy!

Update: Oh, if you want to switch to MELPA, make sure you remove your old SLIME location from Emacs's load-path, otherwise the old version will take precedence.

(cons cat (cons cat nil))

2014-11-28T22:04:00.000+00:00

I thought this tweet was pretty funny, despite the diagrammatic inaccuracy that twitterers were quick to point out. :-) Reminds me of my second cat, whom I named CADR, of course.

SLIME 2.4

2014-03-02T14:40:00.003+00:00

SLIME 2.4 has been released without any exciting release management, but with extensive release notes! :-)

Call for SLIME testers

2014-02-23T16:40:00.000+00:00

As you may have heard, SLIME's recently moved to GitHub where, for the last 3 months or so, code has been refactored, bugs have been fixed and some new features have been introduced.

The last SLIME release has a very long beard, so we'd like to make a new one next weekend. To make it a good release, it'd be very really helpful if fellow SLIME users grabbed the latest code from github (via either git or the ZIP download link), took it for a spin and, if need be, used the issue tracker to report issues that may pop up.

Thanks!

SISCOG's ECLM 2013 slides (finally) up

2013-11-21T12:13:00.000+00:00

It took a while, but our ECLM 2013 slides “SISCOG: a story written in Lisp” have been published. Incidentally, they are also featured in Franz's ACL Success Stories, yay.

On a related note, Vsevolod Dyomkin published his video recordings of the meeting. Sadly, we didn't get permission to publish our video.

Finally, SISCOG is hiring and has open positions for Software Engineers and Operations Research Analysts that require C++ programming, which is rather ironic and anti-climatic for this blog post. :-)

ECLM 2013

2013-06-06T11:46:00.001+01:00

Another odd year, another ECLM. This time around in sunny Madrid at the posh TRYP Ambassador within walking distance from Teatro Real and Plaza Mayor. 63 people from 20 countries turned up.

On Saturday I failed to meet some other lispers for lunch. Apparently, there's two or three Museos del Jamón around Plaza Mayor and I stood for 20 minutes in front of the wrong one. Took the afternoon to tweak my slides. Dinner was nice for catching up with familiar faces from past meetings. Met some new lispers as well. Some of us went for a quick drink at Mercado de San Miguel afterwards.

Sunday started off with a visibly jet lagged Wes Henderson talking about mocl, a Lisp-to-C compiler with two backends tailored for the iPhone and Android environments. Despite the jet lag, he was quite eloquent! mocl makes it trivial to call Lisp from Java/ObjC:

    (declaim (call-in foo)) ; declares FOO can be called from foreign land
    (defun foo (x)
      (format t "~a~%" x))

After compiling this function, it will be available from the ObjC or Java environments. I didn't quite figure out how mocl deals with the mismatch between static and dynamic typing, but apparently that's taken care of. Note that the goal is for mocl to be embedded in Android/Java or iOS/ObjC applications. I.e., you'll be writing your GUIs in Java/ObjC and call out to Lisp for the remaining bits. I don't recall anything being mentioned about calling Java/ObjC from the Lisp side.

I enjoyed his thought out justification of why Common Lisp is a good match for mobile applications. First off, CL has a stable specification and a rich ecosystem with many alternative implementations. That means that, if he follows the spec, our existing code (namely libraries) will work as is. Secondly, CL is fast and that's important for resource-constrained mobile devices. (What does it mean for a language to be fast? I guess it means that there are plenty of implementations around that show how its semantics can be implemented efficiently.) Thirdly, the CL ecosystem has a relatively reduced dependency on foreign libraries which may or may not work on mobile environments. I was convinced. :-)

Wes took the opportunity to announce that mocl will be released at 3581000000 lisp universal time. (That's from memory, I hope I got it right.)

Michael Eisfeld presented ConED, an application to help engineers design the first stages of building structures.

There weren't many lispy details, but the application seemed quite useful and interesting. I'm told this is a hard market to get into, so best of luck to Michael and his team!

Michael Compton talked about Accenture's digital optimisation software, ADO. In a nutshell, ADO will feed various versions of a webpage to users and cleverly measure which ones work best. I didn't understand most of what was discussed, but apparently they're market leaders.

Then, Tiago Maduro Dias and I talked about the company where we work, SISCOG. Tiago presented SISCOG's history, problem domain and products and I tried to illustrate some interesting ways how we use Lisp. Thanks to Murphy's law, projector woes took up a fair chunk of our allotted time and I had to compress 25 minutes of talk into 15 or so. In the end, I think it went OK and the feedback was pretty positive. SISCOG is a 27 year old company with a rich Lisp history (going back to the Lisp Machines!) and it's quite successful in its market: resource planning, management and optimisation for railway companies. Oh, and we have about 70 full-time lispers.

Tapas for lunch.

After lunch, R. Matthew Emerson (rme) talked about Clozure CL's Objective-C bridge. I enjoyed his presentation style, and there were plenty of demos, which is nice. I had heard about CCL's IDE before, but this was the first time I got to see it.

Janusz Podrazik then demoed Opusmodus, written using CCL and the aforementioned ObjC bridge, which looked quite slick! It felt a bit like an Emacs-inspired IDE for writing music with lots of pretty graphs and visualizations. Plus, it lets you write music using Lisp! IIUC, the coding was done by rme who did a very fine job! Looking forward to its release this coming September.

Erik Huelsmann (ehu) talked about ABCL, whose version 1.2.0 RC had been released that morning from the venue. He basically talked about the latest developments and how ABCL is being used out there (including an application of his). I liked how the ABCL developers are using cl-test-grid to assess ABCL's compatibility with major libraries and how Quicklisp's download statistics helps them prioritise bug fixing.

Sven Emtell demoed ScoreCleaner. (I don't recall exactly which bits were written in Common Lisp. It was a very polyglot product, IIRC.) He showed nicely done marketing videos, such as this one.

At dinner, Faré tried to explain to me what Interface Passing Style was all about and how he used it in his Lisp Interface Library. He almost succeeded!

There were lightning talks throughout the day. Here are the ones I remember. Mark Evenson presented a very clear explanation of what went wrong with the latest common-lisp.net migration and how he and other folks plan to fix outstanding issues and generally improve how the logistics of this service are managed. Nick Levine talked about Ravenbrook. Christophe Rhodes released SBCL 1.1.8 live.

All in all, I think this was the best ECLM yet. Lots of cool applications and a vibrant community. Looking forward to the next one!

Kálmán Réti on Lisp Machines

2013-03-18T00:46:00.000+00:00

An interesting video popped up on the Lisp Reddit last week: Kálmán Réti, the Last Symbolics Developer, Speaks of Lisp Machines. The first half goes into satisfyingly low-level detail about the Lisp Machine's history and architecture, and the second half contains the most in-depth demo I recall watching. Enjoy.

CFFI 0.11.0 released

2013-03-09T18:55:00.000+00:00

CFFI 0.11.0 has been released on Feb 25th, 2013, but I only finished writing the release notes today:

new feature: fsbv has been integrated into CFFI, courtesy of Liam Healy. This means we now support passing structs by value via [libffi](http://sourceware.org/libffi/)! NB: in order to differentiate between structure passing by value from structure by reference, (:struct foo) and :pointer should be used. The old way of referring to struct types by bare name is now deprecated.
new feature: a new MKCL backend has been added courtesy of Jean-Claude Beaudoin.
enhancement: the ABCL backend has improved considerably thanks to Mark Evenson and Stas Boukarev.
enhancement: the ECL backend now supports multiple linking strategies thanks to Juanjo Garcia-Ripoll.
optimization: foreign-free is non-consing on SBCL. (Thanks to Stas Boukarev.)
bugfix: in some situations we were not signalling a warning upon defining a foreign type using a symbol from the :cl or :keyword packages. Because of this, some projects will see new compilation warnings that will require fixing.
and various other bugfixes.

This version will be included in the next Quicklisp dist.

Augmenting bordeaux-threads with atomic operations

2012-06-06T02:59:00.000+01:00

Bordeaux-threads is a portability layer that defines a low-level API for programming shared-state concurrency: basic thread management, locks, timeouts, and condition variables. It has been ported to plenty of CL implementations: ABCL, Allegro CL, CLISP, Clozure CL, CMUCL, Corman Lisp, ECL, Lispworks, MCL, SBCL, and SCL. As such, it's an ubiquitous building block for useful higher-level libraries such as lparallel, PCall, CL-STM, and many others.

One important piece missing from bordeaux-threads is the support for atomic operations such as compare-and-swap, fetch-and-add, atomic-swap, etc. Among other things, these operations are useful for implementing scalable, efficient lock-free or wait-free algorithms and data-structures. The availability of such data-structures in a portable fashion would benefit many of the higher-level libraries that use bordeaux-threads. (E.g.: lparallel, when running on SBCL, takes advantage of the lock-free queues provided by that implementation.)

Back when bordeaux-threads was created, virtually no CL implementation supported these operations.¹ Nowadays, though, more and more implementations support at least a subset thereof. What follows is a survey of that support in modern CL implementations, a necessary first step to add an atomic operation API to bordeaux-threads.

compare-and-swap

I'll start with compare-and-swap (CAS), which is the most powerful of the atomic operators we'll be looking into, and consequently the most widely supported in CL implementations. Typically, this operation is syntactically similar to SETF but is restricted to a handful of CL places:

SLOT-VALUE.
SYMBOL-VALUE, i.e. the value of a dynamic variable.
The global value of a dynamic variable, a common extension to standard CL.
Structure accessors.
CAR and CDR.
SYMBOL-PLIST.
SVREF.
MOP's STANDARD-INSTANCE-ACCESS and FUNCALLABLE-STANDARD-INSTANCE-ACCESS.

Table 1. CAS support per place.
	slotval	symval	gsymval	structslot	car/cdr	symplist	svref	mop
ACL 9.0b	✔	✖	✔	✔	✔	✔	✔	✖
CCL 1.7	✖	✔	✖	✔	✖	✖	✔	✖
LW 6.0	✔	✔	✖	✔	✔	✖	✔	✖
SBCL 1.0.54	✔	✖	✔	✔	✔	✔	✔	✔
SCL 1.3	✔	✔	✔	✔	✔	✔	✔	✖

(2012-06-07 update: fixed table to correctly reflect the fact that SCL does support SYMBOL-GLOBAL-VALUE as a CAS place.)

(2012-06-10 update: Nikodemus wrote in to point out that (cas (symbol-value '*foo*) ...) on SBCL modifies the special variable's global value rather than its dynamic binding. Fixed table accordingly.)

As far as I can tell, ABCL, CMUCL and ECL don't yet have any support for atomic operations. I didn't check MCL or Corman Lisp. Also, note that CCL's CAS — ccl::conditional-store — is in fact an unexported interface. All in all, the feature overlap amongst the Lisps that better support CAS (ACL, LW, SBCL, SCL) is not too bad.

Conclusion: bordeaux-threads should support CAS for SLOT-VALUE, CAR/CDR, structure accessors, and SVREF on ACL, LW, SBCL, and SCL. The other partially supported places could perhaps be supported where possible but with a compile-time style warning about their unportability; haven't decided yet whether this would be worth the trouble.

fetch-and-add

A concrete implementation of fetch-and-add is the x86 LOCK-prefixed XADD instruction. It atomically increments an integer using word-sized modular arithmetic, i.e. the addition can overflow and the result wraps.

SBCL has ATOMIC-INCF whose name is slightly misleading because it is pretty much a direct interface to the aforementioned XADD instruction (on x86, of course), and thus only works on word-sized places: structure slots with type SB-EXT:WORD or simple arrays with element-type SB-EXT:WORD.

Lispworks on the other hand, provides ATOMIC-FIXNUM-INCF. Although it also boils down to XADD and the addition will overflow, it works on all the same places as CAS since it handles plain fixnums.

Conclusion: bordeaux-threads can define ATOMIC-WORD-INCF and ATOMIC-FIXNUM-INCF, support those operations efficiently on SBCL and LW respectively, and emulate them on other Lisps using CAS.

atomic-swap

Only Lispworks provides this functionality (via ATOMIC-EXCHANGE). Not yet sure whether it's sensible to emulate this using CAS on other Lisps.

Coming next

Having surveyed the support for atomic operations, we'll proceed to define a useful interface and some slightly higher-level utilities around CAS. There's plenty of inspiration to be found in the various implementations and we should be able to draw various ideas from each.

(P.S.: hopefully this doesn't overlap too much with Nikodemus's MADEIRA efforts.)

¹ (2012-06-07 update) I hadn't realized this until Douglas Crosher wrote in to set some facts straight: turns out SCL has supported SMP (including compare-and-swap) for nearly a decade!

Setting up CommonQt on OSX

2011-12-09T09:39:00.001+00:00

CommonQt can be tricky to setup due to its dependencies from C++ land. Tricky enough to warrant writing down the steps I went through to get it running on Mac OS 10.7:

If you don't already have Xcode, install it because you'll need g++.
Grab and install the Qt Libraries (not the Qt SDK). I used version 4.7.4.
Install CMake. (I used brew install cmake.)

Download, compile and install the SMOKE library:

$ git clone git://anongit.kde.org/smokegen
$ cd smokegen
$ cmake .
$ make install

$ cd ..
$ git clone git://anongit.kde.org/smokeqt
$ cd smokeqt
$ cmake -DSmoke_DIR="$PWD/../smokegen/cmake" .
$ make install

At this point, smokephonon failed to build so I had to manually install the two modules I actually needed:

$ make -C qtcore install
$ make -C qtgui install

Get an SBCL with threads enabled and Quicklisp.
CommonQt needed a couple of tweaks for OSX and recent changes in SMOKE. While said changes aren't reviewed and integrated into the main repository, you can fetch them as follows:
```
$ cd ~/quicklisp/local-projects
$ git clone git://gitorious.org/~luismbo/commonqt/commonqt-luis.git
$ cd commonqt-luis
$ git checkout modular-smoke-and-osx-fixes
```
Start SBCL and (ql:quickload :qt).

Hopefully that went well. Next we'll try and run an application.

Enable the swank-listener-hooks contrib by adding (slime-require 'swank-listener-hooks) to your SLIME configuration.
(asdf:load-system :qt-tutorial)
(asdf:load-system :qt-repl)
(qt-repl:start-gui-thread)
And finally, (qt-tutorial-14::test)!

If not running under SLIME, (asdf:load-system :qt-tutorial) followed by (qt-tutorial-14:main) would be sufficient.

If you want to try and skip the C++ compilation steps, grab this tarball with libcommonqt.dylib and libsmoke*.dylib. You should place libcommonqt.dylib in CommonQt's source directory. The SMOKE libs go into /usr/local/lib or similar.

TIME on steroids

2011-11-16T23:20:00.004+00:00

Nikodemus's recent post about a yet-to-be-released micro-benchmarking tool prompted me to talk about Perfpiece. I describe it as a tool for measuring the performance of Lisp code, not unlike the standard CL:TIME and I wrote over two years ago to measure the performance of SBCL's garbage collector.

Features

Here are its main features:

Lispy interface to PAPI (Performance Application Programming Interface). PAPI is a library that enables access modern CPU’s hardware counters. This allows us to measure several events such as processor cycles, cache misses, number of floating-point instructions, and almost two hundred other events. Perfpiece dynamically inspects the current platform’s supported events at runtime and enables the user to inspect this list and measure these events. We also use this library to measure real (wall-clock) time and user (virtual) time.
Support for other non-PAPI events such as the number of GC runs, CPU usage, and operating system resource usage (via the getrusage() system call).
Transparent support for multi-threaded programs. It includes a helper C library that when loaded through the POSIX LD_PRELOAD mechanism will preempt pthread creation/termination calls and allow for the individual measurement of events across threads created during a measurement session. This includes both Lisp threads as well as threads created by C code. This is rather limited at the moment; only real/user/cpu time is measured for new threads.
Segregation of measurements between mutator and GC.
Support for sampling. Perfpiece can repeat a given a measurement a number of times then calculate and report basic statistic analysis: minimums, maximums, geometric means, and standard deviations for each measured event.

Usage

The simplest way to interact with this library is through the ascertain macro, which works very much like cl:time. The following example shows the default events measured for a very simple arithmetic form:

PERFPIECE> (ascertain (+ 1 1))

                                          non-GC            GC         Total
────────────────────────────────────────────────────────────────────────────
                     Total cycles:         8,343             0         8,343
           Instructions completed:           481             0           481
        Level 2 data cache misses:            78             0            78
 Level 2 instruction cache misses:            24             0            24
                         GC count:             -             -             0
     Involuntary context-switches:             0             0             0
       Voluntary context-switches:             1             0             1
                      Page faults:             0             0             0
                    Page reclaims:             6             0             6
                      System time:             0             0             0
                        CPU usage:       100.00%             -       100.00%
                        User time:       6.00 µs             0       6.00 µs
                        Real time:        613 ns             0        613 ns

0 new threads were spawned

2 ; printed result of (+ 1 1)

Having loaded the helper library using LD_PRELOAD, we can measure multi-threaded code:

PERFPIECE> (ascertain (loop repeat 2 do
                        (sb-thread:join-thread
                         (sb-thread:make-thread (lambda () (sleep 0.5)))))
                      :events '(:real-time :user-time :cpu-usage))

                                          non-GC            GC         Total
────────────────────────────────────────────────────────────────────────────
                        CPU usage:         0.02%             -         0.02%
                        User time:     206.00 µs             0     206.00 µs
                        Real time:       1.335 s             0       1.335 s

2 new threads were spawned
  #0 real: 667.54 ms, user: 101.00 µs, cpu: 0.02%
  #1 real: 667.60 ms, user: 144.00 µs, cpu: 0.02%

The other main function is sample. In the following example, we're measuring FP instructions, invoking some code 10 times, and aggregating measurements in several ways:

PERFPIECE> (sample (lambda () (* 2 pi)) :events '(:papi-fp-ins) :samples 10)

[Floating point instructions]       Min           Max          Mean     Stddev
──────────────────────────────────────────────────────────────────────────────
                   total:            35            36         35.60  ±  1.376%
                  non-gc:            35            36         35.60  ±  1.376%
                 gc-only:             0             0             0  ±  0.000%

sample's got a :report keyword argument that you can use to get machine-readable results:

PERFPIECE> (sample (lambda () (* 2 pi)) :events '(:papi-fp-ins) :samples 10
                   :report nil)

((:PAPI-FP-INS :MIN (35 35 0)
               :MAX (38 38 0)
               :MEAN (184/5 184/5 0)
               :STDDEV (1.0770329 1.0770329 0.0)))

Fork it!

That's pretty much it. There is some SBCL-specific code, but it shouldn't be too hard to port to other implementations. Its use of PAPI could be made optional since that makes this library pretty much Linux-only otherwise. Patches are most welcome!

ECLM videos

2011-11-16T19:07:00.001+00:00

Just a quick note: Vsevolod Dyomkin has posted a couple of ECLM 2011 videos with more to come.

ECLM 2011

2011-10-25T09:18:00.007+01:00

Contrary to my expectations earlier this year, ECLM 2011 happened and ~90 people turned up. Yay!

Saturday, a bunch of us went on a guided tour across Amsterdam. We had a great guide with an awesome sense of humor and this was a nice way to meet new Lispers and catching up with folks I'd met in past meetings.

Dinner was fun. One very prominent Lisper who shall remain anonymous didn't know what Quicklisp was. Everyone sitting at that particular table was shocked. :-)

Sunday started off with Nick Levine talking about learnt lessons from his failed attempt at writing a Lisp book. Midway through, he took the opportunity to rant a little bit about how there were at least 3 overlapping, incomplete websites (cliki, c-l.net, cl-user.net) and how the solution would be to, huh, create yet another one. The consensus seems to be that cliki is the one that's best suited for a starting point, but Nick complains that its focus on Free Software Lisps is too narrow. Anyway, cliki could indeed use a revamp.

Jack Harper talked about using Lisp on portable fingerprint scanners. He had great things to say about Lispworks which, by the way, had 3 people at this year's meeting.

Luke Gorrie did an extended version of the presentation on Teclo I had seen earlier this year in Zürich. This one included even more interesting diagrams, in particular some illustrating the contrast between 3G network performance before and after enabling their product.

After lunch, Dave Cooper talked about his GDL product, which includes yet another object constraint-ish system framework thingie.

Hans Hübner picked a controversial topic: code style and conventions. (A bit reminiscent of Norvig and Pitman's slides.) He picked on the veritable 80-column rule (blasphemy!) and pushed for project/company style guides. The discussion was entertaining. The general conclusion I drew from that is that while it might be useful to write the more important points down, it's even better to programmatically enforce the more important ones like tabs vs. spaces and trailing whitespace. In my experience, good programmers pick up and follow (or challenge) code style fairly quickly without needing to go through an explicit style guide.

Paul Miller demoed his company's data analysis tool written in Lispworks and talked a little bit about how it interacted and integrated with things like Excel via COM.

Lisp hero extraordinaire Xach Beane did an awesome presentation on Quicklisp detailing some of his implementation strategies, what problems Quicklisp purports to solve, its social impact on the Lisp community, and what his vision for the future is. I was particularly excited about his plan to enable hacking on random project à la clbuild. Definitely the juiciest talk in the meeting in my opinion.

Finally, there were lots of lightning talks this year:

Marco Antoniotti announced ELS 2012.
Christophe Rhodes's talked about R's lispiness and demoed his swankr project that brings R to SLIME and has nifty features like graphical presentations.
Erik Huelsmann announced ABCL 1.0! (There were three quite enthusiastic ABCL developers at the meeting.)
Pierre-Yves Baccou shared some thoughts on Zen, his X-server in ~5K lines of CL code.
... and many others including a valiant attempt at subverting the 5+2 minute rule.

Drinks, dinner, then drinks again. Hanging out with Lispers is the best part of ECLM and there was plenty of that. Looking forward to the next one already!

Figuring out the print size of a photo at 300 DPI

2011-08-14T17:00:00.000+01:00

I was looking for an easy way to tell the print size of a photo at 300 DPI. It wasn't as easy to find the answer as I'd expected. For future reference, here's a solution using ImageMagick:

identify -format "%[fx:w/300*2.54] x %[fx:h/300*2.54] cm" photo.jpg