First, here’s a picture of what you can expect to see when using this (click for larger version):

And here’s the code (stick this in your .vimrc):

That’s it, easy as pie, enjoy the folding for Clojure files.

There are a few bugs with this because I’ve never actually written a Vim plugin, so I kind of hacked my way through this until I got it working. I welcome any feedback from someone with vimscript development experience (I’m looking at you Brian Carper )

You can download the slides here, or check them out on slideshare:

I’ve also been testing the new 2.2 pre-release of VimClojure (actually using nailgun this time), and it is sweet so far.

The key (for me) to editing Clojure code in Vim is a combination of two plugins, VimClojure and slime.vim (see associated blog post). One of the difficult things is that slime.vim doesn’t actually exist anywhere on vim.org’s list of scripts, so it has to be downloaded from the aforementioned blog post. Stick it in the ~/.vim/plugins directory to install it.

First, VimClojure. I tend not to use Nailgun at all, some people like it, I don’t. So instead of the regular install for vimclojure, I copy over the files from the autoload, doc, ftdetect, ftplugin, indent and syntax folders to their respective Vim folders. If you think you’ll want the Nailgun functionality, you should use the installation instructions provided by Kotarak.

Now, add the settings you need for VimClojure to your .vimrc:

" Settings for VimClojure
let g:clj_highlight_builtins=1      " Highlight Clojure's builtins
let g:clj_paren_rainbow=1           " Rainbow parentheses'!

I have to say, rainbow parentheses’ is one of the best features of vimclojure, making it easy to see exactly what parentheses closes which statement:

Now that VimClojure is set up, time to set up the integration with Clojure’s REPL, to do that I use slime.vim. Slime.vim uses screen to send the input from your editor to any window in a running screen session, so to get started we’ll have to start up a screen session. To make it easier, you can name it something so you don’t have to look up the pid, I’ll call this session “clojure”:

‹ ~ › : screen -S clojure

If you didn’t name your session, or forgot what you named it, you can use screen -ls to look up all the screen sessions you’ve started:

‹ ~ › : screen -ls
There are screens on:
41837.clojure (Attached)
8970.ttys000.Xanadu (Attached)
8990.ttys001.Xanadu (Attached)
9010.ttys002.Xanadu (Attached)
4 Sockets in /tmp/screens/S-hinmanm.


Now, start a REPL in the screen terminal window (use ‘clj’ or ‘lein REPL’ or however you like to start a Clojure REPL). Next, open a clojure file with Vim, highlight a block of code (slime.vim will automatically select a paragraph if your cursor is in the middle of something like a defn), now, press Control-c + Control-c (Ctrl+c twice in a row). You should be prompted by Vim like this:

Enter the name of the screen term, if you named your session “clojure” you’d enter “clojure”, if you didn’t name it, use the pid number you see from the output of ‘screen -ls’, next it will ask for which window to send the output to:

If you’ve used screen before (and I’m assuming you have), this is the window number your REPL is running on. After you enter this information the plugin will send the paragraph/line of text to the REPL. From here on the session id and window will be cached, so hitting ctrl+c,ctrl+c again will immediately send whatever function the cursor is on to the REPL. You can also select a block of code using visual mode and use ctrl+c,ctrl+c to send everything selected to the REPL. If you used the wrong numbers, use ctrl+c,v (Control+c, then v) to have slime.vim ask you for the numbers again.

There you go, you now have a 1-way pipe from your Vim editor to any kind of REPL (be it Clojure, Ruby or Python). Here’s a couple of screenshots of the plugin in action:

I know this doesn’t even come close to the amount of integration the Emacs has using SLIME, but for me, this is exactly what I want out of a Clojure development environment, develop some code and be able to easily send it to a REPL. Hopefully a Vim user or two out there will find this setup useful.

UPDATE: If you’re interested in my full Vim setup for some reason, you can check it out here.

UPDATE 2: Want to automagically fold Clojure methods when using Vim? Check out this post.

In that post, I told you I was going to use clojure-opennlp to enhance a searching engine, so, let’s start with the problem!

The Problem

(Ideas first, code coming later. Be patient.)

So, let’s say you have a sentence:

“The override system is meant to deactivate the accelerator when the brake pedal is pressed.”

I took this sentence out of a NYTimes article talking about the recent Toyota recalls, just pretend it’s an interesting sentence.

Let’s say that you wanted to search for things containing the word “brake“; well, that’s easy for a computer to do right? But, let’s say you want to search for things around the word “brake“, wouldn’t that make the search better? You would be able to find a lot more articles/pages/books/whatever you might be interested in, instead of ones that only contained the word key word.

So, in a naïve (for a computer) pass for words around the key word, we can come up with this:

“The override system is meant to deactivate the accelerator when the brake pedal is pressed.”

Right. Well. That really isn’t helpful, it’s not like the words “when”, “the” or “is” are going to help us find other things related to this topic. We got lucky with “pedal”, that will definitely help find things that are interesting, but might not actually have the word brake in them.

What we really need, is something that can pick words out of the sentence that we can also search for. Almost any human could do this trivially, so here’s what I’d probably pick:

“The override system is meant to deactivate the accelerator when the brake pedal is pressed.”

See the words in red? Those are the important terms I’d probably search for if I were trying to find more information about this topic. Notice anything special about them? Turns out, they’re all nouns or verbs. This is where the NLP (Natural Language Processing) comes into play for this, given a sentence like the above, we can categorize it into it’s parts by doing what’s called POS-Tagging (POS == Parts Of Speech):

["The" "DT"] ["override" "NN"] ["system" "NN"] ["is" "VBZ"] ["meant" "VBN"] ["to" "TO"] ["deactivate" "VB"] ["the" "DT"] ["accelerator" "NN"] ["when" "WRB"] ["the" "DT"] ["brake" "NN"] ["pedal" "NN"] ["is" "VBZ"] ["pressed" "VBN"] ["." "."]

So next to each word, we can see what part of speech that word belongs to. Things starting with “NN” are nouns, things starting with “VB” are verbs. Doing this is non-trivial. (A full list of tags can be found here). That’s why there are software libraries written by people smarter than me for doing this sort of thing (*cough* opennlp *cough*). I’m just writing the Clojure wrappers for the library.

Anyway, back to the problem, so what I need to do is strip out all the non-noun and non-verb words in the sentence, that leaves us with this:

["override" "NN"] ["system" "NN"] ["is" "VBZ"] ["meant" "VBN"] ["deactivate" "VB"] ["accelerator" "NN"] ["brake" "NN"] ["pedal" "NN"] ["is" "VBZ"] ["pressed" "VBN"]

We’re getting closer, right? Now, searching for things like “is” probably won’t help, so let’s strip out all the words with less than 3 characters:

["override" "NN"] ["system" "NN"] ["meant" "VBN"] ["deactivate" "VB"] ["accelerator" "NN"] ["brake" "NN"] ["pedal" "NN"] ["pressed" "VBN"]

Now we get to a point where we have to make some kind of decision about what terms to use, for this project, I decided to weight the terms that were nearer to the original search term, only taking two of the closest words in each direction, after scoring this sentence you get:

["override" 0] ["system" 0] ["meant" 0] ["deactivate" 0.25] ["accelerator" 0.5] ["brake" 1] ["pedal" 0.5] ["pressed" 0.25]

The red next to each word indicates how heavily we’ll weight each word when we use it for subsequent searches. The score is divided by 2 for each unit of distance away from the original key term. Not perfect, but it works pretty well.

That was pretty easy to follow, right? Now imagine having a large body of text, and a term. First you’d generate a list of key words from sentences directly containing the term, score them each and store them in a big map. On a second pass, you can start scoring each sentence by using the map of words => scores you’ve already built. In this way, you can then rank sentences, including those that don’t even contain the actual word you’ve searched for, but are still relevant to the original term. Now, my friend, you have context searching.

Congratulations, that was a really long explanation. It ended up being a bit more like pseudocode, right? (or at least an idea of how to construct a program). Hopefully after understand the previous explanation, the code should be easy to read.

The Code

(note: this is very rough code, I know it’s not entirely idiomatic and has a ton of room for enhancement and parallelization, feel free to suggest improvements in the comments however!)

The most important functions to check out are (score-words [term words]) method, which returns a list of vectors of words and their score, and the (get-scored-terms ) method, which returns a map of words as keys and scores as values for the entire text, given the initial term.

Here’s the output from the last few lines:

contextfinder=> (pprint (reverse (sort-by second (score-sentences mytext scorewords))))
(["The override system is meant to deactivate the accelerator when the brake pedal is pressed. " 13/4]
["The Obama administration is considering requiring all automobiles to contain a brake override system intended to prevent sudden acceleration episodes like those that have led to the recall of millions of Toyotas, the Transportation secretary, Ray LaHood, said Tuesday. " 5/2]
["Often called a \"smart pedal,\" the feature is already found on many automobiles sold worldwide, including models from BMW, Chrysler, Mercedes-Benz, Nissan and Volkswagen." 3/4]
["That will let the driver stop safely even if the cars throttle sticks open. " 0])

I highlighted the score for each sentence in red, see how sentences such as the third in the list have a score, but don’t contain the word “brake”? Those would have been missed entirely without this kind of searching.

contextfinder=> (println (score-text mytext scorewords))
13/2

Score a whole block of text, self-explanatory.

Anyway, I’ve already spent hours writing the explanation for why you’d want to do this sort of searching, so I think I’ll save the explanation of the code for another post. Hopefully it’s clear enough to stand on its own. If you find this interesting, I encourage you to check out the clojure-opennlp bindings and start building other cool linguistic tools!

From Wikipedia:

Natural Language processing (NLP) is a field of computer science and linguistics concerned with the interactions between computers and human (natural) languages. Natural language generation systems convert information from computer databases into readable human language. Natural language understanding systems convert samples of human language into more formal representations such as parse trees or first-order logic structures that are easier for computer programs to manipulate. Many problems within NLP apply to both generation and understanding; for example, a computer must be able to model morphology (the structure of words) in order to understand an English sentence, and a model of morphology is also needed for producing a grammatically correct English sentence.

Clojure-opennlp is a library to interface with the OpenNLP (Open Natural Language Processing) library of functions, which provide linguistic tools to perform on various blocks of text. Once a linguistic interpretation of text is possible, a lot of really interesting applications present themselves. Let’s jump right in!

Basic Example usage (from a REPL)

(use 'clojure.contrib.pprint) ; just for this example
(use 'opennlp.nlp) ; make sure opennlp.jar is in your classpath

You will need to make the processing functions using the model files. These assume you’re running from the root project directory of the git repository (where some models are included). You can also download the model files from the opennlp project at http://opennlp.sourceforge.net/models/

user=> (def get-sentences (make-sentence-detector "models/EnglishSD.bin.gz"))
user=> (def tokenize (make-tokenizer "models/EnglishTok.bin.gz"))
user=> (def pos-tag (make-pos-tagger "models/tag.bin.gz"))

For name-finders in particular, it’s possible to have multiple model files:

user=> (def name-find (make-name-finder "models/namefind/person.bin.gz" "models/namefind/organization.bin.gz"))

The (make-<whateverizer> "modelfile.bin.gz") functions return functions that perform the linguistic offering. I decided to have them return functions so multiple methods doing the same sort of action could be created with different model files (perhaps different language models and such) without having the pass the model file every time you wanted to process some text.

After creating the utility methods, we can use the functions to perform operations on text. For instance, since we defined the sentence-detector as ‘get-sentences’, we can us that method to split text by sentences:

user=> (pprint (get-sentences "First sentence. Second sentence? Here is another one. And so on and so forth - you get the idea..."))
["First sentence. ", "Second sentence? ", "Here is another one. ",
"And so on and so forth - you get the idea..."]
nil

Or split a sentence into tokens using the tokenize function:

user=> (pprint (tokenize "Mr. Smith gave a car to his son on Friday"))
["Mr.", "Smith", "gave", "a", "car", "to", "his", "son", "on",
"Friday"]
nil

Once we have a sequence of tokens, we can do what’s called POS Tagging. POS Tagging takes a list of words from only one sentence and applies an algorithms (using the morphology model) to determine what kind of tag to apply to each word:

user=> (pprint (pos-tag (tokenize "Mr. Smith gave a car to his son on Friday.")))
(["Mr." "NNP"]
["Smith" "NNP"]
["gave" "VBD"]
["a" "DT"]
["car" "NN"]
["to" "TO"]
["his" "PRP$"]
["son" "NN"]
["on" "IN"]
["Friday." "NNP"])
nil

You can check out a list of all the tags if you want to know what they stand for.

The clojure-opennlp library also features a name finder, however it is extremely rudimentary at this point and won’t detect all names:

user=> (name-find (tokenize "My name is Lee, not John."))
("Lee" "John")

Filters

In the library, I also provide some simple filters that can be used to pare down a list of pos-tagged tokens using regular expressions. There are some preset filters available, as well as a macro for generating your own filters:

(use 'opennlp.tools.filters)

user=> (pprint (nouns (pos-tag (tokenize "Mr. Smith gave a car to his son on Friday."))))
(["Mr." "NNP"]
["Smith" "NNP"]
["car" "NN"]
["son" "NN"]
["Friday" "NNP"])
nil
user=> (pprint (verbs (pos-tag (tokenize "Mr. Smith gave a car to his son on Friday."))))
(["gave" "VBD"])
nil

Creating your own filter:

user=> (pos-filter determiners #"^DT")
#'user/determiners
user=> (doc determiners)
-------------------------
user/determiners
([elements__52__auto__])
Given a list of pos-tagged elements, return only the determiners in a list.
nil
user=> (pprint (determiners (pos-tag (tokenize "Mr. Smith gave a car to his son on Friday."))))
(["a" "DT"])
nil


Check out the filters.clj file for a full list of out-of-the-box filters.

That’s about all there is in the library at the moment, so I hope that made sense. Unfortunately clojars.org does not provide a nice way to public documentation for a library, so the documentation in this post and on the github page will have to do for now.

This library is available on clojars for inclusion in leiningen projects, or on github if you’re interested in the source. This is a fairly new project, and not all OpenNLP features are exposed at the moment so feedback is definitely encouraged. In the next post I’ll explain an in-depth example of how these functions can be used to enhance a searching engine. EDIT: It’s up! Check out “Context searching using clojure-opennlp.”

UPDATE: Hiredman has let me know that the jar on clojars is missing the 3 dependencies used for the library. I’m busy working on writing pom.xml’s for the jars so I can upload them to clojars as dependencies. In the meantime, make sure you have the 3 jars in the lib directory (of the github project) in your classpath. Feel free to report any other issues on the github tracker or in the comments.

UPDATE 2: I fixed the project.clj file and pushed new versions of opennlp.jar and the dependency jars. A regular ‘lein deps’ should work now.

Git has nice colored diffs, like this:

Perforce does not:

A simple script can remedy this:

Here’s the script:

Simply drop it somewhere in your $PATH and use it like so:

p4 diff | p4c.rb

You can find it in my one-offs directory (it’s called p4c.rb). We recently switched from subversion to perforce (definitely not my choice, I pushed for git), and so far it’s awful. I am definitely not a fan. This makes it a little better.

(time (dotimes [_ 10000] (my-function arg1 arg2 arg3))

I normally end up writing something equivalent to that all the time to test reflection vs. type-hinted Clojure code, subtle tweaks and other things. Well, I finally got fed up extrapolating the information I wanted from the output of time (and time didn’t expressly return a number I could do anything with, just printed a string), so I wrote a small library for doing benchmarks a number of times. I called it CBench because I couldn’t think of anything that sounded Clojure-ish (bench-jure? clo-bench? etc).

Anyway, jumping straight in, here’s how it’s used, let’s say you have a couple of functions you want to test (super-simple examples):

(defn inc-no-hint [n] (inc n))
(defn inc-with-hints [#^Integer n] (inc n))


With cbench, you can perform benchmarking with a number of iterations. CBench provides 2 functions: cbench and cbench-pp. cbench-pp performs exactly like cbench, but prints a nice display instead of returning a vector of results. In this example, we’ll use cbench-pp:

(use 'cbench) ; cbench.jar must be in your classpath
(cbench-pp 10000 #(inc-no-hint (rand-int 100)))
(cbench-pp 10000 #(inc-with-hints (rand-int 100)))

Run this in a REPL (or a script), and you’ll see this output:

-----------------------------------
| avg    | 0.0161 ms
| min    | 0.0 ms
| max    | 26.0 ms
| stddev | 0.17861615575831605 ms
| total  | 570 ms
-----------------------------------
-----------------------------------
| avg    | 0.0012 ms
| min    | 0.0 ms
| max    | 8.0 ms
| stddev | 0.04897999523196905 ms
| total  | 238 ms
-----------------------------------

CBench gives the average time a run took, minimum, maximum, standard deviation and total time for running 10000 runs of each function. In this example we can see that type-hinting has made a > 10x improvement in average run time (note: this is a contrived example which doesn’t actually mean much other than hinting is faster than non-hinting).

If we wanted only the results as numbers, use the cbench function, in a REPL you can see the output:

user=> (cbench 10000 #(inc-with-hints (rand-int 100)))
[0.0032 22.0 0.0 0.07995998518952234 52]

The result is a vector (in the same order as the pretty-print version).

If you’re interested, the library is available from Clojars.org here, you can get just the jar, or you can check out the source code at the Github project page. Feel free to drop me an email, IM or github message if you like/dislike the library.

Oh yea, and on a non-technical note, I started a Tumblr blog so I can paste things I find interesting of a non-technical sort. If you’re into that sort of thing, feel free to check it out.

I started off creating a text-replacement robot in Java, but decided to switch it over to Clojure and to make something semi useful. I ended up making a robot to calculate Dungeons & Dragons dice rolls (so something like ’3d8′ means roll an 8-sided die 3 times and add the results). The idea being you could type something like:

“I cast Magic Missle, 2d6″

and this robot would replace it with the text:

“I cast Magic Missle, 2d6 (5)”

Where 5 is the number that was given. See screenshot below:

So, let’s go through how to make this robot. I’m going to end up skipping most of the gritty details of getting the actual robot set up and refer you to this well-written post about the subject, be sure to follow all the instructions and you should have a usable clojure-bot for Google Wave. This post will cover a basic explanation of the (basic) code used for the actual robot.

Finished reading the other post and playing with the code? Cool, here’s what my servlet.clj looks like:

So, let’s go through each function and explain it.

(ns bibly.servlet
  (:gen-class :extends com.google.wave.api.AbstractRobotServlet)
  (:import
    [com.google.appengine.api.users UserServiceFactory]
    [com.google.wave.api Blip Event EventType RobotMessageBundle TextView Wavelet])
  (:require
            ))

The (ns bibly.servlet) code puts us into the bibly.servlet namespace and inside that namespace, we’re doing 3 things:

1. Generating a class that extends Google’s AbstractRobotServlet class. (Exactly what you would do in order to build a robot in Java).
2. Importing Google’s java object
3. Requiring the parts of the clojure-contrib library that we need.

(defn blip-submitted-events
  [events]
  (filter (fn [e] (= (EventType/BLIP_SUBMITTED) (.getType e))) events))

This function (taken directly from spanx) is a filter that returns only the events that match EventType.BLIP_SUBMITTED.

(defn roll-die
  "Roll a die with  sides."
  [sides]
  (+ 1 (rand-int sides)))

This function returns a dice roll (Integer) for a die with <sides> sides. (I removed the commented out line)

(defn calculate-rolls
  "Given a re-pattern match group, calculate the dice roll"
  [txt]
  (let [string (first txt)
        dice (Integer. (second txt))
        sides (Integer. (nth txt 2))]
    (str string " (" (reduce + (take dice (repeatedly #(roll-die sides)))) ")")))

Calualte-rolls is the main function for determining the roll, a vector is passed in, that looks like ["1d6" "1" "6"], this function then assigns each of the 3 values to a variable. Calculate-rolls then repeatedly calls (roll-die sides) dice times, reducing the list of results by adding them together. For example, if this function were passed ["2d4" "2" "4"] , it would take a list of 2 (dice) rolls of a die with 4 (sides), and add them together to get the new value.

After reducing to the result integer, a string concatenating the result to the new string is returned. So for the example vector ["1d6" "1" "6"] (roll a 6-sided die once), the result would be a string: "1d6 (4)".

(defn roll
  "Given a text DnD dice roll, replace it with the calculated result."

  (str-utils2/replace text (re-pattern #"(\d+)d(\d+)") calculate-rolls))

The roll function takes a string containing a XdY statement, and expands the roll inside of the string.

(defn replace-rolls
  "Replace all the rolls in a given blip."
  [blip]
  (let [view (.getDocument blip)
        text (.getText view)]
    (.replace view (roll text))))

Replace-rolls does the exact thing roll does for a string, just doing it for a given blip. First we get the document contained by a blip, and the text in that document, then replace it with the text that the (roll "...") function returns.

(defn -processEvents
  [this bundle]
  (let [wavelet (.getWavelet bundle)]
    (doseq [event (blip-submitted-events (.getEvents bundle))]
      (replace-rolls (.getBlip event)))))

Finally, the -processEvents function, which is the function google wave automatically calls when a capability matching those in capabilities.xml happens, is this function we perform the (replace-rolls ...) function on the list of events filtered by the (blip-submitted-events ...) function.

Well, I hope this example makes sense. Again, 95% of the work for this goes to Spanx‘s post for getting the initial setup, I figured it would be nice to have another example explaining the actual Clojure code for a simple robot’s inner workings.

You can download the code for this DnD robot from my github project, or Spanx’s censorbot example from his github project.

This example isn’t perfect, it needs a lot of work before it’s actually useful, but I thought it was neat.

Feel free to look up the code or fork the project from the Github Forkify project.

The case for pool forking

So, why even do pooling instead of serial forking? Let’s look at an example:

[1, 2, 6, 1, 2, 3].forkify(3) { |n| sleep(n); }

What happens in the background when this is run – forkify will fork 3 processes at a time, each one sleeping for the array’s number, so for the first pass 3 processes will be created, sleeping for 1 second, 2 seconds and 6 seconds (respectively). Forkify will then do a Process.waitpid(<pid>) on all 3 of the processes.

It will end up waiting a tiny bit over 6 seconds (because the longest process will take 6 seconds to finish).

Next, it will create 3 more processes, which will sleep for 1, 2 and 3 seconds respectively.

This will take a bit over 3 seconds.

So, the total time will end up being a little over 9 (6 + 3) seconds (the extra is overhead). Seen here:

% time ruby -I lib -rforkify -e '[1, 2, 6, 1, 2, 3].forkify(3) { |n| sleep(n); }'
0.08s user 0.10s system 1% cpu 9.131 total

How it works with a pool

Let’s take the same example and see how it works for pool forking:

[1, 2, 6, 1, 2, 3].forkify(:procs => 3, :method => :pool) { |n| sleep(n); }

First, 3 processes will be created to perform duties, they will be given 1, 2 and 6 seconds to sleep for. The parent thread will immediately wait for them to finish. The first process will sleep for 1 second, check for work, grab another ’1′ from the Tuplespace and start doing the work immediately. This will continue with the second process (which will finish after 2 seconds and grab more “work to be done” from the Tuplespace).

This ends up taking less time because you don’t block waiting for 1 process that takes a tremendous amount of time while the others have been finished for a while:

% time ruby -I lib -rforkify -e '[1, 2, 6, 1, 2, 3].forkify(:procs => 3, :method => :pool) { |n| sleep(n); }'
0.12s user 0.11s system 2% cpu 8.380 total

Not a lot of difference huh? A lot of the different has to do with the overhead for creating a Tuplespace and sharing it between processes, here’s a better example (pool first, then serial):

% time ruby -I lib -rforkify -e '[1, 1, 6, 1, 5, 1].forkify(:procs => 3, :method => :pool) { |n| sleep(n); }'
0.12s user 0.10s system 3% cpu 7.157 total

% time ruby -I lib -rforkify -e '[1, 1, 6, 1, 5, 1].forkify(3) { |n| sleep(n); }'
0.08s user 0.10s system 1% cpu 11.112 total

Imagine if the array had 100 items in it, but only a few took a really long time to process, as the differences increase, pool forking makes more sense to do.

How it works in the code

The Parent’s job:

(This is extremely stripped down to illustrate what’s happening)

The key here is to make sure that all the work items are in the TupleSpace before forking any children, so they don’t pull while we’re in the middle of writing and mess up the order, this is why I write all the tuples before starting the DRb service with the TupleSpace object.

After starting the children, I only need to wait on 3 pids, then I retrieve the results from the same TupleSpace in the correct order (regardless of the order they were added in)

The Child’s job:

(Again stripped down to essentials)

The interesting parts of this are the TupleSpaceProxy and being able to write objects to a TupleSpace without any kind of mutex. Rinda’s TupleSpaces take care of doing operations multi thread/process-safely, so I don’t have to worry about synchronizing the writing/taking of items out of the TupleSpace.

The child pulls items out of the TupleSpace, calls the block on them and writes the result tuple back into the same TupleSpace, since it’s shared over DRb the parent will be able to see all of the result tuples when it grabs the results.

Are there problems with this code? Yes – hardcoding a port is a bad idea and I’ve run into some issues with deadlock if 2 pooled forkifys are run immediately after each other (even a ‘puts “foo”‘ in between them fixes the problem, which makes me crazy when debugging). I also imagine that while the forkify itself works with processes, it is (ironically) not actually thread or process safe (because of the DRb hardcoded port amongst other things).

So, does anyone have any ideas of how I can do a producer/consumer pool-forking library? Please let me know Sadly there is a criminally-small amount of Ruby Rinda documentation available online right now (most of it is for RingServers instead of TupleSpaces)

Check out the full forkify_pool method and let me know if you have any suggestions. Pool forking requires Ruby 1.9 right now because some of the Rinda stuff doesn’t work in 1.8.

Anyway, if you found this, you probably don’t want to listen to me blabber, you want to know how to get it running on Ruby 1.9.x, so I humbly present a patch!

Download the patch here (older version) version 2

Download the patch here, version 3.

First, download the FastRI tarball (version 0.3.1 is latest at the time of this writing) from rubyforge.

Here’s how to install the patch:

% tar zxf fastri-0.3.1.tar.gz
% cd fastri-0.3.1
% patch -p1 < ../fastri-0.3.1-ruby1.9.1.v2.patch
patching file bin/fastri-server
patching file lib/fastri/ri_index.rb
patching file lib/fastri/ri_service.rb
patching file lib/fastri/util.rb
patching file test/test_functional_ri_service.rb
% sudo ruby setup.rb

And you’re done! Now you’ll need to make sure to do a ‘fastri-server -b‘ to build the cache, and from there you can use FastRI the same as it was in Ruby 1.8, see:

[hinmanm@Xanadu:~]% qri String.each_codepoint
-------------------------------------------------- String#each_codepoint
str.each_codepoint {|integer| block }    => str
From
------------------------------------------------------------------------
Passes the Integer ordinal of each character in str, also known as
a codepoint when applied to Unicode strings to the given block.
"hello\u0639".each_codepoint {|c| print c, ' ' }
produces:
104 101 108 108 111 1593

Notes/Caveats/Disclaimers & Warnings:

This patch is a WORK IN PROGRESS, it may hang, crash, steal your wallet and/or delete files. I hacked it up in a day looking at FastRI because I really like using ‘qri’ for everything and was consistently annoyed by getting docs intended for 1.8. I assume no responsibility for these things.

Also, note that all tests don’t pass with this patch, I’m still working on it.

Feel free to leave any feedback for me

UPDATE:
New version of the patch that fixes doing a lookup on a base class instead of a method. so… version 3 right now. (link above)

UPDATE2:
I created a github project for fastri to build a gem, so you can install the gem (without patching!) using this command:

sudo gem install dakrone-fastri -s http://gems.github.com