A common technique in MapReduce is to input a group of records, calculate a value from that group, and emit each record with the new value attached. While this is easy to do in raw MR jobs, the solution in Cascading is not very obvious. This tutorial introduces a new operation to Cascading called BufferedSum. BufferedSum allows us to calculate values from a group of tuples and emit the group value to individual tuples in a scalable way.

Describing the operation of BufferedSum is clearer when discussed in concrete terms, so let’s work with an example.

Example

When dealing with large amounts of documents in Hadoop, its common to have each input file to contain many documents. Our input file in this case will contain two documents:

a.txt\thello world world
b.txt\tgoodbye goodbye world

Lets say we want to calculate tf-idf for these documents. One of the first values we need is the count of the occurrence particular term within each document.

First, we will split each line into (document_id, body) pairs:

  pipe = new Each(pipe, 
      new Fields("line"), 
      new RegexSplitter(new Fields("document_id", "body"), "\t"));

From there we “tokenize” the document and extract each term:

  pipe = new Each(pipe, // tokenize words by space
      new Fields("body"),
      new RegexSplitGenerator(new Fields("term"), "\\s+"), 
      new Fields("document_id", "term"));

Now our tuple stream is the following:

a.txt hello
a.txt world
a.txt world
b.txt goodbye
b.txt goodbye
b.txt world

Count of term in document_id

We now have (document_id, term) and we want to calculate (document_id, term, term_count_in_document). With Cascading, this is easy, simply group by document_id and term and use the Count() function:

  // count how many times `term` appears in `document_id`
  pipe = new GroupBy(pipe, new Fields("document_id", "term"));
  pipe = new Every(pipe, 
      new Fields("term"), 
      new Count(new Fields("term_count_in_document")), 
      new Fields("document_id", "term", "term_count_in_document"));

Calculating total_terms_in_document

So far, so good. Up to this point Cascading has provided everything we need. However, next we want to get the total terms within each document and keep the tuples we have calculated thus far. Put another way, we have an input of (document_id, term, term_count_in_document) and we want to emit (document_id, term, term_count_in_document, total_terms_in_document)

Our first instinct might be to use GroupBy() and Count() like before. But there is a catch: Every operations emit the operator result with the group tuple (see the Each and Every Pipes in the Cascading User Guide).

This means if we group by document_id and Sum() the total_terms_in_document we will emit (document_id, total_terms_in_document). The number in total_terms_in_document will be accurate, but we lose our term and term_count_in_document.

If we try to save our other fields by grouping on all three of them (document_id, term, term_count_in_document) then we’ve “over-grouped” and every “group” is a single tuple (the input tuple) and we won’t get the count of terms in the document as a whole. BufferedSum was created to solve this problem.

BufferedSum

BufferedSum takes as its input three things:

• The name of the Field to output
• The name of the Field to sum
• The other Fields to “pull through” the operation

Here is how we can use BufferedSum to achieve the desired effect:

// input: (document_id, term, term_count_in_document)
// emits: (document_id, term, term_count_in_document, total_terms_in_document) 
pipe = new GroupBy(pipe, new Fields("document_id"));
pipe = new Every(pipe, 
    new BufferedSum(new Fields("total_terms_in_document"), 
                    new Fields("term_count_in_document"),
                    new Fields("document_id", "term", "term_count_in_document")), 
    Fields.SWAP);

Note: the output selector Fields.SWAP is critical due to Cascading tuple selection.

Memory considerations

One thing to be careful of when using BufferedSum is to try and keep your groups small enough to fit in memory. However, this is not a requirement. BufferedSum uses Cascading’s SpillableTupleList which will spill to the HDFS if it grows too large. That said, spilling is an expensive operation and should be avoided if possible.

Summary

BufferedSum is a widely useful operation when dealing with sums in Cascading. In Part 2 we will use BufferedSum and Cascading to finish calculating tf-idf.

The Code

package com.xcombinator.cascading.operations.buffers;
import cascading.flow.FlowProcess;
import cascading.operation.BaseOperation;
import cascading.operation.Buffer;
import cascading.operation.BufferCall;
import cascading.tuple.Fields;
import cascading.tuple.Tuple;
import cascading.tuple.TupleEntry;
import cascading.tuple.SpillableTupleList;
 
import java.util.Iterator;
 
/**
 * BufferedSum sums a value for every Tuple in a Group and emits every input
 * Tuple with the sum appended.
 * <p/>
 * 
 * 
 * EXAMPLE:
 *
 * {@code 
 *
 * // input: (document_id, term, term_count_in_document)
 * // emits: (document_id, term, term_count_in_document, total_terms_in_document) 
 *
 *     pipe = new GroupBy(pipe, new Fields("document_id"));
 *     pipe = new Every(pipe, 
 *         new BufferedSum(new Fields("total_terms_in_document"), 
 *                        new Fields("term_count_in_document"),
 *                        new Fields("document_id", "term", "term_count_in_document")), 
 *         Fields.SWAP);
 * }
 *
 * @see BufferedSum
 * 
 */
 
public class BufferedSum extends BaseOperation implements Buffer
  {
  private Double sum;
  private SpillableTupleList list;
  private Fields extrasSelector;
  private Fields fieldToSum;
 
  /**
   * Returns a BufferedSum Buffer Operation. 
   *
   * @param emittedSumFieldName a {@link Fields} naming the field to emit the sum value
   * @param fieldToSum          a {@link Fields} naming the field to sum
   * @param extrasSelector      a {@link Fields} naming the other fields to "pull through". These fields *must* be of the same order and size as the input Tuple
   */
  public BufferedSum( Fields emittedSumFieldName, Fields fieldToSum, Fields extrasSelector )
    {
    super( extrasSelector.append( emittedSumFieldName ) );
    this.extrasSelector = extrasSelector;
    this.fieldToSum = fieldToSum;
    }
 
  public void operate( FlowProcess flowProcess, BufferCall bufferCall )
    {
    Iterator<TupleEntry> iterator = bufferCall.getArgumentsIterator();
    sum = 0.0D;
    list = new SpillableTupleList( 10000 );
 
    while( iterator.hasNext() )
      {
      TupleEntry arguments = iterator.next(); // must be called
      sum += arguments.getDouble( this.fieldToSum.get(0) );
      list.add( arguments.getTuple() );
      }
 
    for( Tuple tuple : list )
      {
      bufferCall.getOutputCollector().add( tuple.append( new Tuple( sum ) ) );
      }
 
    }
  }

Share:

Its pretty easy once you know how to do it:

• Create a JobConf that defines the parameters for the streaming job
• Send up the hadoop-*-streaming.jar with your cascading job by putting it in your jar
• Send up the scripts (python, in this case) by using the -file option
• Send up any other dependencies, corpora, etc. by using the -file, -cacheFile, or -cacheArchive options (See the Hadoop Streaming page for more details)

Resources

NLTK

To generate the nltkandyaml.mod zip file do the following:

# download nltk and unzip
cd nltk
zip -r nltkandyaml.zip nltk yaml
mv nltkandyaml.zip nltkandyaml.mod

Note that this technique is taken from Cloudera

WordNet

The WordNet zip file needs to be flat. e.g. don’t zip up the files with a subdirectory. You could create this file like so:

# download and unzip the wordnet corpus
cd wordnet
zip -r ../wordnet-flat.zip *

Streaming Script

In python, we’ll be using zipimport.zipimporter to import the nltk libraries from a zip file. In Hadoop 0.20.0, Hadoop didn’t decompress our wordnet-flat.zip file automatically (but we’ve heard reports that it will, but I’m not sure which versions). For us the .zip file was placed in lib relative to the pwd of the script. This allowed us to keep the WordNet corpus as a zip and read it in that format.

wn = WordNetCorpusReader(nltk.data.find('lib/wordnet-flat.zip'))

(In this code we’re not using the python reducer.)

#!/usr/bin/env python 
import os
import re
import sys
import zipimport
 
importer = zipimport.zipimporter('nltkandyaml.mod')
yaml = importer.load_module('yaml')
nltk = importer.load_module('nltk')
punct = re.compile('[^\w\s]+')
 
from nltk.corpus.reader import wordnet
from nltk.corpus.reader import WordNetCorpusReader
 
nltk.data.path += ["."]
wn = WordNetCorpusReader(nltk.data.find('lib/wordnet-flat.zip'))
 
def mapper(args):
  line = sys.stdin.readline();
  try:
    while line:
      line = line.strip()
      word = line
      all_synonyms = []
 
      string_synsets = wn.synsets(word)
 
      for synset in string_synsets:
        synonyms = [lemma.name for lemma in wn.synset(synset.name).lemmas]
        synonyms.pop(0)
        for synonym in synonyms:
          synonym = re.sub("_", " ", synonym)
          all_synonyms.append(synonym) 
 
      print "\t".join([word, ','.join(all_synonyms)])
      line = sys.stdin.readline()
  except "end of file":
    return None
 
# we're not using this, but we could
def reducer(args):
  for line in sys.stdin:
    line = line.strip()
    print line
 
if __name__ == "__main__":
  if sys.argv[1] == "mapper":
    mapper(sys.argv[2:])
  elif sys.argv[1] == "reducer":
    reducer(sys.argv[2:])

Cascading Code

Here’s the bulk of the code that will achieve the effect we want. Like last time, we’re using two intermediate taps as the input and output of the streaming job. Also, we’re just using TextLine files for simplicity. If you don’t want the intermediate files hanging around, look at the comments towards the bottom for some example code on how to remove the files when the job is finished running.

package com.xcombinator.hadoopjobs.cascadingstreamingtest;
 
import cascading.cascade.*;
import cascading.flow.Flow;
import cascading.flow.FlowConnector;
import cascading.flow.MapReduceFlow;
import cascading.operation.aggregator.Count;
import cascading.operation.regex.*;
import cascading.pipe.*;
import cascading.scheme.*;
import cascading.tap.*;
import cascading.tuple.Fields;
import cascading.operation.Identity;
import java.util.Properties;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.mapred.lib.IdentityMapper;
import org.apache.hadoop.mapred.lib.IdentityReducer;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.log4j.Logger;
import cascading.operation.Debug;
import org.apache.hadoop.streaming.StreamJob;
import java.io.IOException;
 
/**
 * An example file to use a Hadoop Streaming job in cascading
 */
public class Main extends Configured implements Tool
  {
  private static final Logger LOG = Logger.getLogger( Main.class );
 
  public int run(String[] args)
  {
    JobConf conf = new JobConf(getConf(), this.getClass());
    Properties properties = new Properties();
    FlowConnector.setApplicationJarClass(properties, this.getClass());
 
    CascadeConnector cascadeConnector = new CascadeConnector();
    FlowConnector flowConnector = new FlowConnector(properties);
 
    String inputPath  = args[0];
    String outputPath = args[1];
    String intermediatePath1 = args[1] + "-mr-input";
    String intermediatePath2 = args[1] + "-mr-output";
 
    Scheme textLineScheme = new TextLine();
 
    Tap sourceTap = new Hfs(textLineScheme, inputPath);
    Tap intermediateTap1 = new Hfs(textLineScheme, intermediatePath1);
    Tap intermediateTap2 = new Hfs(textLineScheme, intermediatePath2);
    Tap sinkTap   = new Hfs(textLineScheme, outputPath);
 
    // create our first flow, sink to the intermediateTap
    Pipe wsPipe = new Each("wordsplit", 
        new Fields("line"), 
        new RegexSplitGenerator(new Fields("word"), "\\s+"), 
        new Fields("word"));
 
    Flow parsedLogFlow = flowConnector.connect(sourceTap, intermediateTap1, wsPipe);
 
    // Create a pipe and set our mr job for it 
    Pipe importPipe = new Pipe("mr pipe");
    Flow mrFlow;
 
    try {
      JobConf streamConf = StreamJob.createJob( new String[]{
          "-input", intermediateTap1.getPath().toString(), 
          "-output", intermediateTap2.getPath().toString(),
 
          // straight unix
          // "-mapper", "/bin/cat",
          // "-reducer", "/usr/bin/wc"
 
          // ruby
          // "-mapper", "src/main/ruby/word_count_mapper.rb",
          // "-reducer", "src/main/ruby/word_count_reducer.rb",
          // "-file", "src/main/ruby/word_count_mapper.rb",
          // "-file", "src/main/ruby/word_count_reducer.rb"
 
          // python
          "-mapper", "python synsets.py mapper",
          "-reducer", "org.apache.hadoop.mapred.lib.IdentityReducer",
          "-file", "src/main/python/synsets.py",
          "-file", "resources/nltkandyaml.mod",
          "-file", "resources/lib/wordnet-flat.zip",
          });
      mrFlow = new MapReduceFlow("streaming flow", streamConf, intermediateTap1,
        intermediateTap2, false, true);
    } catch(IOException ioe) {
       ioe.printStackTrace();
       System.exit(1);
       return 1;
    }
 
    // create our third "regular" cascading pipe. this is a bit contrived, but
    // the idea is substitute all 'e's with 'x's. it's just here to show how to
    // take the input of a streaming job back into cascading
    Pipe subPipe = new Pipe("subber");
    subPipe = new Each(subPipe,
        new Fields("line"),
        new RegexReplace(new Fields("linx"), "e", "x", true),
        new Fields("linx"));
    Flow subFlow = flowConnector.connect(intermediateTap2, sinkTap, subPipe);
 
    Cascade cascade = cascadeConnector.connect(parsedLogFlow, mrFlow, subFlow);
    cascade.complete();
 
    // to get rid of the intermediate files you could do this:
    // Path tmp = tap.getPath();
    // FileSystem fs = tmp.getFileSystem(conf);
    // fs.delete(tmp, true);
 
    return 0;
  }
 
  public static void main(String[] args) throws Exception 
  {
    int res = ToolRunner.run(new Configuration(), new Main(), args);
    System.exit(res);
  }
 
  }

Share:

interval is a tiny library that provides simple musical note pitch and interval arithmetic. It is intended to do one thing: given a pitch add (or subtract) an interval and give the resulting pitch.

Observe:

p = Interval::Pitch.from_string("c")
i = Interval::Interval.from_string("M3")
p2 = p + i
p2.to_short_name # => "e"
 
i.to_s # => "Major Third"
 
i2 = Interval::Interval.from_string("p5")
i2.to_s # => "Perfect Fifth"
 
(p2 - i2).to_s # => "a"

Interval Quiz

interval was written primarily for learning intervals. interval-quiz is a gem that depends on interval that provides a command-line quiz. Here’s the output of an interval-quiz session:

$ interval-quiz
Here are the intervals:
unison  p1        a1
second  m2 M2  d2 a2
third   m3 M3  d3 a3
fourth  p4     d4 a4
fifth   d5 p5  d5 a5
sixth   m6 M6  d6 a6
seventh m7 M7  d7 a7
octave  p8     d8
enter the intervals you want (or a blank line to quit):
M3
p5

["M3", "p5"]
1. above
2. below
3. both
do you want to be quizzed on intervals above, below, or both?  3
what is a major third below f# ? d
correct!
what is a major third above g# 1/1 (100%)? b#
correct!
what is a major third below b 2/2 (100%)? g
correct!
what is a perfect fifth below eb 3/3 (100%)? a
wrong. the answer is ab
what is a perfect fifth below c# 3/4 (75%)? d
wrong. the answer is f#

Installing

gem install interval interval-quiz

Source

http://github.com/jashmenn/interval
http://github.com/jashmenn/interval-quiz

Share:

Share:

However, sometimes you may have code for an existing MapReduce job or want to drop directly to Hadoop for efficiency. Even if you’re using raw MapReduce jobs, Cascading can still be useful in planning the overall data pipeline.

The code below is an example of how to use a raw MapReduce job in a Cascade. The main thing to take away is that we are creating intermediate sinks and sources and relying on Cascading to schedule the flows in the correct order.

NOTE: this code below depends on commit f0dd84cd which is a patch to MapReduceFlow.java that allows you to specifically set the Taps for a MapReduceFlow. I’ve contacted Chris about integrating this into the trunk.

Also note this patch applies to the branch wip-1.1 and later.

package com.xcombinator.hadoopjobs.mapreducetest;
 
import cascading.cascade.*;
import cascading.flow.Flow;
import cascading.flow.FlowConnector;
import cascading.flow.MapReduceFlow;
import cascading.operation.aggregator.Count;
import cascading.operation.regex.*;
import cascading.pipe.*;
import cascading.scheme.*;
import cascading.tap.*;
import cascading.tuple.Fields;
import cascading.operation.Identity;
import java.util.Properties;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.mapred.lib.IdentityMapper;
import org.apache.hadoop.mapred.lib.IdentityReducer;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.log4j.Logger;
import cascading.operation.Debug;
 
import org.apache.hadoop.mapred.KeyValueTextInputFormat;
 
/**
 * An example file to use a raw MapReduce job in cascading
 */
public class Main extends Configured implements Tool
  {
  private static final Logger LOG = Logger.getLogger( Main.class );
 
  public int run(String[] args)
  {
    JobConf conf = new JobConf(getConf(), this.getClass());
    Properties properties = new Properties();
    FlowConnector.setApplicationJarClass(properties, this.getClass());
 
    CascadeConnector cascadeConnector = new CascadeConnector();
    FlowConnector flowConnector = new FlowConnector(properties);
 
    String inputPath  = args[0];
    String outputPath = args[1];
    String intermediatePath1 = args[1] + "-mr-input";
    String intermediatePath2 = args[1] + "-mr-output";
 
    Scheme textLineScheme = new TextLine();
 
    Tap sourceTap = new Hfs(textLineScheme, inputPath);
    Tap intermediateTap1 = new Hfs(new TextLine(new Fields("line")), intermediatePath1);
    Tap intermediateTap2 = new Hfs(textLineScheme, intermediatePath2);
    Tap sinkTap   = new Hfs(textLineScheme, outputPath);
 
    // create our first flow, sink to the intermediateTap
    Pipe wsPipe = new Each("wordsplit", 
        new Fields("line"), 
        new RegexSplitGenerator(new Fields("word"), "\\s+"), 
        new Fields("word"));
 
    Flow parsedLogFlow = flowConnector.connect(sourceTap, intermediateTap1, wsPipe);
 
    // Create a pipe and set our mr job for it 
    Pipe importPipe = new Pipe("mr pipe");
 
    JobConf mrconf = new JobConf();
    mrconf.setJobName("custom mr");
 
    mrconf.setOutputKeyClass(LongWritable.class);
    mrconf.setOutputValueClass(Text.class);
 
    // the IdentityMapper, in this case, will actually output the key, which is
    // a long of offset in bytes. Not what we'd usually want, but we'll leave
    // it in for now.
    mrconf.setMapperClass(IdentityMapper.class);
    mrconf.setReducerClass(IdentityReducer.class);
 
    // note that your input is straight text-lines. This means in a real mr job
    // you'd most likely need to split the line by some convention
    TextInputFormat format = new TextInputFormat();
    format.configure(mrconf);
 
    // NOTE: this is both here and in the MapReduceFlow below
    FileInputFormat.setInputPaths(mrconf, intermediateTap1.getPath());  
    FileOutputFormat.setOutputPath(mrconf, intermediateTap2.getPath()); // likewise
 
    // create our second flow, this one is for the mrjob. Notice source and sink taps
    Flow mrFlow = new MapReduceFlow("mrflow", 
      mrconf, intermediateTap1, intermediateTap2, false, true);
 
    // create our third "regular" cascading pipe
    Pipe countPipe = new Pipe("count");
 
    // b/c our IdentityMapper is emitting long of offset in the line, just
    // strip that out. You wouldn't have to do this if you had a smarter Mapper
    // class.
    countPipe = new Each(countPipe, 
        new Fields("line"), 
        new RegexParser(new Fields("word"), ".*?\\t(.*)"), 
        new Fields("word"));
 
    countPipe = new GroupBy(countPipe, new Fields("word"));
    countPipe = new Every(countPipe, new Count(), new Fields("count", "word"));
 
    // create the flow for the last count pipe
    Flow countFlow = flowConnector.connect(intermediateTap2, sinkTap, countPipe);
 
    Cascade cascade = cascadeConnector.connect(parsedLogFlow, mrFlow, countFlow);
    cascade.complete();
 
    // if you want to get rid of the intermediate files you 
    // could do something like the following here:
    // Path tmp = tap.getPath();
    // FileSystem fs = tmp.getFileSystem(conf);
    // fs.delete(tmp, true);
 
    return 0;
  }
 
  public static void main(String[] args) throws Exception 
  {
    int res = ToolRunner.run(new Configuration(), new Main(), args);
    System.exit(res);
  }
 
  }

Share:


sudo port install enchant # you'll need MacPorts 1.8.* for this


Then download the dictionaries from the OpenOffice wiki.

Next, place the files in ~/.enchant like so:

$ tree .enchant/
.enchant/
`-- myspell
    |-- en_US.aff
    `-- en_US.dic

UPDATE
You can also set the path to where you want to keep the dictionaries when you compile enchant by passing the --with-myspell-dir option to configure (or aspell, hspell etc).

Share:

Testing multi-process erlang gen_servers can be tricky. Typically one relies simply on pattern matching to verify that the response matches what you would expect.

{expected, Response} = gen_server:call(Pid, hi).

As long as the gen_server call hi returns expected as the first element of the tuple, then the tests pass.

The technique is also the same when building client-server code where both client and server are gen_servers. The common case is to simply test one side at a time; test the response of all client calls and then (independently) test the responses of the server calls.

What about asynchronous cast?

gen_server:call is convenient because it is synchronous and returns a value.
gen_server:cast, on the other hand, is asynchronous and always returns the atom ok. This can make casts difficult to test.

gen_server_mock is a library to mock gen_server processes that expect specific, ordered sets of messages. It allows you to unit test gen_servers by verifying they are receiving the expected set of messages.

Example 1

     {ok, Mock} = gen_server_mock:new(),
     gen_server_mock:expect(Mock, call, fun({foo, hi}, _From, _State) -> ok end),
     gen_server_mock:expect_call(Mock, fun({bar, bye}, _From, _State) -> ok end),
 
     ok = gen_server:call(Mock, {foo, hi}),  
     ok = gen_server:call(Mock, {bar, bye}),  
 
     ok = gen_server_mock:assert_expectations(Mock)

This Mock expects two calls: {foo, hi} and {bar, bye}. Since Mock receives both of these messages, assert_expectations does not raise any errors.

What is verified

gen_server_mock:assert_expectations(Mock) verifies that:

1. all expected messages were received
2. no messages were received that were not expected

You can catch the exit by using the following:

     Result = try gen_server_mock:assert_expectations(Mock)
     catch
         exit:Exception -> Exception
     end,
     % etc...

Example 2

     {ok, Mock} = gen_server_mock:new(),
 
     gen_server_mock:expect_call(Mock, fun(one,  _From, _State)            -> ok end),
     gen_server_mock:expect_call(Mock, fun(two,  _From,  State)            -> {ok, State} end),
     gen_server_mock:expect_call(Mock, fun(three, _From,  State)           -> {ok, good, State} end),
     gen_server_mock:expect_call(Mock, fun({echo, Response}, _From, State) -> {ok, Response, State} end),
     gen_server_mock:expect_cast(Mock, fun(fish, State) -> {ok, State} end),
     gen_server_mock:expect_info(Mock, fun(cat,  State) -> {ok, State} end),
 
     ok = gen_server:call(Mock, one),
     ok = gen_server:call(Mock, two),
     good = gen_server:call(Mock, three),
     tree = gen_server:call(Mock, {echo, tree}),
     ok = gen_server:cast(Mock, fish),
     Mock ! cat,
 
     gen_server_mock:assert_expectations(Mock)

Currently three types of messages are supported: call, cast, and info.

The signature of the fun of each expectation is the same as the corresponding
gen_server:handle_*. So the fun for expect_call has the same signature as handle_call: fun(Request, From, State). See man gen_server for more information.

However, the return value of the fun must be one of:

    ok |                  
    {ok, NewState} |
    {ok, ResponseValue, NewState} |

Anything else will be an error. Note that you can change the state of your Mock by returning NewState.

Arbitrary, non-gen_server messages are handled with expect_info, e.g. Mock ! cat fulfills the expect_info in the example above.

References

• Github Repo (Patches readily accepted)
• Work inspired by this post
• Mocks Aren’t Stubs

Share:

Share:

What you need to do is open up your vmx file and comment out (disable) the following line:

scsi0:0.mode = "independent-persistent"

Others have also reported finding the following line in their vmx file:

snapshot.disabled = "TRUE"

Share:

# Usage: expect install-tripwire.tcl pass1 pass2 
set PASS1 [lindex $argv 0]
set PASS2 [lindex $argv 1]
 
spawn make install
 
expect "Press ENTER to view the License Agreement."
send "\r"
send "q"
expect "license agreement. \[do not accept\]"
send "accept\r"
expect "Continue with installation? \[y/n\]"
send "y\r"
 
expect "Enter the site keyfile passphrase:"
send "$PASS1\r"
expect "Verify the site keyfile passphrase:"
send "$PASS1\r"
 
expect "Enter the local keyfile passphrase:"
send "$PASS2\r"
expect "Verify the local keyfile passphrase:"
send "$PASS2\r"
 
expect "Please enter your site passphrase:"
send "$PASS1\r"
 
expect "Please enter your site passphrase:"
send "$PASS1\r"

Share:

Summary

Setting up a scalable Hadoop cluster isn’t easy, but PoolParty makes it easier
and manageable.

By the time we’re done with this tutorial you’ll have a Hadoop cluster consisting of one master node and two slaves. The slaves are formatted with HDFS and process MapReduce jobs that are delegated to them from the master.

The whole cluster is monitored by Ganglia.

Benefits of PoolParty

The nodes are very interdependent. By that I mean that each node needs to have 2 or 3 configuration files that are based on the other currently running nodes in the cluster. As nodes are joining and leaving the cluster each of these files on every node needs to be updated. PoolParty handles this process for you more-or-less automatically. The benefit is that you don’t have roll your own methods to do this every time you want to setup a cluster.

In PoolParty plugins are first-class citizens. This means you can write your own plugins and they are every bit as powerful as the resources that make up PoolParty core itself. This makes it easy to break up server functionality into modules of code . PoolParty, in a sense, gives you object-oriented server configurations. You can, for instance, take a Ganglia object, call a few methods and PoolParty takes care of executing the required commands to deploy a configured Ganglia cluster.

Architecture

PoolParty is built around the notion of pools and clouds . A pool is simply a collection of clouds. A cloud is a homogeneous set of nodes. i.e. every node in a cloud is configured the same way . Obviously nodes in a cloud will have different sets of working data as they run, but the idea is any node in a cloud could be substituted for any other node in that same cloud.

PoolParty itself is designed to be fully distributed and masterless. There is no required concept of “master” and “slave” in PoolParty itself. That said, many pieces of software, such as Hadoop, do have this concept and PoolParty can be configured to take advantage of that.

We’ll be setting up our pool as two clouds hadoop_master and hadoop_slave. Obviously, hadoop_slave will be a cloud (cluster) of nodes configured to be Hadoop slaves. hadoop_master will also be a cloud of masters. In our example we’re only going to use 1 node as the master. But you could relatively easily configure everything to have more than one master.

Software involved

• Hadoop
• Hive
• Ganglia
• PoolParty

Prerequisites

This tutorial assumes that:

1. You have Amazon EC2 java tools installed. See EC2: Getting Started with the Command Line Tools
2. You have the proper EC2 environment variables setup. See Setting up EC2 on the PoolParty website. For instance, a typical PoolParty install would have these variables in $HOME/.ec2/keys_and_secrets.sh.
3. You have PoolParty installed from source. In theory, you should be able to install the gem. However, today you should probably install from source. Make sure you have git://github.com/auser/poolparty.git checked out and then follow the “Installing” directions on the PoolParty wiki. You only need to complete the two sections Dependencies required to build gem locally and Instructions . This will install all the development dependency gems and then make sure you have all of the submodules. NOTE PoolParty deploys ruby gem versions based on the versions on your local machine. So make sure you have the most recent versions of the required gems installed locally.
4. You have the jashmenn/poolparty-examples repository. git clone git://github.com/jashmenn/poolparty-examples.git /path/to/poolparty-examples
5. You have the jashmenn/poolparty-extensions repository. Note that this directory must be a sibling directory to the poolparty-examples directory. git clone git://github.com/jashmenn/poolparty-extensions.git /path/to/poolparty-extensions

EC2 Security

Now that we have the code issue complete, we now need to deal with Amazon’s security. (See here if you are unclear on how EC2 security works.)

Setup Keypairs

Every cloud in PoolParty must have its own unique keypair. Thats important enough it’s worth repeating: every cloud in PoolParty must have its own unique keypair .

So run the following commands:

ec2-add-keypair cloud_hadoop_slave > ~/.ssh/cloud_hadoop_slave
ec2-add-keypair cloud_hadoop_master > ~/.ssh/cloud_hadoop_master
chmod 600 ~/.ssh/cloud_hadoop_*

Security Groups

You’ll also want to create a security group for our pool .

ec2-add-group hadoop_pool -d "the pool of hadoop masters and slaves"

NOTICE: Hadoop has a crazy number of ports that it requires. The ports below will work but may not be the most secure configuration. If you understand this better than I please recommend better settings. Otherwise proceed knowing that these ports are probably a little too open.

We also need to open a number of ports for this security group:

<code>ec2-authorize -p 22 hadoop_pool               # ssh
ec2-authorize -p 8642 hadoop_pool             # poolparty internal daemons
ec2-authorize -P icmp -t -1:-1 hadoop_pool    # if you want to ping (optional, i guess)
ec2-authorize -p 80 hadoop_pool               # apache
 
ec2-authorize -p 8649 -P udp hadoop_pool      # ganglia UDP
ec2-authorize hadoop_pool -o hadoop_pool -u xxxxxxxxxxxx # xxxxxxxxxxxx is your amazon account id. ugly but true
</code>

Start your cloud

NOTE : There are a number of configurations that rely on the whole cloud being booted. This means that the first time you run cloud-start you may see a few shell errors. This is okay as long as it goes away after subsequent configures. The idea is that all nodes need to be started before the whole configuration will work properly.

cd /path/to/poolparty-examples/hadoop
cloud-list # sanity check, no instances should show up, no exceptions should be raised
cloud-start -vd

Tons of information will fly by. Be patient, this could take upwards of 15 minutes.

Everything done? Good. Now you’re going to need to configure a second time. Now that all the nodes are booted they can be configured to talk to each other properly.

cloud-configure -vd

Again, tons of output should fly by. Wait for it to finish.

Now what we want to do is actually run our hadoop sample job. Open up the hadoop/clouds.rb and find the lines that look like this:

hadoop do
  configure_master
  prep_example_job
  # run_example_job
end

Uncomment the run_example_job line and configure, but this time we only need to configure master.

cloud-configure -vd -c hadoop_master

This should work, but there is a chance the hdfs wont be started in time to load the sample job. If that happens, just configure one more time.
You know it worked if you see output like the following (it wont be at the bottom):

[Fri, 26 Jun 2009 20:09:50 +0000] DEBUG: STDERR: 09/06/26 20:09:11 INFO input.FileInputFormat: Total input paths to process : 3
09/06/26 20:09:12 INFO mapred.JobClient: Running job: job_200906262006_0001
09/06/26 20:09:13 INFO mapred.JobClient:  map 0% reduce 0%
09/06/26 20:09:32 INFO mapred.JobClient:  map 66% reduce 0%
09/06/26 20:09:38 INFO mapred.JobClient:  map 100% reduce 0%
09/06/26 20:09:47 INFO mapred.JobClient:  map 100% reduce 100%
09/06/26 20:09:49 INFO mapred.JobClient: Job complete: job_200906262006_0001
09/06/26 20:09:49 INFO mapred.JobClient: Counters: 17

Congradulations! You now have a scalable Hadoop cluster at your disposal!

What to do when something goes wrong

• Checkout the PoolParty IRC channel, we’re always around and ready to help #poolpartyrb.

This plugin was based on a number of helpful sites on the web. Checkout the following links:

Hadoop

• Michael Noll’s Haddop Tutorial)

Hive

• Apache’s Hive website

Ganglia

• IBM’s Ganglia Tutorial

References

• PoolParty Documentation

Share:

Checkout the screencast.

Share:

multiruby is a great way to make sure your ruby code runs on the multitude of ruby versions (it’s part of ZenTest). It doesn’t install macruby by default. Here are instructions on how to set it up.

Since I installed macruby from the package installer my macruby files are in /Library/Frameworks/MacRuby.framework/Versions/0.4/. If your macruby files are somewhere else, adjust accordingly.

First make sure you already have multiruby setup for other versions (don’t proceed if this doesn’t work):

multiruby_setup the_usual


One problem I had with the above command was that I had RUBYOPT set in my .profile. This was calling each of these ruby versions with RUBYOPT=rubygems when trying to install. This won’t work because rubygems is one of the things you’re trying to install. Make sure you unset this variable etc. before trying to setup multiruby.

ln -s /Library/Frameworks/MacRuby.framework/Versions/0.4/usr ~/.multiruby/install/macruby-0.0.4
sudo ln -s ~/.multiruby/install/macruby-0.0.4/bin/macruby ~/.multiruby/install/macruby-0.0.4/bin/ruby
sudo ln -s ~/.multiruby/install/macruby-0.0.4/bin/macgem ~/.multiruby/install/macruby-0.0.4/bin/gem
touch ~/.multiruby/versions/macruby-0.0.4.tar.gz # fake-out

Now try:

multiruby -e "p 1+1"


Share:

Since I use the command-line twitter gem (by John Nunemaker) I miss out on this feature and find myself using echo and wc to make sure I’m not over the limit.

To fix this problem, I’ve written a small, immediate-feedback utility called tweet. It shows you the number of characters typed so far and colorizes them accordingly.

Screenshots:

warning:

over:

Just download this script and save it as tweet somewhere in your $PATH

Share:

My goal was to temporarily revert one or many commits that I had pushed to the remote server.

http://cheat.errtheblog.com/s/git — specifically the “Fix mistakes / Undo” section was helpful.

What I found:

git revert -n <sha>

#run this for each commit you would like to “undo”

(the -n makes it so that you are not actually creating a commit, but staging the reverse of the changes made by your <sha> commit in your index. git status will show you this)

git ci -a # apply your revision
git push #origin to master (these steps effectively created one commit that was the product of reversing all the commits you picked in the git revert -n step)

now your index looks like:

now, lets say, the time has come to reapply your commits. Because you didn’t just do

git reset --hard <commit3>

or something like that, all you have to do is git reset –hard <sha1> which will “undo your undo”

then

git push #origin master

again and you are back to where you were.

Share:

I want desperately to know how to detach a process from one terminal, and re-attach it to another, without using screen from the get-go. The more I read about it, the more I figure that the answer (you can’t) probably has more to do with my lack of understanding of how processes and terminals work. I read a great post here that introduced me to disown -h (careful) and nohup, some really great bash builtins. I thought, ok, lets try it. This is where I got stuck.

I tried

while true; do sleep 10; done &
disown -h %1
exit

The disown builtin handles a problem with background processes: From the bash man page (and also the above blog)

The shell exits by default upon receipt of a SIGHUP

therefore killing your process you ‘thought’ you put in the background, and then logged out to go home for the night. Disown tells the jobspec not to accept a SIGHUP, and the -h switch tells it to remain in the jobs table. I thought, cool, maybe if it stays in the the jobs table, I could also transfer it to another jobs table of another tty. (no, you can’t…)
but now I had a process on my hands that wasn’t attached to a terminal, and would just run forever unless I rebooted.

The while loop itself didn’t have a process ID, which is interesting, and because of the nature of while, the sleep commands PID kept changing, so a normal ps aux | grep slee[p] | awk '{print $2}' | xargs kill -9 wasn’t working. (This post is loosing topic fast, but the slee[p] in the above command was a cool trick I learned so that I didn’t need a grep -v grep in there).

I *did* find that I could use ps to figure out the ppid (parent process ID) and just kill -9 that, but I was also interested in knowing for sure that it wasn’t in charge of doing something else important. A little digging around, and I came across the UNIX utility pstree which of course didn’t come on my mac, but I quickly figured out that it could be installed with

sudo port install pstree

Yesterday, I had done a similar thing with the UNIX command watch, which also nicely installed using port
And, for those who don’t know, the UNIX command watch is a great poller utility, that will display the first screen’s worth of output of any command, and update it on a regular basis.

I used ps | grep to find the ppid of the sleep process, then ran this command:

watch "pstree $PPID"

This was way cool, as every ten seconds, I watched as the PID of sleep (the child process of this bash process I had just found) changed.

Take away:

sudo port install watch
sudo port install pstree

Share:

shell -bash

Everywhere I read said to put that in your .screenrc file. On my Mac, OS X 10.5, however it seemed to have the (extremely) unwanted circumstance of changing my cwd whenever I started screen to my $HOME folder.

I was able to do some testing my remoting into a *fairly* vanilla fedora box and running the command:

cd /; screen -s -bash

This started bash as a login shell

shopt login_shell #=> login_shell    	on

but did not change my cwd.

I tried moving my .profile file to see if it was the culprit.

I finally came up with this hackish solution.

create file

screen_shell

and place in it the line

$SHELL -l

I knew from other testing that

shell 'bash -l'

didn’t work, course, now, having remembered that I had a similar experience with vi… I wonder if

shell bash\ -l

would work… (nope…)

so, I then changed my .screenrc file to read

shell /Users/bhenderson/.screen_shell

made sure that I did a

chmod u+x ~/.screen_shell

and off I went.

If anyone would like to add to my understanding of what was going on, or a better solution, please feel free to comment. thanks.

Share:

rsync -a /path/to/bar/ /path/to/foo/  # => /path/to/foo/[contents of bar]
rsync -a /path/to/bar /path/to/foo/   # => /path/to/foo/bar/[contents of bar]
 
cp -a /path/to/bar/ /path/to/foo/     # => /path/to/foo/bar
cp -a /path/to/bar /path/to/foo/      # => /path/to/foo/bar

why can’t cp have the same source behavior as rsync?

btw, I found the solution on this blog:

mkdir target_directory
cd source_directory
cp -a . target_directory

(on the blog, the author does cp -ap, but that looks redundant according to the man page)

Share:

def x
  5
end
 
if false
  x = 3
end
 
puts x # => ?

Share:

Erlang has a group of highly optimized list operations in the lists module (Try erl -man lists to see them all). We’re going to be using those to build our Erlang version of #each_with_index.

There are two methods from list we’re going to use: lists:seq and lists:zip. lists:seq simply returns a list of integers between a given range. For instance:

1> Seq = lists:seq(0, 9).
[0,1,2,3,4,5,6,7,8,9]

lists:zip takes two lists (of equal length) and returns a list of tuples. Like so:

2> lists:zip([a,b,c], [1,2,3]).
[{a,1},{b,2},{c,3}]

The last thing we need is a list comprehension, which is similar to each with a block in Ruby. For instance, in Ruby:

>> [:a, :b, :c].each { |element| puts element }
a
b
c
=> [:a, :b, :c]

In Erlang we would do the following:

3> [io:format("~p~n", [Element]) || Element <- [a,b,c]].
a
b
c
[ok,ok,ok]

There is one difference here: the Erlang list comprehension returns a new list which is the value of each call to io:format. In this way an Erlang list comprehension is closer to Ruby’s #collect rather than #each.

We now have all the pieces to build our Erlang version of #each_with_index.

4> EachWithIndex = fun(L, X) -> [
  X(Element, Index) || 
  {Element, Index} <- lists:zip(L, lists:seq(1, length(L)))
   ] end.

Whats going on here?

lists:zip(L, lists:seq(1, length(L)))

creates a list of tuples with each tuple containing an element of L and a companion integer.

[X(Element, Index) || 
  {Element, Index} <- lists:zip(L, lists:seq(1, length(L)))]

tells us to take that list of tuples and do a list comprehension on each of the elements, calling fun X in turn.

EachWithIndex = fun(L, X) -> ... end.

creates a fun (think lambda) and assigns it to EachWithIndex. You could just as easily create this as a regular function in a .erl file somewhere.

Now we can call our new EachWithIndex function by passing in a list and a fun:

5> EachWithIndex([a,b,c,d,e], fun(Element, Index) -> io:format("~p at ~p~n", [Element, Index]) end).
a at 1
b at 2
c at 3
d at 4
e at 5
[ok,ok,ok,ok,ok]

Share:

A new plugin install and mount s3fs volume on an instance using PoolParty.

Getting it

git clone git://github.com/jashmenn/poolparty-s3fs-plugin.git

Usage

NOTICE You must have already created your S3 bucket. This plugin will not
create it for you.

cloud(:app)
  ...
  s3fs(:bucket => "my-fun-bucket")
  ...
end

This will be mounted at /mnt/my-fun-bucket
You can mount multiple buckets and/or change the mount point:

s3fs do
  bucket "my-fun-bucket"
  bucket "my-other-bucket", "/mnt/a-disk"
end

References:

• http://code.google.com/p/s3fs/wiki/FuseOverAmazon
• http://groups.google.com/group/ec2ubuntu/browse_thread/thread/9093236bc07d220b?pli=1

Share:

or git clone git://github.com/jashmenn/poolparty-mrtg-plugin.git

Usage:

Declare mrtg in your cloud block

mrtg do
   monitor :cpu, :uptime, :memory, :open_files, :processes, :apache, :network_interfaces
end

Share:

Posted below are the slides. The slides alone certainly are not enough for a complete introduction to PoolParty. However, I hope they have some information that is helpful to beginners.

Check out the PoolParty wiki for more documentation.

Share:

 
curl http://www.whatismyip.com/automation/n09230945.asp ; echo

Then alias that badboy E.g.:

 
alias whatsmyip='curl http://www.whatismyip.com/automation/n09230945.asp ; echo'

Happy day.

Share:

for f in `find . -regex ''.*cp.*snippets''` ; do cp bbb.html $f/ ; done

Which returned:

cp: cannot stat `bbb.html': No such file or directory

bbb.html is a symlink and the default behavior of cp (I think) is to try and copy the actual file, which didn’t exist from the current directory. So just adding the -R made cp copy it as a symlink with the final command being:

for f in `find . -regex ''.*cp.*snippets''` ; do cp -R bbb.html $f/ ; done

Share:

$ ls ~-/

The neat thing about this expansion is that it does tab completion. What this lead me to discover is that you can also do ~N, where n is a number refering to your directory stack. (the bash man page says you can use the command `dirs’ to view the stack, but it doesn’t seem to work for me. When you put in a 0 (zero), it is the current working dir.  Combined with tab completion, this gave me a great way for creating symlinks. Before, I had to do something like

$ ls `pwd`/

Which does do tab completion. But if you want to save a whopping 3 key strokes!, you can do something like this:

$ pwd
 
/Users/bhenderson/programming/ruby/guides.rubyonrails.org
 
$ ln -s ~0/index.html ~/Sites/ruby_guides
 
$ ls -l !:$
 
ls -l ~/Sites/ruby_guides
 
lrwxr-xr-x  1 bhenderson  staff  68 Jan 12 08:47 /Users/bhenderson/Sites/ruby_guides -> /Users/bhenderson/programming/ruby/guides.rubyonrails.org/index.html
 
$

Guide spidered from here.

Share:

$ for f in `ls -1 | grep string`; do mv $f ${f/string/replace};done

I’d like to find a method of using the bash command `find’, but this is pretty clean, and I don’t yet know how to manipulate the `{}’ in find -exec {} \;

great site for explaining bash variable manipulation.

Share:

If you’re having trouble getting do_mysql to install on Mac OS X. Try the following:

$ locate mysql_config
/usr/local/mysql/bin/mysql_config

The directory /usr/local/mysql above is your mysql directory. Then try to install do_mysql with the following command (split for readability):

$ gem install do_mysql -- \
   --with-mysql-dir="/usr/local/mysql/" \
   --with-mysql-config="/usr/local/mysql/bin/mysql_config"

This solution was found here. I experienced the problem while trying to get merb to run with datamapper on mysql.

Share:

sake is a tool for system-wide rake tasks. It was created by Chris Wanstrath and is now maintained by postmodern. It provides a handy way to distribute generally-useful rake tasks. The advantage of sake over, say, a script in ~/bin is that 1) you have a predictable interface and 2) its easy to distribute tasks with sake -i http://any-url (in theory).

Examples

Ben Schwarz has written a great sake task for installing a ssh-public-key on a remote server. You can install it with sake -i http://github.com/benschwarz/sake-tasks/tree/master%2Fssh%2Fssh.sake?raw=true . (If you haven’t seen highline before, check it out. It’s a nice abstraction and clean alternative to using gets and puts.)

A sake task for installing remote git repos

Based on Ben’s task I’ve created a task for installing remote git repos.

The above task will do two things:

Create the bare repository on a foreign server
Optionally create a local working copy

There’s a bug in the above authentication code, did you catch it? For authentication, it will use your ssh keys if it finds it otherwise it will ask you for a password. However, if your ssh keys aren’t valid then it simply fails. Ideally it would ask you for a password and retry in this case.

I’m not too concerned about this however; if you’re creating a git repo remotely, you probably want to clone it and you also probably have your ssh keys setup to do so. (I probably shouldn’t be checking for a password at all.)

sake disadvantages

I believe sake is a great idea, but there is some rough edges in the current implementation.

Current Gem has a bug

Today, if you gem install sake there is a bug such that sake wont even run. Your best bet is to clone drnic’s repo sake >= 1.0.16 and install from there.

My guess is that this is because sake’s creator recently put out a request for someone to take over the project and the person who took it over hasn’t upgraded the rubyforge gem.

Task installation

Another bug that seems to be specific to this version of sake, when I run sake -i the task names and namespaces are created but the ~/.sake file contains no code.

Again, this just seems to be a minor bug with the current version and as such I see it as temporary. I’m planning on working on a patch on for this, though I’m not sure I’ll fix it before someone else.

Task development

The built-in way to do sake task development is to write a little bit, then sake -i your file, then write a little bit more etc. This gets old fast. Because of that drnic has written a great project for sake task development. In that project you run sake testrun taskname.

drnic’s project is handy and makes sake task development a snap. However, I think this fact could slow the adoption of sake; it’s a barrier to entry if you have to install a second project just to develop the first.

Even so …

I still think that the idea of sake is solid. There is definitely a need for easily distributable, reusable, system-wide rake tasks. Despite its current shortcomings, sake is definitely worth a look.

Share:

We start with the problem: Write a ruby script that can print itself out, without it reading its own file.

The restriction “without it reading its own file” is important, otherwise a trivial solution would be:

#!/usr/bin/env ruby
 
puts File.read( __FILE__ )

Without thinking about it too much and just starting naively, we may try:

#!/usr/bin/env ruby
 
puts "#!/usr/bin/env ruby\n\nputs \"#!/usr/bin/env ruby\\n\\nputs \\\"#!/usr/bin/env ruby ...

We soon find ourselves entering into an infinite regress that we quickly realize has no escape. Seems we have to be a little bit more clever.

Before getting to the solution, the interested reader is urged to stop reading at this point and try to come up with a solution independently.

Here’s one way to accomplish this. Create the following script, which is just the first step toward the final script:

#!/usr/bin/env ruby
 
require 'cgi'
 
dna = 'DNA'
cell = CGI.unescape dna
puts cell.sub( 'DNA', dna )

Save the above file as “pregodel.rb”. Next create this helper script:

#!/usr/bin/env ruby
 
require 'cgi'
 
puts CGI.escape( STDIN.read )

Save this as “godelize.rb”, and run it taking pregodel.rb into STDIN:

$ ruby godelize.rb < pregodel.rb

This outputs a cgi-encoded string whose initial characters are '%23%21%2Fusr%2Fbin%2Fenv ...'. Finally, copy pregodel.rb to godel.rb and edit godel.rb. Replace the first instance of DNA with the output of godelize.rb, and save it.

godel.rb should now be:

#!/usr/bin/env ruby
 
require 'cgi'
 
dna = '%23%21%2Fusr%2Fbin%2Fenv+ruby%0A%0Arequire+%27cgi%27%0A%0Adna+%3D+%27DNA%27%0Acell+%3D+CGI.unescape+dna%0Aputs+cell.sub%28+%27DNA%27%2C+dna+%29%0A'
cell = CGI.unescape dna
puts cell.sub( 'DNA', dna )

When run, the output is identical to the file contents. A conclusive test is comparing md5 sums:

$ cat godel.rb | md5
4b92a6303568cde3a98d042a67616d2d

$ ruby godel.rb | md5
4b92a6303568cde3a98d042a67616d2d

(This was run on a Mac—many linux distros have md5sum instead.)

Discussion

So what does this have to do with DNA in cells? The godel.rb script is about as simple of a self-replicating ruby script that one can write (again, without invoking the file itself). Its function and structure is surprisingly similar to how a biological cell works. The entire script is stored in the dna variable as a cgi-encoded string, just as a cell is encoded in its nucleus with deoxyribonucleic acid:

dna = '%23%21%2Fusr%2Fbin%2Fenv+ruby%0A%0Arequire+%27cgi%27%0A%0Adna+%3D+%27DNA%27%0Acell+%3D+CGI.unescape+dna%0Aputs+cell.sub%28+%27DNA%27%2C+dna+%29%0A'

The first step in the script's self-replication process is to unescape itself from a cgi-encoded string to ruby code. This is analogous to a cell's ribosomes performing its role in translating DNA into protein. (The intermediate step of RNA transcription is omitted for simplicity without loss of accuracy.) Protein is encoded as sequences of nucleic acids in DNA, just as ruby code is encoded into cgi-encoded character sequences.

cell = CGI.unescape dna

Finally, a cell's DNA must make a copy of itself during mitosis and insert it into the cell's copy. The analog of this is the last line of the script:

puts cell.sub( 'DNA', dna )

Though the dna string has a representation of the whole script, observe how it was able to avoid the infinite regress. The place in the string that references itself, rather than storing itself again (which would lead to an infinitely long string) it instead uses a symbol for itself, namely, DNA. This acts as a placeholder for where the dna variable will be substituted into when the string is printed out.

For further reading, check out the book Gödel, Escher, Bach by Douglas Hofstadter, which applies the above ideas to not only self-replicating cells, but to the nature of mathematical truth, the art of Escher, the music of Bach, and human consciousness, among other related topics.

Share: