CodeBits - Tested Complex Code!

Implement Word Count in Accumulo

2012-04-21T22:28:00.000-04:00

It seems like the first map-reduce program that everyone tries is counting words. This first program reads a piece of text using the mapper to tokenize the text and outputs a "1" for each token. Then the reducer adds up the "1" values to produce the word counts.

Accumulo provides the same functionality without needing to write a single line of code by using a SummingCombiner iterator. Below is a complete example.

Actually this example is more powerful because the same code can be used to sum across any time dimension.

This example shows how to sum across days. First start the accumulo shell.Then follow these steps:

> createtable --no-default-iterators wordtrack
wordtrack> setiter -t wordtrack -p 10 -scan -minc -majc -class org.apache.accumulo.core.iterators.user.SummingCombiner
SummingCombiner interprets Values as Longs and adds them together.  A variety of encodings (variable length, fixed length, or string) are available
----------> set SummingCombiner parameter all, set to true to apply Combiner to every column, otherwise leave blank. if true, columns option will be ignored.: true
----------> set SummingCombiner parameter columns, <col fam>[:<col qual>]{,<col fam>[:<col qual>]} escape non-alphanum chars using %<hex>.: 
----------> set SummingCombiner parameter lossy, if true, failed decodes are ignored. Otherwise combiner will error on failed decodes (default false): <TRUE|FALSE>: 
----------> set SummingCombiner parameter type, <VARLEN|FIXEDLEN|STRING|fullClassName>: STRING

Insert records for a daily rollup.

wordtrack> insert "Robert" "2011.Nov.12" "" 1
wordtrack> insert "Robert" "2011.Nov.12" "" 1
wordtrack> insert "Parker" "2011.Nov.12" "" 1
wordtrack> insert "Parker" "2011.Nov.12" "" 1
wordtrack> insert "Parker" "2011.Nov.12" "" 1
wordtrack> insert "Parker" "2011.Nov.23" "" 1
wordtrack> scan
Parker 2011.Nov.12: []    3
Parker 2011.Nov.23: []    1
Robert 2011.Nov.12: []    2

Get all counts for a given day:

wordtrack> scan -c 2011.Nov.12
Parker 2011.Nov.12: []    3
Robert 2011.Nov.12: []    2

Let's talk about that "--no-default-iterators" parameter for a moment. Normally, Accumulo uses an iterator that only displays the one value (the value with the latest timestamp) based on the uniqueness of the key/column family/column qualifer combination. If you leave that iterator in place, your counters will get essentially reset to one each time a compaction is done.

Using Accumulo To Calculate Seven Day Rolling Average

2012-04-11T22:49:00.003-04:00

Without commenting on if this is a good idea, let me show how you can use Accumulo to store the seven values needed to perform a rolling average. Log into the shell. Create the table. Then configure iterators to retain seven values instead of just the default single value. Finally insert some values.

bin/accumulo shell -u root -p password
> createtable rolling
rolling> config -t rolling -s table.iterator.scan.vers.opt.maxVersions=7
rolling> config -t rolling -s table.iterator.minc.vers.opt.maxVersions=7
rolling> config -t rolling -s table.iterator.majc.vers.opt.maxVersions=7

rolling> insert 2012.02.20 "" "" 21
rolling> insert 2012.02.20 "" "" 22
rolling> insert 2012.02.20 "" "" 23
rolling> insert 2012.02.20 "" "" 24
rolling> insert 2012.02.20 "" "" 25
rolling> insert 2012.02.20 "" "" 26
rolling> insert 2012.02.20 "" "" 27
rolling> insert 2012.02.20 "" "" 28
rolling> insert 2012.02.20 "" "" 29
rolling> insert 2012.02.20 "" "" 30

rolling> insert 2012.02.21 "" "" 51
rolling> insert 2012.02.21 "" "" 52
rolling> insert 2012.02.21 "" "" 53
rolling> insert 2012.02.21 "" "" 54
rolling> insert 2012.02.21 "" "" 55
rolling> insert 2012.02.21 "" "" 56
rolling> insert 2012.02.21 "" "" 57
rolling> insert 2012.02.21 "" "" 58
rolling> insert 2012.02.21 "" "" 59
rolling> insert 2012.02.21 "" "" 60

You can use the 'scan' command to see all information in the table. Or you can use 'scan -b 2012.02.21 -e 2012.02.21' to see information about a single row id. You can use Java program to calculate the rolling average:

public class RollingAverageDriver {

 public static void main(String[] args) throws AccumuloException, AccumuloSecurityException, TableNotFoundException, TableExistsException {
  String instanceName = "development";
  String zooKeepers = "localhost";
  String user = "root";
  byte[] pass = "password".getBytes();
  String tableName = "rolling";

  ZooKeeperInstance instance = new ZooKeeperInstance(instanceName, zooKeepers);
  Connector connector = instance.getConnector(user, pass);

  Scanner scanner = connector.createScanner(tableName, new Authorizations());

  RollingAverageCalculator raCalculator = new RollingAverageCalculator(scanner, 7);
  int rollingAverage = raCalculator.calculate("2012.02.21");

  System.out.println("7 Day Rolling Average: " + rollingAverage);
  

  System.out.println("END");
 }
}

Of course, you'll also need the RollingAverageCalculator class:

public class RollingAverageCalculator {

 Scanner scanner = null;
 int minNumberOfValues = 0;
 
 public RollingAverageCalculator(Scanner scanner, int minNumberOfValues) {
  super();
  this.scanner = scanner;
  this.minNumberOfValues = minNumberOfValues;
 }

 public int calculate(final String rowId) {
  scanner.setRange(new Range(rowId, rowId));

  int sum = 0;
  int count = 0;
  Iterator<Map.Entry<Key,Value>> iterator = scanner.iterator();
  while (iterator.hasNext()) {
   Map.Entry<Key,Value> entry = iterator.next();
   Value value = entry.getValue();
   String sValue = new String(value.get());
   sum += Integer.parseInt(sValue);
   count++;
  }
  return count < minNumberOfValues ? 0 : (sum / count);
 }
}

It should be fairly straightforward to change the code to perform any kind of rolling average.

Reading A Row With Accumulo

2012-04-11T21:06:00.002-04:00

Just another example of reading information with Accumulo. This example needs the accumulo-core and hadoop-core jar files.

import java.util.Iterator;
import java.util.Map;
import java.util.Map.Entry;

import org.apache.accumulo.core.client.AccumuloException;
import org.apache.accumulo.core.client.AccumuloSecurityException;
import org.apache.accumulo.core.client.Connector;
import org.apache.accumulo.core.client.Scanner;
import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.client.TableNotFoundException;
import org.apache.accumulo.core.client.ZooKeeperInstance;
import org.apache.accumulo.core.data.Key;
import org.apache.accumulo.core.data.Range;
import org.apache.accumulo.core.data.Value;
import org.apache.accumulo.core.security.Authorizations;
import org.apache.hadoop.io.Text;

public class Accumulo_Scan_A_Row_Driver {

 public static void main(String[] args) throws AccumuloException, AccumuloSecurityException, TableNotFoundException, TableExistsException {
  String instanceName = "development";
  String zooKeepers = "localhost";
  String user = "root";
  byte[] pass = "password".getBytes();
  String tableName = "user";
  String rowId = "John";

  ZooKeeperInstance instance = new ZooKeeperInstance(instanceName, zooKeepers);
  Connector connector = instance.getConnector(user, pass);

  Scanner scan = connector.createScanner(tableName, new Authorizations());
  scan.setRange(new Range(rowId, rowId));

  Iterator<Map.Entry<Key,Value>> iterator = scan.iterator();
  while (iterator.hasNext()) {
   Map.Entry<Key,Value> entry = iterator.next();
   Key key = entry.getKey();
   Value value = entry.getValue();
   System.out.println(key + " ==> " + value);
  }
 }
}

Thinkering - A new meme for 2012

2012-01-01T11:28:00.001-05:00

Computing Power Like Water

2011-12-22T10:30:00.000-05:00

Many SF stories are predicated on the idea that computing power is available like water. Simply make a request and a swarm (or cloud) of computers assemble as needed in order to answer your question. Web searches are great but they rely on pre-computed information and actually answer very limited questions and in very narrow ways. The Apache Whirr project allows the technically-savvy person to link tens of computers together for very little money. Each computer hour costs less than 5 cents. And when I say technically-savvy, the learning curve is quite gentle. A few keyboard commands gets them going without any need to understand the real complexity behind the scenes. This is an important step because people can focus on problem solving instead of plumbing. Imagine if every science project in the world could shave six months off their start-up time because they can get a computer network running inside of a day. No more need for expensive, hard-to-find computer network administrators. And if you've got the network topology wrong just tear it down and restart with just one day downtime and zero hardware replacement cost. If you thought that science was already moving too fast, you'll be shocked by the next ten years.

Detecting Unsorted Spring Beans

2011-06-17T07:54:00.002-04:00

I like my Spring Beans definition files to be sorted. So I wrote the following code. Eventually I might turn it into a JUnit test for my continuous integration server.

package play;

import java.io.IOException;
import java.util.ListIterator;

import org.jdom.Attribute;
import org.jdom.Document;
import org.jdom.Element;
import org.jdom.JDOMException;
import org.jdom.input.SAXBuilder;

public class UnsortedSpringBeans {

 public static void checkSortOrder(final String filename) throws JDOMException, IOException {
        System.out.println(filename);
        System.out.println("-------------------------------------------------");                  
        String lastBean = null;
        SAXBuilder parser = new SAXBuilder();
        Document doc = parser.build(filename);
        Element root = doc.getRootElement();
        @SuppressWarnings("unchecked")
  ListIterator children = root.getChildren().listIterator();
        while (children.hasNext()) {
          Element child = children.next();
          Attribute id = child.getAttribute("id");
          if (id == null) {
              System.out.println("Skipping    : " + child.getName());                  
           continue;
          }
          if (lastBean != null) {
              int cmpValue = id.getValue().compareTo(lastBean);
              if (cmpValue < 0) {
                  System.out.println("Out Of Order: " + id.getValue());                  
              } else {
                  System.out.println("              " + id.getValue());                  
              }
          } else {
              System.out.println("              " + id.getValue());                  
          }
          lastBean = id.getValue();
        }
        System.out.println("");
    }
    
    /**
     * @param args
     * @throws IOException 
     * @throws JDOMException 
     */
    public static void main(String[] args) throws JDOMException, IOException {
        checkSortOrder("config/bean-definition.xml");
        checkSortOrder("config/database-beans.xml");
        checkSortOrder("config/hypersonicContext.xml");
        checkSortOrder("config/url-processor-beans.xml");
    }

}

Free-style Paper Roller-coaster at ReverseSpace.

2011-05-23T00:10:00.001-04:00

See the full gallery on Posterous

Six kids had a great time.

Human Logic Gates

2011-03-24T07:12:00.000-04:00

Each data person gets candy Each logic person gets instruction. Then logic guys process the data guys. Good for Kids?

Handmade Hat for Odyessy of the Mind

2011-03-14T19:20:00.000-04:00

I was a style judge this year. Loads of fun!

Running CEPH on 32bit Ubuntu kernal

2011-01-24T21:56:00.002-05:00

DON'T DO IT! I spent several hours setting up three virtual servers - one master and two nodes - to run ceph. When it finally came time to run the cfuse command, I saw this message: WARNING: Ceph inode numbers are 64 bits wide, and FUSE on 32-but kernals does not cope well with that situation. Expect to crash shortly. Sigh.

Active Relation Explained

2010-09-18T12:28:00.001-04:00

Action Relation is an implementation of map-reduce. The predicate map the data. And the operations reduce it.

Tracking RSPEC Test Execution So Debugging Statements Are Segmented

2010-09-16T14:33:00.001-04:00

Throw the following into an initializer:

module Spec
  module Example
    module ExampleMethods
      def execute_with_print_logging(run_options, instance_variables)
        p "RSPEC TRACE: #{description} => #{@_proxy.location}"
        execute_without_print_logging(run_options, instance_variables)
      end
      alias_method_chain :execute, :print_logging
    end
 end
end

How Do I Expose the Current RSPEC File and Description

2010-09-16T14:05:00.002-04:00

My goal with this monkey patch was to expose the rspec filename and description as global variables so that my exception handling code can know which rspec test caused the exception.

module Spec
  module Example
    module ExampleMethods
      alias :old_execute execute
      def execute(run_options, instance_variables)
        $rspec_location = @_proxy.location
        $rspec_description = description
        old_execute(run_options, instance_variables)
      end
    end
 end
end

Compiling and Running Hadoop WordCount Example in NetBeans

2010-09-12T18:16:00.002-04:00

In order to compile and run the Hadoop WordCount example in NetBeans, I followed these steps:

Create a new Java project called hadoop.
Copy the src/examples/org/apache/hadoop/WordCount.java to src/hadoop.
Select Run>Set Project Configuration>Customize and change the Main class entry to hadoop.WordCount. Also set the arguments entry to 'input output".
Add the following jar files as libraries: hadoop-0.20.2-core.jar, commons-cli-1.2.jar, commons-logging-1.0.4.jar, commons-httpclient-3.0.1.jar

Add the following method to the WordCount class:

static public boolean deleteDirectory(File path) {
        if (path.exists()) {
            File[] files = path.listFiles();
            for (int i = 0; i < files.length; i++) {
                if (files[i].isDirectory()) {
                    deleteDirectory(files[i]);
                } else {
                    files[i].delete();
                }
            }
        }
        return (path.delete());
    }

Add the following lines of code just before the "new Job" line:

  // delete the output directory.
   WordCount.deleteDirectory(new File(otherArgs[1]));

Create the input directory.
Copy a text file into the input directory.
Press F6 to run the program.
Read the files in the output directory.

MonoDevelop Rocks!

2010-08-12T23:09:00.003-04:00

MonoDevelop - I am doing pro-bono work for a local non-profit (Hope and a Home) that has a GoDaddy Windows-based account. I, of course, only have Ubuntu running at home. A short web search found MonoDevelop. Ten minutes later, I had written by first ASP application. Easy to install and easy to use - highly recommended!

Writing methods which write methods

2010-07-26T11:07:00.001-04:00

http://www.raulparolari.com/Ruby2/define_method - Here is a nice writeup showing how to use Ruby's define_method feature.

Suggestion for Community Service Hours - Curating a Community Calendar

2010-07-25T13:11:00.002-04:00

http://elmcity.cloudapp.net/ - This is an interesting website which allows you to provide calendar events in several different formats. A great community project would be to aggregate school events in a locality.

Using ImageMagick To Create Horizontally Repeating Background Image

2010-07-11T10:35:00.003-04:00

Find an image that you want to slice vertically, which I'll refer to as the base image. This slice is what will be repeated across the background of the web page.
Use the identify home_page.jpg command to find out some details about the base image. The result of this command looks like this: home_page.jpg JPEG 1300x1380 1300x1380+0+0 8-bit DirectClass 736KiB 0.240u 0:00.240
For our purposes, the first set of numbers is sufficient. They hold the width and height.
The next step involves extracting a vertical slice (top to bottom) from the base image. Use this command: convert -extract 10x1380+0+0 home_page.jpg body_background.png. Notice that you can vary the width of the slice using the first number. In this case, I use 10 but some other number is probably more appropriate for you.
If you want to test various widths, you can 'tail' the background image using display -update 1 body_background.png which redisplays the image every second.
Once you've got the vertical slice, you need to integrate it with your web page.

Use CSS similar to this:

body {
  background-image: url('body_background.png');
  background-repeat: repeat-x;
}

Using module_eval to Define Instance Methods in a Ruby Gem to Enable Per-Model Configuration

2010-07-01T17:50:00.003-04:00

In the last post, I mentioned that I wrote a small gem to post changes to ActiveRecord models to Twitter. In that version, the configuration was handled by a YAML file. I wanted to evolve the gem so that developers could switch Twitter accounts for each model.

Basically, I wanted to able to the use the following:

class Place < ActiveRecord::Base
  alastrina :twitter => { :username => 'alastrina_gem', :password => 'QQQQQQ' }
end

After a bit of tinkering and searching, I decided to use the following code in my gem.

ALASTRINA_CONFIGURATION_FILE = 'config/alastrina.yml'

module Alastrina
  def self.included(base)
    base.extend(ClassMethods)
  end
  
  module ClassMethods
    def alastrina hash
      module_eval do
        def configuration
          throw "Missing #{ALASTRINA_CONFIGURATION_FILE}" unless File.exists? ALASTRINA_CONFIGURATION_FILE 
          @config ||= YAML::load(File.read(ALASTRINA_CONFIGURATION_FILE))
        end
        if hash[:twitter]
          def send_to_twitter?
            true
          end
          eval"def twitter_username\n\"#{hash[:twitter][:username]}\"\nend\n"
          eval "def twitter_password\n\"#{hash[:twitter][:password]}\"\nend\n"
        else
          def send_to_twitter?
            @twitter_flag ||= !configuration['twitter'].blank?
          end
          def twitter_username
            @twitter_username ||= configuration['twitter']['username'] if send_to_twitter?
          end
          def twitter_password
            @twitter_password ||= configuration['twitter']['password'] if send_to_twitter?
          end
        end
      end
    end
  end

  def after_save
    if send_to_twitter?
      require 'twitter' 
      throw "Missing Twitter userid" if twitter_username.blank? 
      throw "Missing Twitter password" if twitter_password.blank?
      send_via_twitter
    end
  end

private

  def send_via_twitter
    if changes.size > 0
      httpauth = Twitter::HTTPAuth.new(twitter_username, twitter_password)
      client = Twitter::Base.new(httpauth)
      begin
        client.update(changes.to_yaml)
      rescue
        RAILS_DEFAULT_LOGGER.error "alastrina.send_via_twitter; Unable to send change. Message[#{$!}] Change[#{changes.to_yaml}]"
      end
    end
  end
  
end

ActiveRecord::Base.class_eval { include Alastrina }

The key insight to this code is how the hash passed on the alastrina of the Model is passed down into the instance. The code is fairly straightforward once you see what the eval is doing.

Alastrina - a gem to push model changes to twitter

2010-06-28T19:11:00.001-04:00

http://github.com/medined/alastrina - the gem

http://github.com/medined/alastrina_demo - a demo application

http://twitter.com/Alastrina_gem - the result

I was doodling in code this weekend and the result is a small gem that hooks into the after_save callback to send model changes to a Twitter account.

Make sure you install the Twitter gem. Then in the demo app, update the config/alastrina.yml file with your Twitter credentials. Open a console and do:

a = Place.new
a.name = "GGG"
a.save

And, presto, the model changes should be posted to your Twitter feed.

Connecting acts_as_solr and cucumber (resolving the whiny_nil error)

2010-06-21T11:00:00.002-04:00

When I installed cucumber into my application and ran 'cucumber' I got an error that said "undefined method 'symbolize_keys' for nil:nilclass". This message happened because acts_as_solr tried to load a 'cucumber' environment from config/solr.yml and was unable to find it.

Ubuntu and RocketFish RF-FLBTAD bluetooth dongle

2010-03-22T00:24:00.001-04:00

I bought the RocketFish RF-FLBTAD bluetooth dongle today. When it was plugged into the front USB ports of my computer, the dongle would not be recognized as an HCI device. However, when I plugged it into one of the USB ports in the back of my computer, it registered just fine. Lots of good messages in /var/log/syslog. Update: The dongle disappeared when I rebooted. However, it came back when I manually ran: sudo /etc/init.d/bluetooth restart

Heroku Tip; Use a Ping or Uptime Service As a Cron Job.

2010-02-09T09:45:00.002-05:00

This is a pass-along tip from Toby Hede (http://stackoverflow.com/users/14971/toby-hede) on StackOverflow. Just give the Ping service the URL of the controller doing the work instead of your root page.

Rails Tip - How Do I See Params From My Forms?

2010-02-09T09:07:00.002-05:00

I like to document what my forms are sending to my action methods. So most of the time, I create a method in my controller like this:

def to_export
  render :text => params.inspect
end

Then I use copy and paste to put the result into my code as a comment.

# {
#   "commit"=>"Upload File", 
#   "authenticity_token"=>"17FvBH+Z4Meni6WvsCdPByrrk751BV9mNklzKDAo2Vc=", 
#   "action"=>"do_import", 
#   "import_criteria"=>{
#     "primary_language_name"=>"Creole", 
#     "secondary_language_name"=>"English"
#   }, 
#   "form"=>{
#     "file"=>"traveling_haiti.txt"
#   }, 
#   "controller"=>"import"
# }

Another Fix for EMAIL not authorized to access APP

2010-02-08T22:00:00.002-05:00

I wanted to change the primary email associated with my Heroku account. So I removed the email from the authorization list. The next time I pulled from git, I got the following error:

heroku@david-medinets.otherinbox.com not authorized to access crisiscamp-translation

The email address is pulled from your public/private keys which can be recreated using the following command:

ssh-keygen -t rsa -C EMAIL