The more interesting thing to note is not that we ended up with Cassandra, but what we used to make that decision. When it came down to Cassandra and HBase, they both had their pros and cons for our use/case. In fact, it’s likely that either one of them would have worked out just fine in the long run. We actually made our decision based on the community and the roadmap (but mostly the roadmap).

The product roadmap is simply something that says where the product is intending to be in the next few weeks, months or years. And it can be important because if the product roadmap in 12 months is in alignment with where you see your technology in 12 months, then it would seem like a pretty good fit. And it potentially allows your organization to help influence the development of new technologies. I’m sure some of you are thinking that it’s no fun being the guinea pig or being on the bleeding edge of everything. But when you are pushing the limits of today’s technology stacks and applications, you have to be on the cutting edge every now and then.

Product roadmap doesn’t just tell you about the application itself, it tells you about the community. If you see yourself aligned with the roadmap in the next 12-24 months, then you are also aligning yourself with the community. And the community is hopefully full of people that have a like-minded set of goals for the use of the product in question. So don’t just think about what’s good for you now, think about what going to be good for you (or your organization) in the future too.

Related posts:

1. Culture of Product vs. Culture of Code
2. What Does Web 2.0 Mean To You?
3. Bing! Hunch! Decision Engine!

Looking at this from a slightly different perspective, Google is saying that if you just make your site SSL available, then you can continue to have your referrers. And that is ultimately what people (read marketers and SEO folks) want anyway. To oversimplify a bit, making one’s site available over SSL is as easy as going to GoDaddy or the like and buying and installing an SSL certificate on your web server.

But what does having this certificate really do? It allows a website to be loaded in a secure, encrypted environment. It also allows the browser and the user to validate that the site is who they say they are according to a set of authorities like Verisign or Thawte. These are the folks whose job it is to verify that the certificate is being issued to a valid company (note that I said valid, not necessarily reputable as it’s not the job of certificate authorities to determine reputation).

And on a more technical level, as a user, SSL certificates keep traffic between you and the website you are interacting with more secure. Looking at this via the OSI model for networking; since all HTTP traffic happens at the application layer (layer 7), when SSL is not present, everything happens over plain text communications and can be sniffed. SSL, which is a network protocol, occurs at layer 6 (the presentation layer) and therefore can encrypt and decrypt all the communications that happen at layer 7 (if used).

So if we all bit the bullet and added SSL capabilities to our sites, the net result would be a more secure internet from a user perspective. There are plenty worse things that Google could be doing than forcibly making the internet more secure.

Related posts:

1. 10 More Tips Towards Securing Your Linux System
2. What Does Web 2.0 Mean To You?
3. Custom Google Maps Marker With YM4R_GM

So I took my SysAdmin based love of making things quicker and learned enough Applescript to let Alfred make my life easier. I wrote a script that when a domain argument is passed to it in Alfred, it will open up a Quantcast tab, a Compete tab, and an Alexa tab for that domain in Google Chrome (Note: With the script below, it MUST be Google Chrome). To install, go to the Alfred preferences and create a new Applescript extension.

Fill out the text boxes as follows:
Title: domstat
Description: Get the domain statistics for
Keyword: domstat

Now put this in the script box:

To execute, just fire up Alfred and type: “domstat eric.lubow.org” and it will fire up Google Chrome and open up the tabs.

Update:
You can even download the Alfred extension directly from here.

Related posts:

1. The Next Step In Browser Evolution

The error looks something like this:

The reason for the error is because CentOS (at least the RightScale version 5.6.8.1 has an old certificate authority bundle: /etc/pki/tls/certs/ca-bundle.crt.

I backed up the existing certificate file just to be on the safe side.

To fix the issue, just download a new certificate bundle. I used the one from haxx.se.

Related posts:

1. Stopping Curb From Segfaulting
2. Fixing zlib Errors On Capistrano Deploy

In the usual spirit of social coding, I have added the script to Github: https://github.com/elubow/ec2-consistent-snapshot.

Running it is just this:

The options used here (for reference) are telling ec2-consistent-snapshot to use –mongo, on the –xfs-filesystem /data, in the us-east-1 –region (note that it’s just the region and not the availability zone within that region), to be backed up with the listed –description of the specified volumes. You can even throw a –mongo-stop in there to have Mongo stopped before the file system freeze and then restarted after the volumes have been backed up. Don’t forget that you need to set your Amazon keys in you environment variables (AMAZON_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for your key and secret respectively).

I attempted to keep the usage style consistent with Eric Hammond’s original version, just add Mongo support for it.

Note: I also mentioned this on the mailing list. But given the amount of messages that fly around on the list daily, some folks may have missed it.

References:

• ec2-consistent-snapshot blog entry by Eric Hammond
• ec2-consistent-snapshot on Github with Mongo DB support
• Backing up MongoDB on EC2 (10gen)

Related posts:

1. Getting a Random Record From a MongoDB Collection

1. Boxcar

   
   Just is case you haven’t heard of Boxcar, it’s what notifications for the iPhone should have been. You can get push notifications for a ton of different services ranging from Facebook, Twitter, and email to Github or even something more custom (for those of you techies who read this blog). This awesome iPhone application has recently been released for Mac desktop. This means that those same notifications that you used to have to have tabs open for Facebook, Twitter, RSS feeds, Email, Github, or whatever other services you use are now all centrally located. Boxcar for Mac is still beta-ish so expect it to get a lot better. But centralized notifications helps to prevent you from checking all 80,000 (or so) locations for new items to distract you.

2. Alfred

   
   Alfred App is what Spotlight should have been plus some. It is by far the application that I use the most on my Mac. It means that I grep through files, search my entire filesystem and either open a file or find the containing folder and open it up. And with the Powerpack you have the clipboard manager (which happens to be my favorite feature). It does favorite snippets and can save old clipboard contents for long period of times that can searchable. If you try Alfred and it doesn’t make your life easier, then you are using it wrong. I could go on for hours with how Alfred can make your Mac life better, but it’d faster and easier to just read the Tips Blog.

3. Caffeine

   
   Caffeine is not really a productivity app, but something more to prevent annoyance and generally an all around handy app to have. It does one thing and does it well. Caffeine prevents your computer from going to sleep. This is great if you have a short screen saver that you don’t feel like changing or if you are watching a movie on Netflix and don’t want your computer to go to sleep. There is something to be said for simplicity and doing something well.

4. Notational Velocity

   
   Mac sticky post-it style notes are good, but Notational Velocity has taken it to the next level. It’s freeform, searchable, remote-syncable, taggable notes (too many buzzwords, right?). But the fact is, you just start typing and it saves as you go. When you are done, you can add tags. And if you have an iPhone, then you can install SimpleNote and have your notes from the computer sync’d to your phone (and vise versa). But my favorite thing is just the fact that you can start typing and it is immediately searchable. I have it open on all my spaces and I am constantly making notes. I take items from my Alfred clipboard and paste them into NV as notes for how I get stuff working. This way I keep track of everything I tried and then just remove the things I don’t use (and then use those notes to write a blog post).

5. Homebrew

   HOMEBREW

   I have tried the gamut of package management for the Mac. I compiled things from source (and that just gets messy). I have also tried Fink and Macports and they just both felt a little hackish given the naturally usable feel of OS X in general. So I installed Homebrew and everything just sort of fell into place. It’s just as simple as “brew install $package” (after Homebrew is installed of course). And since every package installed is installed in isolation (/usr/local/Cellar), removing and upgrading can also be done with ease. If there was a solid GUI in front of it, I would recommend Apple adopt it as a 3rd party package management system.

If there are other packages or apps for the Mac that has had a great impact on your productivity, let me know.

Related posts:

1. Things (Todo App)
2. Transferring Email From Gmail/Google Apps to Dovecot With Larch
3. Creating Dummy Packages On Debian

This allows me to navigate the bash console with the usual vim suspects: h, j, k, l. In addition to that, I also get some fun ones like word movement w and the dw that goes along with it.

But the big winner for me is now I am able to use the vim environment and movement keys inside the irb (Ruby), Mongo, MySQL consoles (still not Redis though). To do that, just add the following lines to the following files:

No related posts.

• Count all the items in a bucket. The SQL equivalent being: SELECT COUNT(*) FROM foo.
  -- Assuming that 'visits' contains all visits to your website (for example)
  -- Returns: (100L)
  total_visits = FOREACH (GROUP visits ALL) GENERATE COUNT($1);
• Grouping on multiple elements in a bag. Assuming you have a bag with 4 tuples that looks like this: (1,Football),(2,Soccer),(1,Soccer),(2,Soccer). You may want to know how many of user type 1 are “Football” or “Soccer” and how many of user type 2 are “Football” or “Soccer”. Note: If you want user_type and sport in a separate bag, just remove the FLATTEN($0).
  -- Group by user_type and then by sports interest
  -- Returns: (1,Football,1L),(1,Soccer,1L),(2,Soccer,2L)
  --          {group::user_type: chararray,group::sport: chararray,total: long}
  sports_interests_by_user_type = FOREACH (GROUP user_type BY ((CHARARRAY)$0, (CHARARRAY)$1)) GENERATE FLATTEN($0), COUNT($1) AS total;
• Add a field to a every element in a bag. From my understanding, this next bit is a Pig 0.6ism. This will join each by 1 thus creating a tuple with an implicit join of 1. The outcome will be a similar effect to an array push of a field onto the end of every tuple in a bag.
  -- Add total visits to every sports_interest_by_user_type
  -- Returns: (2,Soccer,2L,100L)
  --          {sports_interests_by_user_type::group: chararray,sports_interests_by_user_type::total: long,long}
  sports_interests_by_user_type_fraction = JOIN sports_interests_by_user_type BY 1, total_visits BY 1;
• Let’s take field that we added to the end of the tuple and get a percentage out of it. This will return the total out of 100%.
  -- Divide the number of user_types per interest by the total
  -- Returns: (2,Soccer,1.0F)
  --          {sports_interests_by_user_type::group::wv: chararray,sports_interests_by_user_type::group::area: chararray,float}
  sports_interests_by_user_type_percent = FOREACH sports_interests_by_user_type_fraction GENERATE (CHARARRAY)$0, (CHARARRAY)$1, (FLOAT)(((FLOAT)$2 / (FLOAT)$3) * 100);

This post is another example of work that I could not have accomplished without the help of people on #hadoop-pig on irc.freenode.net. Also worthy of note are the Pig Latin manuals here and here.

Related posts:

1. Pig Queries Parsing JSON on Amazons Elastic Map Reduce Using S3 Data
2. Creating a Process Table hash in Perl
3. Using Unique Keys and Key Groups with Background Jobs in Gearman::Client

First off, I am going to assume that you have an Amazon Web Services account (AWS) and you have also signed up for Elastic Map Reduce (EMR). For all this, I followed the instructions in this video by Ian @ AWS to get me going. Now SSH into the machine so we can get started.

As of the time of this writing, EMR is using Hadoop 0.20 and Pig 0.6. Everything I am going to talk about is for Pig 0.6. With any luck, upgrades to Pig will have taken a lot of this into account. Once you are on the EMR master host, type the following commands to get elephant-bird downloaded. We are going to use it to build a jar that will parse our JSON (big thanks to Dmitriy for all the help here). Note: The reason we are pulling with wget as opposed to git directly is that we want the jsonloader branch and this is just easier.

At this point we should have 3 jars in the pig-jars directory. I created an S3 bucket for myself and put the jars in there so I only have to do the compilation once. From here on in, I will be referencing those jars using my s3 bucket. If you like how my logs are organized, then I strongly recommend checking out Cloudera Flume for log aggregation. <shameless plug>I also wrote a blog post here on getting it going.</shameless plug>

Another item worthy of note is that I store all my logs in gzip format. Although this isn’t the best format for Hadoop in the long run because it can’t be split into chunks, it’s what I used. I had trouble getting everything going because Pig doesn’t decompress files when running in local mode. Please learn from that mistake of mine.

Now let’s get into the code a little. First thing we do is register all the jars necessary to parse JSON. This is done using our jars that we put directly on S3. Then we load up the JSON into maps. This is done using the JsonLoader(). You have to use the full path to it (which is listed in the code sample). Now the “interesting” thing about my log files is that they have 3 distinct types of log lines in them. Types i,b, and c. Each log line type has a different meaning so I sort them into 3 groups with the SPLIT command and some conditionals.

Now that I have my data broken out into the 3 buckets, I can start doing what I want with them. Let’s say that in log type i, there is a widget_value. And that widget_value is a string of any number. To show the top 5 values of that widget_value has, I just pull out all instances of widget_value in i. Then I iterate over those values and group them together by type (thus getting aggregate values). And finally I sort them in descending order and show only the top 5.

Last thing I want to share is some tips on getting everything going with Pig:

• Start small and continue small until everything is working
• Use subsets of your data that you have a good idea of what the results are going to be before you run your queries
• Step through you queries to ensure each step is doing what you think it’s doing
• Cast your data types to avoid weird behaviors. Map doesn’t always leave your variables in the type you want/expect

And I can’t forget to say thanks for all the help to the people who hang out in #hadoop-pig on irc.freenode.net.

Related posts:

1. JSON Benchmarks in jRuby
2. Distributed Flume Setup With an S3 Sink

As CTO of SimpleReach, a company that does most of it’s work in the cloud, I’m constantly strategizing on how we can take advantage of the cloud for auto-scaling. Depending on the time of day or how much content distribution we are dealing with, we will spawn new instances to accommodate the load. We will still need the logs from those machines for later analysis (batch jobs like making use of Elastic Map Reduce).

I am going to attempt to do this as step by step as possible but much of the terminology I use is described in the users guide and there is an expectation that you have at least skimmed it prior to starting this HOWTO. I am using EMR (Elastic Map Reduce) on EC2 and not the provided Hadoop by Cloudera. Additionally, the Cloudera version that I am working with is cdh3b3.

Context
I have 3 kinds of servers all running CentOS in the Amazon cloud:

1. a1: This is the agent which is producing all the logs
2. c1: This is the collector which is aggregating all the logs (from a1, a2, a3, etc)
3. u1: This is the flume master node which is sending out all the commands

There are actually n agents, but for this example, we’ll keep it simple. Also, for a complete copy of the config files, please check out the full gist available here.

Initial Setup
On both a1 and c1, you’ll have to install flume-node (flume-node contains the files necessary to run the agent or the collector).

On u1, you’ll need to install the flume-master RPM:

On each host, you need to copy the conf template file to the site specific config file. That is to say:

First let’s jump onto the agent and set that up. Tune the $master_IP and $collector_IP variables appropriately, but change your /etc/flume/conf/flume-site.xml to look like:

Now on to the collector. Same file, different config. Replace all the variables with you $master IP address (you should be using Amazon’s internal IPs otherwise you will be paying the regional charge). The $account and $secret variables are both your Amazon EC2/S3 account key and secret Access key respectively. The $bucket is the S3 bucket that will contain the log files. Also worthy of pointing out is the flume.collector.roll.millis and flume.collector.dfs.compress.gzip. The millis is how frequently the log file gets truncated and the next file begins to be written to. It would be nice if this could be done by file size and not only by time, but it works for now. The other config option is flume.collector.dfs.compress.gzip. This ensures that the logfiles are compressed prior to being dumped onto S3 (saves LOTS of space).

While we are still on the collector, in order to properly write to S3, you’ll need to make 4 file adjustments and all of them will go into the /usr/lib/flume/lib/ directory.

1. commons-codec-1.4.jar
2. jets3t-0.6.1.jar
3. commons-httpclient-3.0.1.jar
4. emr-hadoop-core-0.20.jar

The one thing that should be noted here is that the emr-hadoop-core-0.20.jar file replaces the hadoop-core.jar symlink. The emr-hadoop-core-0.20.jar file is the hadoop-core.jar file from an EC2 Hadoop cluster instance. Note: This will break the ability to seamlessly upgrade via the RPM (which is how you installed it if you’ve been following my HOWTO). Keep these files around just in case. I have added a tarball of the files here, but they are all still available with a quick Google search.

And now on to the master. There was actually no configuration that I did on the master file system to get things up and running. But if flume is writing to a /tmp directory on an ephemeral file system, then it should be fixed.

Web Based Setup

I chose to do the individual machine setup via the master web interface. You can get to this pointing your web browser at http://u1:35871/ (replace u1 with public DNS IP of your flume master). Ensure that the port is accessible from the outside through your security settings. At this point, it was easiest for me to ensure all hosts running flume could talk to all ports on all other hosts running flume. You can certainly lock this down to the individual ports for security once everything is up and running.

At this point, you should go to a1 and c1 run /etc/init.d/flume-node start. If everything goes well, then the master (whose IP is specified in their configs) should be notified of their existence. Now you can configure them from the web. Click on the config link and then fill in the text lines as follows (use what is in bold):

• Agent Node: $agent_ec2_internal_ip
• Source: tailDir(“/mnt/logs/”,”.*.log”)
• Sink: agentBESink(“$collector_ec2_internal_ip”,35853)

Note: I chose to use tailDir since I will control rotating the logs on my own. I am also using agentBESink because I am ok with losing log lines if the case arises.

Now click Submit Query and go back to the config page to setup the collector:

• Agent Node: $collector_ec2_internal_ip
• Source: collectorSource(35853)
• Sink: collectorSink(“s3n://$account:$secret@$bucket/logs/%Y/%m/%d/%H00″,”server”)

This is going to tell the collector that we are sinking to s3native with the $account key and the $secret key into the $bucket with an initial folder of ‘logs’. It will then log to sub-folders with YYYY/MM/DD/HH00 (or 2011/02/03/1300/server-.log). There will be 60 gziped files in each folder since the timing is setup to be 1 file per minute. Now click Submit Query and go to the ‘master’ page and you should see 2 commands listed as “SUCCEEDED” in the command history. If they have not succeeded, ensure a few things have been done (there are probably more, but this is a handy start:

1. Always use double quotes (“) since single quotes (‘) aren’t interpreted correctly. UPDATE: Single quotes are interpreted correctly, they are just not accepted intentionally (Thanks jmhsieh)
2. In your regex, use something like “.*\\.log” since the ‘.’ is part of the regex.
3. In your regex, ensure that your blackslashes are properly escaped: “foo\\bar” is the correct version of trying to match “foo\bar”.
4. Ensure any ‘/’ are inserted as ‘%2F’ in the Amazon account and secret codes.

Additionally, there are also tables of Node Status and Node Configuration. These should match up with what you think you configured.

At this point everything should work. Admittedly I had a lot of trouble getting to this point. But with the help of the Cloudera folks and the users on irc.freenode.net in #flume, I was able to get things going. The logs sadly aren’t too helpful here in most cases (but look anyway cause they might provide you with more info than they provided for me). If I missed anything in this post or there is something else I am unaware of, then let me know.

References

• http://wiki.apache.org/hadoop/AmazonS3
• Flume Users Guide
• irc.freenode.net #flume
• https://issues.cloudera.org/browse/FLUME-66
• Config files gist

Related posts:

1. Pig Queries Parsing JSON on Amazons Elastic Map Reduce Using S3 Data
2. File::Find