hernan43's github blog

Finally something to work on

2010-01-25T00:00:00-08:00

Months ago, I became inspired by Giles Bowkett’s idea of doing a miniapp a month. He made a very cool app to read Hacker News in the format of a traditional newspaper.

I am always in the process of developing my skills, so I thought the concept of using miniapp development was awesome. I can come up with an idea for a simple app, then start working on it. I’m not prolific enough to do one per month, but I figure something is better than nothing.

Recently, I’ve become obsessed with learning more and more about MongoDB. After reading scads of articles on MongoDB and mongomapper on John Nunemaker’s blog I decided I just have to wrap my head around MongoDB and figure out if it is something that is valuable to me.

Having trouble as I usually do trying to figure out what miniapp to work on next, I think I’ve finally come up with something that I can work on that will snuggle up with MongoDB nicely. I don’t want to get into what it is just yet. If I did and found out I was out of my depth I’d feel like a total nozzle. So hopefully you can watch this space and see what I come up with.

Filtering my tweet spam with Ruby

2009-12-14T00:00:00-08:00

I watched Hillary Mason’s video titled How to Replace Yourself with a Very Small Shell Script a few weeks ago, and it got my creative juices flowing. The gist of the video was that by using document classification you could determine the importance of an email message.

In the video Hillary suggests using this method on your Twitter feed. This was where I got all fired up. It is all too common for me to hear Tweetie bleep, I swivel my chair around to see what new tidbit someone has tweeted to find, “I like candy…” Or some other inane pap.

If I could use some method of document classification, maybe I could train a Twitter client to identify which tweets where interesting, I could avoid the constant “need coffee” tweets. A ha!

Enter the Ruby classifier gem. The classifier gem takes care of all the mathematical leg for you. It makes it a simple exercise to train my tweets as being ‘interesting’ or ‘uninteresting’. At least in theory.

# create my classifier
c = Classifier::Bayes.new
c.add_category "interesting"
c.add_category "uninteresting"

# train items
c.train 'uninteresting', "#{uninteresting_tweet.content}"
c.train 'interesting', "#{interesting_tweet.content}"

# classify/guess which type of tweet it is
c.classify "#{tweet.content}"

Using the above code I can train my tweet classification database and determine which tweets are interesting and which ones aren’t.

So far, I have only done one round of testing. I trained about 30-40 tweets and ran the classifier on at least another 30-40. I am pretty surprised at the accuracy. The ones it marked as being uninteresting were ones I would have normally ignored. The ones it marked as interesting were ones I would mostly consider worth reading. At least as worth reading as a tweet can be ;-).

This first experiment has paved the way for me to expand this into a bigger project. Considering I am not particularly happy with any one Twitter client, I think I may write my own and incorporate the Bayesian classification. At least as an excuse to play with this idea a bit more.

Wrapping my brain around long polling with Orbited/Rails

2009-11-02T00:00:00-08:00

I’ve been reading about Comet/long polling for a while now. It seems to be becoming a popular topic of discussion. There are a lot of opinions and theories on it. Not having any experience with it myself, I’ve only pseudo-understood what it was.

Wikipedia has a nice breakdown of what it is. Basically, it is a way for a web application to ‘push’ data to the client using long running HTTP requests.

Now that I’ve gotten around to reading up on it, I decided a week or so ago to try and do a simple implementation of it. I noticed that a lot of people use the chat room concept with Comet, so I decided to try and make a simple app that implemented a chat room. Kind of like when I discovered Ruby on Rails, I decided to do the quintessential ‘make a blog with rails’ experiment.

Try #1 – Nginx push module

Without knowing any better, I decided to try out the new-ish Nginx http push module. I’m familiar enough with Nginx. I figured it would be easy. It sort of was.

I followed the instructions for install Nginx with the http push module. Everything seemed to work as advertised. I even tried the sample code from Nginx & Comet: Low Latency Server Push. Everything worked as advertised. I was pretty impressed.

In my app I used the delayed_job plugin to queue messages to be sent to the publish URL. For the most part it appeared to be working correctly. I could see the messages entering the queue and I could see that they were getting published to my Nginx polling server.

Unfortunately, my Javascript/Prototype skills are clearly not up to par. I was unable to get my chat room to work reliably. It would only receive messages sometimes and it seemed to lose track of my polling server.

I spent a day trying to figure out what I was doing wrong and finally threw my hands up in defeat, grabbed my Xbox controller, and cooled off to some Borderlands.

Try #2 – Orbited

The difference between my first try and this try were night and day. The Orbited setup was seemingly simpler than my Nginx one.

Orbited is a Python server using the Twisted framework. Twisted is an extremely stable/reliable networking framework. It has been around for as long as I have been using Python

Instead of having to have Nginx running + Apache + the delayed job runner running, I could slim it down to just Apache + Orbited. Also the Orbited plugin made integration into Rails extremely easy. Installing the plugin created a config file with defaults set the same as the defaults in the Orbited config. So I really didn’t have to configure it to point to Orbited at all.

In my HTML template I had to add the following to my head section:

<%= orbited_javascript %>

And then for the chat channels I just did:

<%= stomp_connect(params[:slug]) %>

The params[:slug] variable is set to whatever my current chat channel is. It was really simple.

In my messages controller I have a private method called send_message_via_orbited(message). This method sends newly created messages to the default Orbited STOMP service(MorbidQ) and then to my channel’s page.

def send_message_via_orbited(message)
  orbited_data = render_to_string :update do |page|
    page.insert_html :bottom, 
		'messages', 
		:partial => 'channels/message', 
		:locals => {:message => message}
    page['message_form'].down('form').reset
  end
  Orbited.send_data(message.channel.name, orbited_data)
end

All this code does is updated the messages div with the new message. This is done via RJS which sends Javascript through Orbited to the page where it is eval’d.

Aaaaaaand done!

Orbited was extremely easy and worked pretty reliably for the little bit of testing that I did. You can browse the source on GitHub, I called it pollchat.

My so called Hacker Blog

2009-10-18T00:00:00-07:00

After trying out Wordpress for a few months now, I’ve decided to leave it in favor of GitHub Pages and Jekyll. Jekyll is a static site generator with a penchant for blogs.

Years ago, I used to use a piece of software called, PyBlosxom. PyBlosxom is a Python based blog generator, that stores posts in individual files and can be used to generate a static site.

Much like PyBlosxom, Jekyll also stores blog posts in flat text files. It can process many of the most popular markup engines and is highly customizable. One nice feature of Jekyll is that it is what powers GitHub Pages, and therefore you are able to use your free GitHub account to run a blog.

My favorite part about this is that I can edit posts using VIM, push the changes to GitHub, and BOOM! My blog is updated.

Why leave Wordpress?

I really like the Wordpress blogging platform. It is free, extensible, and has a really nice administrative interface.

But I am a hacker at heart.

I like the idea of using a simple text editor and some command line tools to be able to publish new posts. I also like the idea of having a distributed, version controlled copy of my blog site.

Migrating my Wordpress posts

I hemmed and hawed about whether I should take my Wordpress.com posts along with me on this new adventure. Jekyll had a bunch of converters for the popular blog engines, including Wordpress, but they required database access. Since my blog was on Wordpress.com, I only had access to an XML dump of all my posts.

Relenting, I decided to bring my old posts over. I used Hpricot to whip up a simple shell script that would parse out all of my old posts, and convert them to the Jekyll file convention.

#!/usr/bin/env ruby
#
# usage:
# ./wordpress_xml_to_jekyll.rb wordpress.xml <jekyll-dir>/_posts
#

require 'rubygems'
require 'hpricot'
require 'erb'

# my posts are pretty basic
post_template = ERB.new <<-EOF
---
title: <%= title %>
layout: post
---
<%= content %>
EOF

# wordpress xml file
xml_file = ARGV[0]
# where to put the posts, i.e. <jekyll_dir>/_posts/
output_dir = ARGV[1]

# read xml in
xml = open(xml_file).read
# convert to Hpricot object
doc = Hpricot(xml)

(doc/"item").each do |item|
  # get the particulars for each post
  title = "#{item.search('title').first.inner_text}"
  slug = item.search("wp:post_name").first.inner_text
  date = DateTime.parse(item.search("wp:post_date").first.inner_text)
  content = item.search("content:encoded").first.inner_text
  # create a file name for the post
  file_name = "#{date.year}-#{date.month}-#{date.day}-#{slug}.liquid"
  output_file = File.join(output_dir, file_name)
  # render my ERB template
  file_contents = post_template.result(binding)
  
  file = open(output_file, 'w')
  # write the post to its respective file
  file.write(file_contents)
end

Wanna try blogging with Jekyll?

Here are some really good resources on using Jekyll, Git, and GitHub Pages.

first post!

2009-10-15T00:00:00-07:00

This is my first post to my github pages page.

Revisiting an old friend

2009-09-28T00:00:00-07:00

A few years ago, circa 2003, I started a new job at my current employer. Everyone there used IRC for everything. We had a guy that was telecommuting so it made it easier to collaborate with the whole team. As long as we discussed it in IRC, everyone was able to participate

We had a large number of systems that we monitored using Big Brother. It was quite nice, but we always had to have our web browser tuned to the Big Brother monitoring app, or we had to be sure to pay attention to our email.

Naturally we didn't always do either so sometimes 5, maybe 10 minutes would go by before we noticed a system was having trouble. I thought it would be cool to create an IRC bot that posted system notifications in an IRC channel.

This worked great because everyone's client would let you know when there were new messages in the channel. If we wanted to we could (and did) colorize 'info' messages one color, and 'severe' messages another and so on.

We loved the bot so much that on my last day there, I accidentally deleted my home directory on that machine (I got a little overzealous with the rm -rf). My boss had me leave the box on so they could leave the bot running in core memory for a number of months. I didn't have another copy of the code and needless to say I was cheesed off.

Last year I decided to sit down and re-create it as best as I could. I sent it to my old team mates so they could use it again if they wished. I've also open sourced it in case anyone else would like to play with it.

I call it the Telnecho Bot.

A few days of testing

2009-09-16T00:00:00-07:00

So I started an exercise to learn TDD with Ruby on Rails.

I created a project on git and started coding it up whenever I get a few spare minutes here and there.

It has been slow going but I am creeping along. Thoughtbot's factory-girl and shoulda have been making it much easier.

I am currently reading 15 TDD steps to create a Rails application and its children to help me get started with getting into the TDD mindset.

My Rails Rumble 2009 Entry

2009-09-09T00:00:00-07:00

So I didn't create a finals-worthy entry in the 2009 Rails Rumble but that doesn't mean it isn't useful

I've decided to open source my entry so that other people can take a look at it and maybe even use it or steal from it.

Check out the github repository for laterz here.

The Rumbly in My Tumbly

2009-08-24T00:00:00-07:00

So I threw my hat into the ring this weekend and participated in the 2009 Rails Rumble. The gist of it is you have 48 to make a web app based off of a Rack compatible web framework.

I've never done anything like this before and I was pretty nervous. My idea was revolved around a common habit of mine. When I write these posts, I tend to build up folders of bookmarks that I will use for research. I go through my bookmarks, do my post, and then I trash the folder. Sometimes I forget to delete these folders or I neglect to do it for whatever reason. Next thing I know, my bookmarks bar is full of these mothers and I'm deleting a whole bunch of them.

So my app, Laterz, lets me use a bookmarks bar shortcut to auto-save URLs for me to go over later. I jokingly refer to it as time shifting URLs.

With Laterz, you login with OpenID and have a storage area for URLs. You can create categories to organize them if you want. Each category gets its own shortcut that can be copied to your bookmarks bar to auto-save URLs to that category.

It is pretty simple. I wish my design skills were a bit better so that it wouldn't look like some kind of PHP Nuke template from 1998 but oh well.

Rumble in the Internet Jungle

2009-07-13T00:00:00-07:00

Well I've casted my lot and am entered in the Rails Rumble 2009. The Rails Rumble is a 48 hour coding competition using Ruby on Rails.

The actual competition doesn't happen until August 22nd, 2009. I have a good idea about what I am going to do. The only real question is how much I can accomplish in 48 hours.

I am positive I can get the basics working in 48 hours but my concept has some pretty grand ideas and I would love to finish a few of them before the time is up. Who knows, maybe I'll even win a prize.

See the Rails Rumble Blog for more info.

Been workin

2009-06-25T00:00:00-07:00

I've been working on a really small project that I've decided to share with the world. I would love for people to try it out. Hopefully I can get a handful of users using it so I can see my creation in action.

Without further ado, I present my little baby, eniv.us.

Here is a blog entry that has more detailed info:

http://enivus.wordpress.com/2009/06/25/eniv-us-my-little-experiment/

He also has his own Twitter page.

Got Root?

2009-06-02T00:00:00-07:00

I was walking to lunch this afternoon and for some reason I conjured up a long lost memory. A memory from a headier time when I was a young college student. I worked for a small department on campus at Michigan State University as a Linux system administrator.

This was not a memory of a great triumph or some great prank. This was a memory of an event that changed my life as a coder and as a sysadmin. The memory that came into my head was of the very first(and hopefully last) time I got rooted.

As I sit here typing this up on a rainy evening, I am reminded of the day it happened. It was an early morning just as a rainstorm was finishing up drenching everything. It was mildly foggy as I walked into the now demolished Paolucci building on the campus of Michigan State University.

Prologue

I came upstairs to the student office and was greeted by a fellow student. "Some guy from the Computer Center called. He said something was wrong with one of our computers. He wanted you to call him as soon as you got in. Also they shut off the internet connection to our building."

"What the hell? What could have been wrong with one of our machines. We only have a couple that do anything of any substance." I sat down and started dialing the number my coworker gave me. The person on the other end of the phone told me that one of our boxes had been hacked, and that it was DDoS'ing the campus network hardware north of the river. It was causing network trouble so they shut off our building's network connection. He also told me to touch nothing, he was going to be there in a little bit.

Needless to say there was no way I was going to not touch anything. Sorry security dudes. I promptly logged onto the machine and started poking around. Even with my limited knowledge I was able to somewhat easily find out what was the cause of all the trouble. The attacker didn't even really make any attempt to cover their tracks. Luckily this machine was a new install so it didn't have anything I couldn't stand to lose.

The Day Before

The day before I decided to setup a DHCP server for my office to allocate IPs to the various laptops, desktops, and whatnot that were in our office. I thought it would be a clever use of my time and maybe I'd learn something new, and boy was I right!

That same day I installed a copy of RedHat(which was probably version 6.2 or 7). After the install finished I booted it up and left for lunch with a hungry belly and a promise to come back later and finish setting everything up.

Well as it turns out I forgot all about the box and got caught up with other things to do. I finished out my day and left for home, thinking of better days and brighter futures.

Meeting the Security Dude

So that leads me into the next day. Or as I like to call it, The Day I Learned What "Owned" Meant. There I was waiting for the Security Dude to show up. My heart was pounding. What were they going to do? What were they going to say? Was I going to get fired? Were they going to think I did it? It was too late to think about all that the Security Dude was here. The interaction went something like this:

Bud	"Hello. My name is Security J. Dude III. Nice to meet you. You may call me, Bud."
Me	Hey, my name is hernan43. Oh well, that wasn't so bad. He seemed nice. He even let me call him by a nickname(I changed the name in case you couldn't tell).
Bud	Did you make sure not to touch anything?
Me	Uh well I logged in before I called you and poked around, but I haven't messed with it after that. Oh yeah I lied like my life depended on it.

Bud either bought my lie or did a good job at faking annoyance at my ignorance, either way he sat down at the affected machine and got to work. He typed in a series of seemingly complicated commands and watched the text scroll by soaking in all of the information before turning in his chair to address me.

Bud informed me that the particular version of RedHat I used installed an ftp server by default that had a known exploit. He told me the attacker used a rootkit to assume control of the machine. He showed me the terminal where he had one of the system logs being displayed. He pointed to a single line, "Right here is the point where you were pretty much owned."

He mentioned that had I left the machine alone it would have been more likely that it could have been used to catch the attacker. Finally he told me about Tripwire and the importance of installing the OS with the network unplugged until I was in a position to patch it.

Bud was a little cheesed that I was trying to setup my own DHCP server in the office since there was a possibility that if I didn't configure it correctly it could interfere with the main campus DHCP service.

Despite the situation he was pretty kind and understanding. I was expecting him to chastise me for being such a dilettante, but I guess he sees this stuff so often that he probably couldn't help but feel sorry for me.

The Aftermath

Once Bud was finished, he told me that they would reactivate our network when he got back to his office since he was pretty much confident that the problem was solved.

I was worried that I was in some major caca when my boss got in, but when I told him the story he seemed indifferent. "Did anything get taken?" "No" "Did they turn the internet back on?" "Yes" "Ok well, don't do it again. Back to work."

I learned many a valuable lesson that day. Specifically I started taking security way more seriously. I started believing people when they said things like, "someone can crack your machine at the drop of a hat so make sure you lock down XYZ thing."

Luckily for me I have not been rooted since. It is a feeling that I do not want to experience ever again. Even when I was a "green as green" green horn and I knew it, I still felt embarrassed beyond belief having to look Bud in the eye and come to grips with the fact that I got a box rooted. I can only imagine how low I would feel today.

Hmm... I better stop typing and check my patches...

Walk the Command Line

2009-04-30T00:00:00-07:00

When I was a young whipper snapper in college I was learning how to use UNIX and I came from a world of Windows and Mac OS. I found the CDE interface to be putrid at best. I also thought the command line stuff we had to do was an archaic holdover from caveman days.

I hated using the command line but I thought, "well, I have to crawl before I can walk." I mean, if it doesn't have a GUI it must be a pile of trash, right? So I put my head down, sucked it up, and trudged my way through countless classes stuck on command line UNIX hoping for better days ahead. No wonder no one uses UNIX operating systems.

Fast forward 13 years and here I am a command line chauvinist. I do everything I can on the command line. I seek out command line interfaces to *everything*. If some new thing floats across my way and I like it, the first thing I ask myself is, "I wonder if it has a command line interface..."

Thanks to places like commandlinefu.com I can now feed my need like never before. Like Delicious bookmarks for command line snippets, commandlinefu.com is loaded with gems.

Some of my favorites are:

The command line is a treasure trove of automation and productivity enhancements. Not to mention it is seemingly a lost art. It seems as though, more and more, when I tell people to fire up a command line and run a specific command I am greeted with a groan. Is the command line truly dying? Or do I just hang out with too many GUIites?

Why add my voice to zillions of others

2009-04-27T00:00:00-07:00

I signed up for a Wordpress API key many a year ago. I needed it to try out Akismet. I was teaching myself how to write Django apps and was doing the quintessential "build your own blog" method of self instruction.

What I did not realize at the time was that it created a Wordpress.com blog for me when I created my account. Maybe I did realize it at the time and didn't care. Either way, I forgot it existed only to stumble across it years later.

Now here I am at a Wendy's killing time before I have to go back to work. I have TextMate open and a spicy chicken sandwich at my side. I'm not really a writer and I have tried blogging in the past only to let my poor blog languish away in a furious firestorm of inactivity.

So why am blogging now? Well I read somewhere that in order to make yourself more appealing to potential employers you should have a blog that you use to engage with the community.

I have no idea if this is at all effective or if I will even make my way back into the world of blogging but I guess I should try. Summer is coming back, the weather is changing, my garden is getting ready to burst forth with life, and I have a new found excitement for the spice of life.

Hello world!

2006-08-05T00:00:00-07:00

Welcome to WordPress.com. This is your first post. Edit or delete it and start blogging!