unethical blogger

Async sockets in Ada

2013-04-07T00:00:00-07:00

Recently I've been experimenting with the concept of a high-performance (high parallelism, high concurrency, low CPU/memory footprint) TCP server in Ada. Developing on top of Linux, you cannot help avoid the epoll(7) I/O event notification system.

Unlike my previous post on dynamic task creation in Ada, this post won't have much in the way of sample code and walk-through. Mostly because I didn't want to spend the entire post explaining epoll(7) concepts.

If you're currently unfamiliar with epoll, I found this post from banu.com interesting and useful. In general, I think that it is very useful to understand how both epoll(7) and its BSD-friendly pal kqueue(2) work, and how the two have influenced evented I/O systems that have become popular in the past few years.

For my experiment, the primary challenge with using epoll(7) is wrapping the C code with Ada, which on its face this isn't terribly difficult. That said, mapping C arrays and other stupid pointer tricks that C libraries are prone to using, can make writing an Ada binding much more difficult. I don't intend to dive too deep into how one can create such a binding since I think Felix Krause's post on writing thin/thick C bindings in Ada is a pretty good introduction on that subject. Personally I prefer "thicker" bindings for use in Ada, just as I expect Ruby libraries to abstract away the more fickle aspects of the underlying C API.

What I did want to share in this post was the actual working results of my tinkering, which can be found here, on GitHub.

The basic flow of the program (found in main.adb) is similar to the example found in the epoll(7) man page, in that it sets up the epollfd, binds the listening to it and then waits for Epoll.Wait to return with an array of socket descriptors which have activity on them.

My next step is to combine this basic epoll binding with my previous task related code and create a TCP server which uses a task pool to perform "work" on the incoming sockets given to it by an epoll-listener task.

Safe, Dynamic Task Creation in Ada

2013-03-09T00:00:00-08:00

A few years ago, Ada become my hobby/tinker programming language of choice, for a number of reasons, concurrency being one of them. In this post I'd like to walk you through an example of dynamic task creation in Ada, which uses Ada.Task_Termination handlers, a new feature in Ada 2005.

(If you're familiar with Ada, you can skip this next section)

Note: You can find all this code, and more in my ada-playground repository on GitHub

Similar to C, Ada supports stack allocated variables as well as heap allocated variabels, it also defaults to stack allocation. For example:

procedure Main is
    -- A stack allocated `Integer` object
    Enough_Memory : constant Integer := 655360;
begin
    null;
end Main;

If you wanted to allocate that Integer onto the heap, then you would use the new keyword:

procedure Main is
    -- A heap allocated `Integer` pointer
    Enough_Memory : access Integer := new Integer'(655360);
begin
    null;
end Main;

I won't dive too much into the minutia of what is going on here, if you're not familiar with Ada already you can learn more about access types on the Ada Programming Wikibook. Basically we're heap allocating a new Integer and using an access type (aka: typed pointer) to keep track of it. Keen readers will notice we didn't do anything with that Integer access type, and we're technically leaking the memory. To solve this we use the generic unit Ada.Unchecked_Deallocation, which gives you a facility for properly freeing memory (more details here).

Tasking Trickiness

Concurrency is part of the language in Ada, and is handled through tasking. A basic example of might be:

with Ada.Text_IO;

procedure Main is
    task Counter;

    task body Counter is
    begin
        for Count in 1 .. 10 loop
            Ada.Text_IO.Put_Line (Count'Img);
        end loop;
    end Counter;
begin
    null;
end Main;

The way tasks in Ada work means that the Counter task will be created, started and then the execution of the Main program will block until the Counter task completes (important detail).

The trickiness starts to arrive when you talk about dynamically allocating task objects, and combine that with something like an infinite loop, such as one might find in a server program, e.g.:

procedure Main is
begin
    -- Socket set up omitted
    loop
        declare
            Client_Socket : Socket_Type;
            Request_Handler : Handler_Ptr := new Handler;
        begin
            -- Block until we receive a new inbound connection
            Accept_Socket (Server_Socket, Client_Socket, Server_Addr);
            -- Dereference the Handler_Ptr and call `Process` on the Handler
            -- task
            Request_Handler.all.Process (Client_Socket);
        end;
    end loop;
end Main;

(The code above is an abbreviated version of echomultitask_main.adb which can be found here).

The issue with this code is that we're allocating a new Handler task for every in-bound connection, and we have no means of ever cleaning them up properly. If we were to create an Array of Handler_Ptr, we still would have to find some mechanism (which exists) to check the status of each Handler to determine if we should clean it up. Problem being, we'd have to loop through all the active tasks, checking for a "terminated" status, in order to deallocate them. It'd be much better if a task could tell us when it's finished, rather than us polling every one.

Fortunately in Ada 2005, a mechanism was added to make it easier to add "clean-up" to tasks: Ada.Task_Termination. The package allows you to set up a termination handler for the a specific task, which the runtime will call when that task terminates. Unfortunately however, the handler procedure that can be invoked when the task terminates will not be passed a pointer to the task itself, but rather the Task_Id (Ada.Task_Identification.Task_Id).

So close to being able to properly deallocate these dynamic tasks, but we need one more component, a protected object with a hash map inside of it:

package Server.Handlers is

    protected Coordinator is
        procedure Track (Ptr : in Handler_Ptr);
    private
        Active_Tasks : Handler_Containers.Map;
    end Coordinator;

end Server.Handlers;

Then back in main.adb:

    Accept_Socket (Server_Socket, Client_Socket, Server_Addr);
    Request_Handler.all.Process (Client_Socket);
    -- Make sure the we keep track of the Request_Handler in order to properly
    -- deallocate it later
    Server.Handlers.Coordinator.Track (Request_Handler);

(The code above is an abbreviated version of echomultitask-worker.ads which can be found here)

The singleton protected object Coordinator not only will give us protected (aka thread safe) access to the Active_Tasks map, but also gives us a protected object to hang our Ada.Task_Termination.Termination_Handler protected procedure off of:

protected body Server.Handlers is

    protected body Coordinator is

        procedure Last_Wish (C : Ada.Task_Termination.Cause_Of_Termination;
                             T : Ada.Task_Identification.Task_Id;
                             X : Ada.Exceptions.Exception_Occurrence) is
        begin
            -- Deallocate our task identified by `T`
            -- and make sure we remove it from `Active_Tasks`
        end Last_Wish;

        procedure Track (Ptr : in Handler_Ptr) is
            -- Dereference our `Handler` task, and fish out its Task_Id
            Handler_Id : Ada.Task_Idenfitication.Task_Id := Ptr.all'Identity
        begin
            -- Add `Handler_Id` to our `Active_Tasks` map
            Active_Tasks.Insert (Handler_Id, Ptr);

            -- Set up the Last_Wish procedure to be executed after our task has
            -- terminated
            Ada.Task_Termination.Set_Specific_Handler (Handler_Id, Last_Wish'Access);
        end Track;

    end Coordinator;

end Server.Handlers;

(The code above is an abbreviated version of echomultitask-worker.adb which can be found here)

This approach will allow us to safely create new dynamic tasks to handle the incoming requests, but will also make sure that the tasks are cleanly deallocated when they terminate.

If you're interested in concurrency in Ada, I highly recommend purchasing Concurrent and Real-Time Programming in Ada by Alan Burns and Andy Wellings, it's been tremendously helpful for my own concurrency exploration in Ada.

Tyler goes to Yurp

2012-12-13T00:00:00-08:00

As I mentioned in my previous post I'll be going to FOSDEM this year. Unlike last year, I will be spending a lot more time in Europe this time around.

I will be arriving in Frankfurt on the 29th of January and spending a few days in Köln (Cologne). Naturally I will be in Brussels for FOSDEM for a couple days, but after that, I will be heading to Berlin for a few weeks.

I've already spent a good amount of time in Germany, so I won't be on the "tourist track", having seen most of the sights in Berlin already.

So I will be working from Berlin, and my weekends and evenings will be open to as many meetups/hackathons as I can find. Even if you don't want to hang out (it's okay, I understand), but know of some good events in Berlin that are worth checking out, please send me an email (tyler@linux.com) and let me know!

What the hell is Yurp?

Having spent a large amount of my time in the southern US, "Yurp" is how far too many southern people pronounce "Europe", including those in my own family. That's why I'm going to Yurp.

FOSDEM: Testing and Automation Dev Room CFP

2012-11-18T00:00:00-08:00

For this next year's FOSDEM, I'm organizing the "Testing and Automation" developer room. And wouldn't you know it, we've just opened up our Call for Participation.

I'm quite excited about the dev room, and look forward to reviewing and selecting the talks.

Hope to see you there :)

Video: Rebuilding the airplane - Continuous Deployment with Gerrit and Jenkins

2012-10-21T00:00:00-07:00

I promise this is the last video I'll post for a while. This talk was given at Jenkins User Conference 2012 in San Francisco at the end of September.

This talk can be considered part of the same overarching narrative behind my Integrating Selenium after the fact presentation. Both talks are addressing the same continuous deployment/delivery project at Lookout.

I apologize for some of the clipping of the slides in the presentation, use your imagination to fill in the space to the left of the screen.

Slides

Video: Red, green, reprovision. Test Driving Operations

2012-10-20T00:00:00-07:00

One more presentation video from PuppetConf a couple weeks ago.

To be honest, I'm not terribly pleased with this presentation. It's the first presentation in a long time where I felt like I was overly-nervous to the point where it was affecting my ability to present. Watching the video after the fact though, seems like the presentation went alright.

(I also did a crap job explaining what BDD. Shit happens I suppose).

I still feel strongly about the subject, and look forward to revising this talk more in the future to cover "more advanced" topics.

Slides

Video: We'll Do It Live - Operations Anti-Patterns

2012-10-09T00:00:00-07:00

Another presentation video, this time from PuppetConf a couple weeks ago.

This presentation was a lot of fun to prepare for a give, since it was nothing but schadenfreude and beer, two of my favorite things. Most talks at PuppetConf were 35 minutes, but because he likes me so much Mike let me run over my time resulting in a crazy 45 minute talk.

Video: Integrating Selenium After the Fact

2012-10-08T00:00:00-07:00

A few weeks ago I had the fortune to speak at the SF Selenium Meetup with a talk titled: "Better Late than Never: Integrating Selenium After The Fact".

This talk is part of a larger Lookout narrative that I've discussed before around our epic journey to continuous deployment. Alongside some of the process changes we've enabled, we've also addressed a huge backlog of missing browser tests which has helped raise our confidence level before shipping code.

The talk might be a bit rambly, but I hope you find it useful.

Slides here

Juggling rice

2012-09-02T00:00:00-07:00

For the past couple weeks I've been learning how to juggle in my spare time (and a bit at work). The act of juggling itself is surprisingly easy, so I learned how to juggle in about 2 hours one afternoon, since then I've been learning how to juggle well, which is much more difficult.

Thus far I don't know how to do anything other than the standard 3-ball cascade, partly due to "poor equipment" (partially due to "sucking at juggling").

The juggling balls that I purchased on Amazon turned out to be incredibly light, which makes them much more difficult to catch properly. After lamenting this fact on the ##juggling IRC channel (on Freenode) I was linked to this guide to making your own juggling balls with rice and balloons.

Overall I think my wife and I spent about 30 making 5 balls out of probably 20 balloons, including screw-ups. The end result being the nice rice-filled balls you see below.

I'm looking forward to the eventual disaster that's going to happen when the cat rips one of these open, but until then, I've got some great, weighty juggling balls to continue practicing with!

Pairing with the fourth wall

2012-08-28T00:00:00-07:00

Some time ago I found myself captivated by watching another developer work. Not anybody that I work with or know in person, but the infamous Notch. Throught some twist of fate I stumbled across a live stream of his and spent a non-trivial amount of time watching boxes and code fly by in Eclipse.

Inspired, I figured I would try out the same set up on my own. The first major hurdle to overcome was actually streaming my desktop to Justin.tv. After a full evening of searching, and tinkering with ffmpeg, I managed to get this script working well enough to send audio and video across the internets.

With the "infrastructure" figured out, I started hosting a couple of sessions while I worked on projects that I'm passionate about. I found the live coding concept immediately useful when I managed to get live help from an engineer at Puppet Labs while I dug through the Puppet::AST parsing code.

Taking the experiment further, I even tried helping some folks on IRC with "live support sessions" which were so fruitful that I posted a couple videos (1, 2) from them. In the office, I can say "hey, come over here and look at this" when others need help, and the live coding stream allowed me to extend that concept to practically anybody I wished to help.

I haven't helped anybody in a few months now, I use my typically viewer-less sessions to help me focus on a single project. Whenever I turn the stream on, I'm pair-programming with a companion who may or may not be there (considering the #codingwithrtyler channel on Freenode is generally empty, I'm guessing there's nobody watching), but I focus just as if I had another engineer sitting next to me.

It's almost like a weird hybrid of pair programming and rubber duck debugging now that I think about it.

The projects I work are not nearly as visually pleasing as Minecraft, and I don't have the geek fame that Notch has but I'm perfectly content to stream my desktop to zero viewers as long as it proves useful for evening and weekend hack-sessions.

I'll be speaking at PuppetConf 2012

2012-08-09T00:00:00-07:00

Today the organizers of PuppetConf released an "almost complete" list of speakers and talks for this year's PuppetConf in San Francisco, guess who's on it?

Through some twist of fate, both of my proposed talks were accepted. Originally I had submitted two sessions to improve the chances that I would be able to speak on at least one of the many ideas I had swirling around in my head. I didn't quite expect to have both talks accepted, but here we are.

I've included the titles and abstracts for both my talks below, be sure to register soon so you can also take one of Puppet Labs' beta certification exams while you're in town.

Red, green, re-provision: Test-driving Operations

The developers you work with have a new service they're building, and it is your job to add the appropriate Puppet modules to provision it. Where do you start? A base image, Vagrant and some tinkering? Editing files in "/etc/puppet" on a staging Puppet host? What is the most maintainable work-flow which won't saddle you with technical debt in the future?

In this talk, I will take you from the "outside" inward, writing tests along the way with Cucumber and RSpec. This approach encourages you to think more about where the lines are drawn between the various services and resources you're configuring. While the work-flow is a departure from the "traditional sysadmin" role, the resulting modules can be more easily re-used and re-factored.

We'll do it live: Operations Anti-Patterns

I have done unspeakable things.

The best indicator of industry experience in Ops is an engineer with regrets. An engineer who has designed near-perfect systems but due to outside constraints has had to cut corners. Let's be honest with ourselves and air some dirty laundry.

In this talk, I'll take a jovial romp through some of the most heinous offenders of what could be considered "Ops Anti-Patterns." I'll share maddening stories from companies that I have worked at as well as stories from colleagues, who wish to remain nameless.

Names may be changed to protect the innocent, or guilty, or whatever.

JUC Tokyo 2012 Slides and Recap

2012-08-08T00:00:00-07:00

Just over a week ago I returned from a trip to Tokyo for the Jenkins User Conference 2012 - Tokyo. I attended the summer Developers Summit and a Jenkins hackathon while I was in Tokyo, but JUC 2012 was the main event.

I can't say enough good things about the Japanese Jenkins community, they are as friendly (if not more so) as all of the other hackers, users and testers that I've met throughout the years working with the project. My only regret is that I had to take so much of Kohsuke's time for translation or assistance due to the language barrier.

The talk that I gave at JUC Tokyo was titled "Rebuilding the airplane at 10,000m. Continuous Deployment with Jenkins and Gerrit"

I've uploaded my materials:

Unfortunately I don't yet have the video to post, so for now you'll have to take my word when I say the talk went well.

I look forward to traveling to Tokyo again next year to enjoy the heat, Soba and to try more of whatever delicious food Sotoru and Takashi try to feed me.

Recipes from the 4th Cookathon

2012-07-07T00:00:00-07:00

This past week I hosted a cookout at my place, and cooked a lot of (in my opinion) good food. Most of the recipes that I used were from one of my favorite food blogs: Food Wishes with the always charasmatic Chef John (aka Chef Juan).

I started preparing foods at about noon and continued cooking effectively until around 8 p.m., with help from some of my guests who pitched in as Sous Chefs and Grill Keepers throughout the afternoon.

In order of appearance the dishes I made were:

Home-made Sangria
Home-made Salsa Cruda

This was probably the most time-consuming dish since I diced almost a kilo of grape tomatoes in order to make it. It was very tasty on tortilla chips though.
Balsamic Strawberry Goat Cheese Bruschetta

To me, this was the star of the show. I don't have any pictures of it though! If you click through to the link from the Food Wishes blog, you'll see some photos, mine looked pretty much exactly like Chef John's, and were probably just as tasty.
Turkey Shish Kaburgers

Pictured above on the grill, these turkey burgers are exceptionally flavorful, and surprisingly light. Whereas beef burgers typically leave me feeling a little bloated and greasy, these were filling in a way that allowed us to continue feasting on various other meats and veggies.
Dave's Roasted Potato Salad

Not a recipe from Chef John, one from my good friend Dave. This simple recipe is as follows:
- potatoes (small red or Dutch creamer, baby Yukon Gold)
- green onions
- Some mayonaise and mustard
- olive oil, sea salt
Cube the potatoes, toss with olive oil and salt, roast at 450 until tasty. Let it cool, toss with mayo and mustard and finely chopped green onions. Serve warm!
A fire

Not really a dish, but no cookout in our backyard is complete without a fire, typically accompanied by Smores.

FW: FW: FW: FW: The Gays are Invading!

2012-06-26T00:00:00-07:00

I received a forwarded email which was passed around members of my extended hillbilly family.

Just in case you weren't sure whether or not the US was filled with unadulterated fucking idiots, behold this political action email.

All links from the email are "REDACTED" because these dicknozzles can go fuck themselves.

Dear Pro-family American,

The Radical Homosexuals infiltrating the United States Congress have a plan:

Indoctrinate an entire generation of American children with pro-homosexual propaganda and eliminate traditional values from American society.

Their ultimate dream is to create a new America based on sexual promiscuity in which the values you and I cherish are long forgotten.

I hate to admit it, but if they pass the deceptively named "Student Non-Discrimination Act," (H.R. 998 & S. 555) that's exactly what theyll do.

Better named the "Homosexual Classrooms Act," its chief advocate in Congress is Rep. Jared Polis, himself an open homosexual and radical activist.

And it's dangerously close to becoming the law of the land.

H.R. 998 already has 157 co-sponsors in the House!

And S. 555 already has 37 co-sponsors in the Senate!

That's why I need you to act quickly -- right away -- to protect our nation's youth.

I have prepared the official "Protect Our Children's Innocence" Petition to Congress for you to sign.

Please click here REDACTED to sign it right away so I can rush it to the Capitol with thousands more.

You and I must defeat this disastrous legislation.

You see, the Homosexual Classrooms Act contains a laundry list of anti-family provisions that will:

* Require schools to teach appalling homosexual acts so "homosexual students" don't feel "singled out" during already explicit sex-ed classes;

* Spin impressionable students in a whirlwind of sexual confusion and misinformation, even peer pressure to "experiment" with the homosexual "lifestyle;"

Exempt homosexual students from punishment for propositioning, harassing, or even sexually assaulting their classmates*, as part of their specially-protected right to "freedom of self-expression;"

Force private and even religious schools to teach a pro-homosexual curriculum and purge any reference to religion* if a student claims it creates a "hostile learning environment" for homosexual students.

And that's just the beginning of the Homosexual Lobbys radical agenda.

In fact, it will set them up to ram through their entire perverted vision for a homosexual America.

My friend, I'm sure I dont have to tell you this is not a fight we can afford to lose.

That's why Public Advocate is leading the fight against this immoral legislation.

But I need your help to do it.

You see, it's becoming harder and harder to send my message to unaware pro-Family advocates.

The Radical Homosexual Lobby is working day and night to censor Public Advocate's warning about the Homosexual Classrooms Act.

That's why, along with your signed petition, I need you to seriously consider chipping in a special contribution REDACTED to Public Advocate.

A contribution of just $20 will enable me to email 1,000 potential new supporters.

This is a battle for the survival of American values and the fact is, there's no time to waste.

The Homosexual Classrooms Act will turn America's schools into indoctrination centers and its classrooms into social laboratories -- and they're pulling out all the stops to pass it.

You see, they've disguised the bills wicked purpose behind an innocent name: "The Student Non-Discrimination Act."

The Homosexual Lobby knows that if the public knew the truth about their radical agenda, they'll have no hope of success.

And their dangerously close to ramming their perversity into law.

H.R. 998 already has 157 co-sponsors in the House!

And S. 555 already has 37 co-sponsors in the Senate!

You and I need to take action right now to stop the growing momentum of this disastrous legislation.

I've developed a massive program to launch the second they try to push this bill through -- mail, email, phones, and even radio and TV ads.

But that's only possible with your support...

None of these things are cheap. In fact, running a program of the size necessary to defeat this bill can get quite expensive especially with increases in postage and printing costs.

That's why I need your generous contribution. In addition to your signed "Protect Our Children's Innocence" Petition, will you contribute $250, $100, $50 or even just $35 REDACTED right away.

$250 will allow me to email more than 10,000 potential new supporters. $100 or even $50 would mean 5,000 or 2,500 new chances to alert more Americans to the threat of the Homosexual Classrooms Act.

Every dollar counts in this fight. *Even if you can only chip in $10 or $20 it will mean 500 or 1,000 new emails.

This agenda is nothing new.

In fact, other countries like Britain are already experimenting with this kind of legislation, such as mandating public schools inject pro-homosexual content into every aspect of education.

Word problems in math classes are now to include homosexual characters. History classes will document the "civil rights" struggle against the "oppressive" pro-family establishment.

And it's even started to infiltrate our state governments.

In California, lawmakers want to "require schools to portray lesbians, homosexuals, transsexuals ... as positive role models to children in all public schools."

Sexual deviants being held up as models of virtue?

If that makes you as sick as it makes me, you simply must join me in this battle for America's children.

Please sign the "Protect Our Children's Innocence" Petition to Congress and then send a generous contribution of at least $20 REDACTED right away. Your action will make all the difference.

And if all that wasn't enough to convince you that action must be taken immediately, there's more.

Many say that there will always be private schools and traditional homeschool families to teach traditional values to the next generation.

But the truth is, * this radical agenda is NOT restricted to public schools* .

Kevin Jennings, Obama's "Safe Schools Czar," has clearly stated that "every school, public, private or parochial has an obligation" to teach a pro-homosexual curriculum.

In fact, Jennings denounced school choice programs as "very dangerous" because they make it much harder to impose the Homosexual Agenda on our kids.

"Lord forbid a Baptist or Mormon school," he added.

Jennings' ultimate goal is for all curriculum in "kindergarten, and first grade, and second grade every grade" be infused with a pro-homosexual slant.

Traditional values will be squashed and demonized as old fashioned or out of date, or even as bigotry.

You and I cannot let them succeed.

Please sign the "Protect Our Children's Innocence" Petition REDACTED to Congress I've prepared for you right away.

And along with your petition, would you please send a generous contribution of $250, $100, $50 or even just $35 REDACTED .

The Homosexual Lobby has done their worst to censor my message about the Homosexual Classrooms Act -- and it's absolutely critical that I send more emails.

If you contribute as little as $20, I will be able to email 1,000 new people, maybe even more!

You see, every dollar counts in this fight so even if you can only chip in $10 or $20 REDACTED it will make a significant difference.

Thank you so much for all you have done.

Sincerely,

Eugene Delgaudio President, Public Advocate of the U.S.

P.S. Radical Homosexuals in Congress have a plan to indoctrinate our children in schools -- both public and private.

But Public Advocate is taking a stand. Will you join me?

Please sign the "Protect Our Children's Innocence" Petition REDACTED to Congress I've enclosed.

And please, along with your petition, would you please send a generous contribution REDACTED of $250, $100, $50 or even just $35 right away?

Every $20 will allow me to contact up to a 1,000 potential new pro-Family supporters -- or more!

Because Public Advocate of the U.S. lobbies against the Homosexual Lobby, contributions are not tax deductible for IRS purposes. This email was not produced or e-mailed at taxpayer expense.

This email is never sent unsolicited. It was sent because of your online request to receive ongoing news & offers. We respect and value your time and privacy. We will be happy to remove your address immediately.

Damned gays and their agenda. Derka derka derrrr.

Outside-in for Ops, with code!

2012-06-19T00:00:00-07:00

Last week I discussed some ideas about how to apply some BDD concepts to operations, primarily the "outside-in" approach.

Over the course of the past couple days, I've been experimenting with some ideas with my "PentaGram" project, and I think I've made enough progress to write a follow-up blog post.

It's worth mentioning now that the commits to the repository have a lot of commentary in the commit messages and are meant to help guide you along step-by-step.

I do recommend opening the linked commit messages in a separate window and check them out as you follow along at home.

In order to best emulate a true outside-in workflow, I started with this commit which really just added the apphost.feature file mentioned in the previous post:

Feature: Serve the PentaGram web application
  In order to serve up PentaGram (and satanic phots)
  to millions of devil worshippers

  As an operations engineer

  App hosts should be configured to run the PentaGram
  web application

As you might expect, this by itself is pretty useless so I wrote out the first scenario and committed that too.

I should mention now that between just about every commit, and at every step of the way I was both running Cucumber and RSpec.

In order to realistically test the Puppet code underlying the entirety of the apphost.feature, I needed to spin machines up and down rapidly, so I integrated another one of my projects: Blimpy. Since this was very experimental, the majority of my first few hours of work on PentaGram was consumed making the first couple steps in the scenario work:

Scenario: Provision a fresh app host
  Given an empty host
  And the host is to become a PentaGram app host
  When I provision the host

This commit concluded the work on just spinning machines up and down, and marked the turning point. After this commit I could dig deeper into actually writing rspec-puppet tests and real Puppet code.

At this point I was ready to start driving the code necessary make the "When I provision the host" step run correctly. This required me to do a couple things:

Define a "pentagram-web" module
Set up the whole module structure inside of the pentagram repo

Unfortunately as you can see here getting rspec-puppet up and running properly can be a little bit of work. I found out as I started to look at pulling in third party modules to manage things such as Apache, that I needed to create symlinks for them inside my "pentagram-web" module in order for rspec-puppet to find them properly.

With the dependencies properly being loaded, I could pull in the Ruby, Apache and Passenger modules with these commits: apache, ruby, passenger.

With these dependencies being pulled into my "pentagram-web" module, I felt ready to step back out into Cucumber land.

What I found out when I ran the Cucumber scenario again, and actually provisioned the machine, was that the Passenger module required some more dependencies to be installed! Sure, I could have found this out by actually reading the full documentation on the Passenger module, but who has time for that these days? I addressed this in this commit, which was a good reminder that it doesn't matter how good your RSpec is, there's nothing quite like actually running your code.

Once everything was provisioning "mostly" properly, I was left to jump back "out" to Cucumber and implement the last two step definitions:

Then it should be running a web server
And it should be responding to web requests

I thought about how best to implement this, and decided on the most simple solution (in my opinion): ssh into the box and look! The reason I went for this approach was to keep things simple, I could have made web requests to the machine, but I wanted to focus on the bare minimum needed to pass the test. For the first step I executed "pgrep httpd" and made sure it returned a zero exit code. For the second step, I invoked curl against localhost and made sure that it could connect properly.

What I found out when I implemented these two steps was that...I actually was failing my Cucumber! In another great example of there being no substitute for running the code, I discovered an unspecified dependency in the Puppet resource graph between the Passenger module and Apache, which was fixed in this commit. Even though Puppet was provisioning the host correctly, it was failing to correctly start Apache, and thus failed the scenario!

Lessons Learned

The most important lesson I learned in this exercise was that you must execute your Puppet against a real machine before going to production. Correctly parsing Puppet, passing puppet-lint and rspec-puppet checks is not enough, it must execute on a machine or else you're really just guessing that it will work in production.

A couple weeks ago when I wrote about Puppetizing OpenGrok I mentioned using "Tinker Driven Development." At the time I really didn't know anything about OpenGrok or how I would manage it with Puppet. I am more convinced now that if I had taken an "outside-in" appraoch with the puppet-opengrok module, I would have been able to actually use Behavior/Test Driven Development. Even though I really didn't know anything about how OpenGrok worked, I knew what behavior I wanted from the system.

Provisioning isn't the end. In my previous post, I talked about "Operations as a stake-holder" which for this level of "outside-in" I think is fine, but in the end, Operations is a part of a greater organization. Even though the code implemented thus far provisons an app host which has effectively everything it needs to run PentaGram, it's not actually running PentaGram. From the "business perspective" I still haven't done anything! There's no PentaGram app running, there's no deployment happening, we're still not satisfying our users' need to post daemonic photos! I'm sure this can be fleshed out in additional scenarios that could be addressed after I met the needs of the Operations group.

It's clearer to me now where outside-in for Operations fits in the overall scheme of things. I can very easily imagine sitting down with an Operations Manager and discussing what needs to be done to "make PentaGram production ready" and in doing so, writing out the features and scenarios mentioned in the previous post.

There's still some tooling work that I think needs to be done to make this easier, but I'm optimistic about the future of behavior-driven operations!

SSH as a Hidden Service with Tor

2012-06-13T00:00:00-07:00

For quite some time I've been using Tor for a number of things, but my most recent use revolves around a lesser-known feature Tor provides: Hidden Services.

With fewer and fewer IPv4 addresses running around, especially among residential consumer lines, it has become increasingly difficult to "play a part" in the internet without gnarly port forwarding hacks combined with dynamic DNS.

Hidden Services offer a way to bypass a lot of those hacks altogether, but it comes at a cost of some latency. Basically you need to connect both ends of the connection, the SSH server and SSH client, to the Tor network and let it handle discovery and routing for you.

Set up a Hidden Service by following these instructions
Store your .onion hostname on your "client" computer
Run Tor on your "client" computer

Add the following entry to you ~/.ssh/config:

 Host *.onion
   ProxyCommand /usr/bin/nc -xlocalhost:9050 -X5 %h %p

Run SSH to your .onion hostname: ssh username@shallots.onion
Profit? Probably not, but who cares, you can now access all of your machines on any network from anywhere around the globe!

It's worth noting again that latency will be sticking point, so I probably wouldn't use this for developing in a remote tmux session, but I would use it for using SFTP to transfer files to and from the server (for example).

Addtionally, with the instructions above it's trivial to set up hidden web servers, hidden jabber servers, etc.

Use responsibly!

Outside-in for Operations

2012-06-10T00:00:00-07:00

Update: I've posted the update to this post here which contains some code and more fully fleshed out ideas

I have been thinking a lot about "outside-in" developement for operations as of late, primarily focusing on how Cucumber might fit into the equation.

cucumber-puppet reached its "end-of-life" recently as the developer can no longer provide the tender love and code (TLC) that the project requires. I don't particularly like the way cucumber-puppet works but at the same time, I've been faced with more and more questions about how to bridge into, or out of, rspec-puppet.

As a development tool rspec-puppet is quite handy, but at the end of the day, Puppet manifests should be run in an actual Puppet environment on real or virtualized hardware before they get shipped off to the production site.

What does that look like?

Before you answer that, I think it is important to answer "who are the stake-holders?" for the work that operations is doing?

Developers as Stake Holders

Let's say I'm developing an application called "PentaGram", this is a simple Rails application that will help Satan worshippers upload evil photos and share them with their fellow demonic friends.

To host PentaGram, I'm going to need a web server, an Oracle database (this application is evil, after all) and memcached. I think for now I'll just break up my Cucumber .feature files into machine types (web, db, cache).

For this post, i'll focus on featurees/apphost.feature:

Feature: Serve the PentaGram web application
  In order to serve up the PentaGram to millions of devil worshippers
  As a developer
  Web hosts should be configured to run the application

I think this makes sense as an 'introduction' to the feature, I need web hosts to serve my web application.

Feature: Serve the PentaGram web application
  ...

  Scenario: Provision a fresh web host
    Given an empty host
    And the host is of type "pentagram-app"
    When I provision the host
    Then it should be running a web server
    And it should be responding to web requests

Alright, I'm not supremeley thrilled with this approach, one such example is that "pentagram-app" is a Puppet module, and I'm not very comfortable with that leaky of an abstraction. So I'm going to reword it to: And the host is a PentaGram app host and start to define a vocabulary between my stake-holders instead of letting Puppet module names leak upwards into Cucumber.

The more I think about it, I'm not sure "developers" or even product-owners are the stake holders to address with these features. As a developer, I care very little about the operational details, so long as PentaGram works!

Operations is its own compelling stake-holder, but that presents a complication of making "outside-in" features such as these too leaky in terms of abstractions and not very descriptive.

Operations as Stake Holders

That web host you provisioned above is nice and fancy and all that, but as far as I am concerned (being a fellow Ops engineer), there's a lot more to an app host than just running the web server.

Feature: Serve the PentaGram web application
  ...

  Scenario: Add app hosts into the load balancer
    Given a load balancer with a pool for app hosts
    And an empty PentaGram app host
    When I provision the host
    Then it should be added to the load balancer
    And it should be receiving requests from the load balancer

I'm reasonably pleased with this, I'm going to pretend writing all these step definitions is going to be rather easy to write, and forge ahead writing out more Scenarios.

  Scenario: Create the appropriate firewall rules
    Given an empty PengtaGram app host
    When I provision the host
    Then traffic should be allowed to port 80
    And traffic should be allowed to port 443

  Scenario: Nagios checks for app hosts
    Given a Nagios server
    And an empty PentaGram app host
    When I provision the host
    Then Nagios HTTP checks should be added for the host
    And Nagios HTTPs checks should be added for the host

At this point I've not written any step definitions or Ruby code to run these Cucumber features, but I think this enumerates what I want out of a PentaGram app host.

I know the following things are needed for an app host, just by reading apphost.feature:

A web server should be running (but how do we know it's not "It Works!" coming from Apache instead of the PentaGram app?)
The app host should be added to the load balancer's app-host pool when it is provisioned
The app host should port 80 and 443 open, ostensibly because Apache is listening on those ports (we didn't say that it should be though! should we?)
When the app host comes online, it should have Nagios checks added for its service running on port 80 and port 443 (again, this doesn't verify the PentaGram app is actually running there).

I am not sure if this is the best approach, or even an approach that doesn't suck, this entire blog post has been me riffing on what outside-in might look like for Operations.

In my next post on the subject, I should have some example code available with fleshed out step definitions and some valid/passing scenarios.

Grok code with Puppet

2012-06-02T00:00:00-07:00

As the Lookout code-base grows, both in individual repositories but also in the sheer number of repos we maintain, I've found it often difficult to find what I'm looking for.

Some of that difficulty is due to ActiveRecord's determination to implement the world through meta-programming, but for everything else I'm turning to OpenGrok.

OpenGrok itself is a Java-based code cross reference and search engine, which works surprisingly well with Ruby. While not perfect, it certainly blew LXR away in both ease-of-use but also cross-referencing of Ruby, Java, and C code.

Below is an example of searching for the AWS class in my Blimpy project.

You'll notice in the search results that there is a green-colored annotation to the right of the code snippet denoting the actual definitions compared to other search results.

When I click on aws.rb and dive into the code itself, I can navigate based on method definitions, but also pull in git-annotate(1) style annotations (not pictured).

Since I'm no longer in the business of hand-crafting machines, I went ahead and created puppet-opengrok. This OpenGrok Puppet module, while lacking in rspec-puppet tests (see @glarizza, I'm a hypocrite), will allow you to stand up a simple OpenGrok instance on an Ubuntu Server of your choosing.

Take the following manifest for example:

node default {
  include opengrok

  opengrok::repo {
    "puppet" :
      git_url => 'git://github.com/puppetlabs/puppet.git';
  }
}

The module will handle installing the packages tomcat6, git-core and a few others to make things possible, but after you've run Puppet you can navigate to: http://yourhostname.lan:8080/source and you'll find OpenGrok has indexed the Puppet code base for you!

The module will also install a cronjob which updates the source trees and indexes every 10 minutes.

Currently puppet-opengrok is quite rough around the edges, since I used TDD (Tinker Driven Development) to build it intead of TDD (Test Driven Development). It will most certainly work on Ubuntu 10.04 LTS, but anywhere else your mileage may vary :)

If you're familliar with Puppet Forge, you can install the rtyler-opengrok module with the puppet-module tool.

Bullshit.

2012-05-30T00:00:00-07:00

"Ruby on Rails is a full-stack web framework optimized for programmer happiness and sustainable productivity. It encourages beautiful code by favoring convention over configuration."

Almost nothing in that description describes any Rails code base I have ever seen.

(Taken from Ruby Gems.org)

Hub board: Call for testers

2012-05-27T00:00:00-07:00

I've been working on Hub board off and on for the past couple months, and I finally think it's ready for some more broad beta testing.

Hub board itself is meant to be a more visual Kan-ban style board built on top of GitHub Issues. There are three states for issues "Open / Assigned", "In Progress" and "Closed." By default Hub board will not show you any issues that are not assigned to you; unfortunately there is not a means to assign existing issues to yourself through Hub board issue #52).

Please visit the intro page to learn more, and file issues on GitHub.

For the curious, Hub board is built with Amber Smalltalk a JavaScript implementation of the Smalltalk language and runtime. It is also built and deployed with the Resin which embeds Amber and couples it with a simple Sinatra web server to make developing Amber applications exceptionally easy.