Highgroove Studios - Blog - Pit-side with Highgroove

Why Highgroove has a Personal Trainer

2012-02-21T00:00:00-05:00

At Highgroove, we have a personal trainer, Cherri, on-staff and on-site, available twice a week to us, scheduled via Appointment Slots using Google Calendar. Our personal trainer has been with us since December of last year, and we just added more sessions. We have been delighted at the opportunity to get in shape (although, perhaps, temporarily less thrilled when "core day" came around). Personally, having someone motivate us to exercise -- someone who thoroughly knows what they are doing was exactly the motivation I needed to start working out again. But we've also realized getting a gym session in during the afternoon has benefits for developing software (along with developing sweet abs).

The sessions are personally catered to each of us, last just thirty intense minutes, and with the gym just downstairs, it's a quick walk over to start working out. It's an even quicker walk when Cherri sends you an email, reminding you that your session starts in 1 minute. With such a short work out session, and scheduled in the middle of the work day (almost all sessions are after lunch), you would think it would cause a break in productivity, but the opposite actually happens.

In fact, Lifehacker has posted about the benefits of a workday workout, explaining that it improves energy and alertness as well as productivity. The article fits my and other Highgroover's experiences perfectly: working out gives an energy boost that used to be gained by an extra double-shot of espresso, and also gives us time to mentally work out the programming problems wracking our brains. We can return to work refreshed (if a little sweatier) and ready to tackle things from a new perspective. It almost seems counter-intuitive, but after a work out, we have all remarked on the productivity increase. It's real!

What healthy habits give you a productivity boost?

Working Hard or Working Efficiently?

2012-02-21T00:00:00-05:00

Companies often feel that they are getting the most out of their employees if everyone is working as hard as possible for as long as possible. However, this can be exhausting on the employees and can lead to diminishing returns. We feel that the better approach is to emphasize efficiency over effort, and our level of productivity speaks for itself.

Working in a ROWE gives us an advantage in working efficiently because we can better handle common productivity pitfalls. Stress rears its ugly head in many projects. In a typical 9-to-5 (or even later) the employee is chained to their chair and must work until the end of the day, or until a moment of escape presents itself. How much work is actually done in those last few hours as stress goes up and morale goes down? Because we have the freedom to work when we want we are able to step away from our projects for a few hours, do something that relaxes us, and try to tackle the problem in a better mindset. Given the break time we may work fewer hours in the day overall, but those fewer hours produced more than the stressed employee. The resulting code is also likely of a higher quality due to the more focused employee making fewer careless mistakes.

We are also able to avoid burnout by taking advantage of productivity bursts. Every developer has been on a programming tear, where they are able to crank out code at a blistering pace and don't want to stop for fear of losing momentum. When they are done they may have completed a day's work in a few hours. What to do next? Whatever you want, as long as you're meeting your client's needs.

Emphasizing efficiency over effort leads to happier employees, fewer errors, a higher quality product, and happy clients. Everyone wins. What other strategies can you use to maximize efficiency?

Another Reason To Use Github

2012-02-20T00:00:00-05:00

While getting to know the very sweet Github Api v3, I ran into a little bug when trying to page though results coming back from the service. While it was frustrating at the time, Github's support and resolution blew me out of the water.

For a new in-house project, we would like to be able to grab the commit messages for a number of our repositories that live in Github. Using the octokit gem with the auto_traversal option turned on, this is trivially easy. Because of a bug in the Github api, when the gem tried to page though the results, the link it got for the next page was always the same. This caused an infinite loops of http requests, and me to try to pull all of my hair out.

Once I threw a few pry bindings into the project and was able to track the issue, I was able to easily hunt down Github's support for issues with the API (it was right on the docs page for the API). All that was required to get a fix deployed for the issue was a quick email with steps to reproduce the bugs and less than 24 hours of patience!

The fact that the Github crew was able to submit a fix to their live service before I could submit a pull request to octokit with a workaround reminds me why they stand out in my mind as a great web app. Not only is their site feature rich and easy to use, but they've got a great team backing the technology up. That's why we at Highgroove trust them with our code and why we recommend them to clients.

Live from StartupRiot MAKE

2012-02-18T00:00:00-05:00

The StartupRiot MAKE event kicked off last night at ATDC with the help of Sanjay Parekh and some very awesome sponsors and teams.

We're happy to be a part of this awesome event. At 5 pm on Friday, teams started trickling in and sponsors started setting up displays. The food arrived and many people began networking and planning already.

At 6 pm, Sanjay announced the teams, and each team got up and spoke about what they were building, and what they needed -- be it a designer, some mobile development help, or back-end development.

There are some awesome prizes, but I am pretty sure the teams that are here, while they are happy to win, are actually here because they enjoy making and building things using technology. You can actually tell, because in addition to building products (we've already seen a few working demos), they're helping each other, offering encouragement, and feedback.

A few of us from Highgroove are here this weekend offering help in the form of knowledge and programming chops. Since we've built products and services for companies both big and soon-to-be-big, we're pretty opinionated about what we've seen that works and doesn't work in a modern web application. Our actual real service, though, that we're offering is something that comes natural to us: "Just Keeping it Simple" and saying "No" to:

complicated features (keep it simple!)
overly generic solutions (hyper-specialize instead)
building it yourself (instead of leveraging open source)
scaling too soon (start small, scale later)

Good luck teams!

Tech Talk on Visualizing Scaling

2012-02-17T00:00:00-05:00

This week's Tech Talk was on Visualizing Scaling, and Consistent Hashing techniques for scaling data.

Patrick talked through several visualizations he created in HTML5 to visually explain many common scaling techniques.

If you've heard of the techniques and algorithms: "sharding" or "master-slave", or even "consistent hashing" for distributing data across servers, but never really known how they work, this talk goes through the basics and shows visually how data can be scaled out across multiple servers.

"I never really understood some of these techniques until I saw them visually", says Patrick, who has implemented many of these techniques for Highgroove's many clients. He details the pros and cons and even how and when to start considering these techniques for scaling.

Check out the animations here on Github:

https://github.com/vanstee/distributed-data

And, be sure to check out the video on Vimeo:

Consistent Hashing Tech Talk

Interested in attending (or even presenting) at our weekly Tech Talks? Check out our upcoming Highgroove Events on Facebook.

If the Shoe Fits, Ship it!

2012-02-17T00:00:00-05:00

In January of 2012, after spending the holidays 'funemployed' (OK, just on vacation) down in sunny Florida logging some EPIC miles, I very excitedly boarded the Highgroove ROWE-boat as our first Account Director.

The month of January left me dazed but not confused. Working in a ROWE environment means results are expected immediately (and that results are all that matter). I hit the ground running and quickly found that everyone here is equally vested each project's success. That is to say, pardon the pun: "the rising tide lifts all boats" here at Highgroove. By leveraging constant communication, the best tools, pair programming, code audits, and project retrospectives -- we truly are an agile shop.

Read on to find out how I learned about what we do best, what makes us different, and what that means to me as our Account Director.

After only having been on the job not even two months, I am still new but no longer the new guy. Reading Andy's Post 2 Months in at Highgroove, I feel humbled to be listed alongside some pretty amazing colleagues here. Similar to Will's First Reactions at Highgroove, I've been good friends with several Highgroovers for years. Plus, I knew Highgroove worked on amazing projects with even more amazing clients.

What we do Best

We know that our clients benefit from constant communication with developers. Small issues are consistently caught before they become large problems. Our collaborative environment is filled with brilliant people producing refined, simple solutions in Ruby and Ruby on Rails to complicated and challenging backend software application projects. Not to mention, those brilliant people don’t have to freak out if they're running "late" into the office because of the worst traffic in the country. As Daniel points out: There is no late to the office, since work can be done (and is done) when and where it makes sense. Highgroove uses collaborative, agile tools because they lead to extremely high quality code at every stage of a project. My long time friend and resident Methodologist, Chris Kelly, emphasized this in a great post on iterations last year. Let's review:

We're Project-Manager-less. We offer clients direct Contact with developers who actually do the work. No watered down communication or confusion.
We're hyper-specialists. We're experts in backend web development in Ruby and Ruby on Rails. We know what we're good at, and stick to it.
We're ROWE. We don't care when or where work happens, all that matters is that the work gets done.
We're Agile. Highgroove walks the walk and follows all 12 Principles on ALL projects.

What makes us Different

Ok, those are neat characteristics but what REALLY makes us different? Our friends at Matchstic helped us uncover our Core Values (Attributes) at their Brand Camp. Hear it straight from the horse's mouth, as CBQ shares the camp's impact on our business.

Our Core Values are:

Personable
Optimistic
Trusting
Craftsmanship

Knowing our core values helps us not only know what shoes fit in the client department, but also how to hire, how to communicate and what firms we should partner with. I have found (like all Highgroovers) that I identified with each attribute, which is how I knew I was at the right place.

What that Means to me as our Account Director

Now back to the 'shipping'. Since we're an agile shop, one of my favorite parts about working at Highgroove is our 'Bias Towards Action'. We just Ship it! and then we iterate and ship it again until it's the best it can be. As Account Director this all means one thing: The value we offer resides in our proven process and unique brand. Our office manager Megan underscored the importance of this this process at an Agile Workshop she attended, because it supports client needs at the highest level, encourages focused development, and fundamentally promotes efficiency. And there's where the rubber meets the road and why being in the 'Highgroove' truly sets us apart. We know what shoe fits whether it means:

the right client fit for a project,
the right tools and
the right team

I am now a bona-fide member of the just 'Ship It' club, in my both my personal and work life. At Highgroove we know what fits best, we know what sets us apart, and we just 'ship it!'

Have you ever thought about what makes your company really unique?

You Must Teach To Be Taught

2012-02-14T00:00:00-05:00

Waaaayyy back in December, I had the pleasure of attending a code retreat. In that post, I discussed what I learned.

This month, I had the pleasure of facilitating a code retreat a few weeks ago. Thanks to Highgroove, TapJoy, FourAthens, and my co-coordinator Travis Douce, the Athens Code Retreat was a resounding success.

Also, a special shout out to our Code Retreat homies in South Africa led by Corey Haines, who handed off the baton to us late in their day but early in ours.

Read on to find out how lessons learned from facilitating compares to attending, how the general "You" actually means "I" in the blog title, and how many times it takes (me) to learn the four rules of simple design.

The purpose of a Code Retreat is to get better, to practice one's craft in a setting without the pressure of "getting it done, yesterday(tm)." Throughout the Code Retreat, we emphasize the 4 rules of simple design. Another way to paraphrase them:

Tests pass.
Good naming.
No duplication / Don't repeat yourself (DRY).
Small.

To facilitate self-improvement, we dedicate the last session of the day to the closing circle. In the closing circle, three questions are asked of every participant.

What did you learn?
What most surprised you?
What are you going to do differently going forward?

In my original blog post), I boldly claimed that one could code the Game of Life in 45 minutes if and only if they practice four or five times and skimp on tests. Oh, how wrong I was!

One pair, which included Andy, managed to write a completely functional version in 30 minutes. They even had time to invent their own spaceship. How is this even possible? Other groups took the fast and loose approach. "We know the problem domain. We have the technology. Tests? Where we're going, we don't need no stinkin' tests!"

What was the first pair's secret sauce? TESTS! GOOD NAMES! DRY! SMALL METHODS! SMALL CODE BASE! Amazing!

I learned magnitudes more facilitating than I did attending. I saw novel approaches that impressed and stretched my understanding of the problem domain. I saw good coding practices win out over the quick and dirty style again and again.

What have you learned when teaching that you didn't learn as the student?

Mailcatcher: Making email testing a breeze

2012-02-09T00:00:00-05:00

One of my least favorite chores as a developer is dealing with email. I’m not talking about my inbox. That is a post for another day ;). I’m talking about emails sent by web applications. Whether it is a sign up confirmation email, a receipt from a purchase, or reminder for your dog’s birthday. Chances are, if you have a web application, it sends email.

Traditionally, my workflow for testing these emails has not been very elegant or even efficient. It would either involve creating a bunch of users with different emails accounts I own, or telling the back-end to send all emails to my email address. While both of these work to some extent, the former is very time consuming and the later isn’t really testing the system the way it is meant to be used.

Mailcatcher one-ups both of these methods big time. Mailcatcher provides you with a local SMTP server for you to send your emails to in your development environment. Mailcatcher also provides you with a webmail interface to view all the emails your system has sent.

Mailcatcher is written in ruby and is available as a gem. The gem's site offers detailed instructions on how to use Mailcatcher with a Rails app or even a PHP app. Without going into the details here, after you have installed Mailcatcher, you basically just set whatever is sending mail to use Mailcatcher's SMTP server. Then any emails sent by your site will be collected in Mailcatcher's webmail interface. Check it out:

Mailcatcher has really eased the pain of dealing with emails in web application development. What are some tools you have found or made that have made development more fun for you?

Red, Green, Refactor - The Tools For Success

2012-02-07T00:00:00-05:00

It's easy to say "We're agile" and "We use Behavior/Test Driven Development" and thus "we use the right tools to empower our developers!" but what are those tools? For me that discussion is entirely about the tool stack you choose, how that stack empowers you as a developer to do things right the first time. Luckily thanks to the ruby community as a whole we have a large number of high-quality choice to choose between.

Generally when we talk about TDD and being agile, beyond the process of choosing features to deliver the maximum customer facing value, we refer back to the simplest possible workflow of "Red/Green/Refactor." Within that workflow you write tests that fail, make them pass (with the focus on getting that done without polishing), then refactoring, to polish where necessary. The subject of testing is a huge one with a large body of prose already written on the subject. Rails Test Prescriptions and The RSpec Book are fine examples of such, along with a huge menagerie of blog posts, podcasts, and screencasts to choose from.

I propose a slightly different day-to-day workflow that I see used to great success.

Hack around and choose your course
Red - write some tests that fail but will pass when your code meets your needs
Green - the code is written, the tests pass, life is good.
Refactor - Don't Repeat Yourself (DRY your code out). Or don't, if Ya Ain't Gonna Need It (YAGNI).
Continuous Integration
PANIC OVER SECURITY
Ride Bikes

This isn't the "pure" TDD process, but one that just adds a couple of unspoken facets of the process that are already used widely, just don't fall into the pithy "Red/Green/Refactor" process.

Ruby has a functional programming feeling with its composition of methods, distinction of methods with side-effects from those that do not (via the ! suffix), and it comes with some variety of REPL via IRB (or the new awesomeness in Pry). This allows us as developers to spend some time touching code and experimenting before we decide exactly how we might like a function to operate, and thus before getting into the main development workflow for a given feature.

Tools to support step 0:

Irb - you already know about this but it's easy to overlook.
Pry - the new slickness that is 1-part Irb replacement, 1-part debugger, and a pinch (or three) of awesome. Watch the screencast to get a feel for it!
good-ole-pencil-and-paper, a whiteboard, some origami paper
whatever tool gives you a feeling for the right way to proceed without being trapped in analysis-paralysis.

Once you're ready to write some tests that will fail, you likely already have a test framework of choice, and even a workflow for running those tests. You may run them from within your editor, have a terminal window ready, willing, and able to run them with a single command (and a shortcut into your history like ctrl-R), or you may let something else automagically run them for you and report back.

At Highgroove many of us find a combination of running tests manually and automatic test running and reporting to be most useful. To accomplish this I use the testing framework selected for the project along with Spork and Guard. Spork is a test server that constantly keeps your rails code loaded and ready to be tested rather than loading each time in a rake task. Spork should be your first step to speeding up slow-running test-suites. Guard is Spork's right hand man. It watches code for changes on disk and automagically runs the correct associated tests. Guard will even notify you via Growl, Libnotify, or your other notification framework of choice that you've broken things, or that everything is still hunky-dorey and you should keep hacking away and/or ship!

When I run tests manually I often want to run a single test, or set of tests manually. Using Rspec2 you can accomplish this with tags or calling the specific spec by line number via bundle exec rspec <path to spec>:<line no. of spec to run>.

Testing is not only a matter of making your tests all turn green. Poorly written tests can certainly be green and still not indicate that your code is working as intended. Perhaps you simply forgot to include any assertions (I've certainly never done that) or you're testing your factories rather than your models (never ever once again). Code quality tools can help you with that as well, and it's not entirely certain as to whether they fit in the "Red/Green" phase of development, or in the "Refactor" phase. I like to split the playing field and say that code-coverage tools belong in the red/green phase as they allow you to check the quality of your tests which firmly live in "red/green" while other code quality tools (such as static analysis tools for cyclomatic complexity) belong in the "Refactor" phase. We've discussed code-coverage in Ruby before so have a look there for a refresher, but if you're not using one of these tools it may be time to roll it into your workflow. Cover-me even has a nice post-test HTML coverage report to show you how well you've done. This allows a very tight development loop to make sure the tests are covering the right things the first time.

Tools to support steps 1 and 2:

Testing frameworks: MiniTest, Test::Unit, Rspec, Cucumber, etc.
Spork and Guard
Coverage tools: cover_me, SimpleCov, rcov (for Ruby 1.8 apps)

So you've got good tests, good test coverage, everything is green, and you're ready to refactor. If you are struggling to start your refactoring process then either it's good enough as it is or maybe you need some inspiration. Sometimes your code just doesn't need to be refactored. It is easy to fall into the premature optimization problem and abstract/refactor/clean your way into a corner that you will undoubtedly undo later when the customer asks for more or different flexibility. So maybe just delay refactoring if there isn't an obvious need. In the case of needing some hints as to where to refactor the Rails Best Practices site has you covered.

There is no replacement for a good set of eyes on the code and intuition as to where you can refactor. To this end Highgroove's internal code reviews go a long way to getting our code refactored quickly and elegantly.

Since we are not constantly reviewing or pairing there are some tools to help us along including the excellent Flog and Flay, a pair of tools to find complex code and duplicate code, along with an interestingly macabre theme. Another tool is Excellent which produces warnings about "smelly code" via static analysis. Finally we have the big kahuna: Rails Code QA. This package wraps up other code quality packages (including Flog and Flay) to make a single rake task that does it all.

Tools to support step 3:

At this point you have your local code automatically being tested, you've run the tests you need to focus on, you've refactored with the help of tools, the rest of your development team, and the community's collective knowledge. If you are working on a feature branch, that branch needs to be merged into a staging or production branch, and hopefully tested before being pushed to client or customer facing environments. This is where Continuous Integration (CI) comes in.

A CI environment is constantly running all of your tests on whatever branches it needs to in order to make sure that any merge artifacts don't make their way into production. You may choose to run your own CI environment via Jenkins or Goldberg, both of which are lightweight and well supported, or you may choose to "outsource" your CI to a service. Travis-CI has gotten a large following in the Ruby community thanks to its super simple setup, integration with Github, and quality service. I was recently introduced to Tddium by a co-worker while struggling to setup an internal CI system for a customer. It looks very well backed and supports pretty much all of the services we employ day to day. If you don't want to do operations to make sure your CI is up and running then looking to one of the latter services is a great route.

Tools to support step 4:

Local CI tools: Jenkins, Goldberg
Hosted services: Travis-CI, Tddium

Great! Your code is tested locally and in CI, everything looks solid in terms of functionality, and Rails automagically does everything right for security. It's time to go ride bikes right? Well maybe not. There is a comprehensive and excellent Rails Security Guide because it's not so simple. This isn't to say that Rails security is as hard as it could be, but as with everything getting it right requires knowing and following the best practices created by the community.

Luckily for us the Brakeman Scanner project integrates beautifully with the tight testing workflow described above. Brakeman is a static-analysis based security-focussed scanner aimed at Rails applications. It is under active development and the progress has been phenomenal. We did a brief tech-talk about it recently and to say we are enamoured of it may be an understatement. Even a couple of the bugs described in the tech-talk have been fixed since then! Combining Brakeman's fantastic scanner and our automated testing tools works out of the box in two ways. Firstly there's Brakeman's reporting which is excellent and could certainly be wrapped in a rake task to create a cover_me style report after each test run. Secondly there is guard-brakeman for local-scanning and testing as well as notifications via your system notifier when new security issues are introduced, just as if a test failed. For CI environments Brakeman works with Jenkins out of the box and adding support to other CI environments should be a fairly simple operation.

Tools to support step 5:

Brakeman, Guard-Brakeman

Making your development and testing loop as tight as possible allows you to have confidence that your code has been written in a process that promotes testing, coverage, and automated quality tests so that you can focus on what's important: shipping code and riding bikes.

Tools to support step 6:

Kumbaya, My Award

2012-01-31T00:00:00-05:00

At Highgroove, we love giving each other compliments. In fact, since everyone at Highgroove kicks ass in some way, compliments are constantly flying around (actually, I think that in and of itself was a compliment).

Well-deserved compliments foster teamwork, increase morale, and make us better as a team than we could be on our own.

One specific way we compliment one another is by giving the Highgroove Award. The award can be given by anyone to anyone; the recipient is recognized on the website and with a physical trophy (it's a bit over the top on purpose!).

Read on for how we added a technical twist to giving the award.

Highgroove specializes in backend web application development, and while some of the backends we develop actually drive mobile applications on iPhone, iPad, Android, and other mobile devices, we don't generally develop the frontends of these mobile applications in-house.

Even so, a few of us are curious and wanted to learn more about how to develop all aspects of mobile apps. It helps us understand how to build better backends.

The Highgroove Award iPhone App was born as a recent weekend hack project.... And refined on a recent hack night.

The app allows any Highgroover to give the award via his or her iPhone. A recipient is selected and the reasoning typed in; the backend portion of the app redeploys the website with the new awardee!

Now anyone can give the Highgroove Award ... even when we are on-the-go.

The Highgroove Award is a bit silly, but it helps us give each other serious praise in a light-hearted way. How does your workplace encourage compliments?

Aside: For a more detailed explanation of what goes into the backend and frontend of a mobile application, check out Dear business people, an iOS app actually takes a lot of work!.

Great Office Rearrangement: Sit where you want

2012-01-27T00:00:00-05:00

At Highgroove we are are always trying out new ways to improve our process and environment. One of my favorite experiments has been doing away with assigned seats. Our Results Only Work Environment allows each person to decide when and where they do their work. While it is true no one is required to come into the office, the reality is many people prefer to be in the office. It is not hard to see why. Every member of the team gets a massive monitor, a super comfy Aeron chair, and all the espresso and snacks anyone could ever need. Although most people come into the office regularly, each team member's hours can vary wildly. When we had assigned seats you could come in the office and be isolated just because your neighbors on a different schedule. Conversely, you could come in and be surrounded by a couple developers talking out a difficult problem when you really need to get something else done. In short, assigned seats just aren't very ROWE.

The Setup

The fact that we hyper-specialize in back-end Ruby development means that everyone uses the same equipment and has basically the same needs. While this made doing away with assigned seats much easier than it would have been for a firm where there are many different types of specialized workers, we still had some issues to work out. First we had to deal with everyone's "desk stuff". If you have worked at a place with assigned seats you have probably noticed that you and your co-workers tend to "nest", building up a collection of trinkets, tools, and other things that may or may not be needed to get the job done. The amount of "desk stuff" has always been limited at Highgroove by small desks. Our work is done in the cloud using tools like Google Apps, Pivotal Tracker and Github. We generally don't need desk space for much more than a keyboard, mouse, snacks and maybe a latte. If the desk is holding much else, chances are that stuff is in the wrong place. If the "desk stuff" is information you or your team members need and exist nowhere else, you are working in a silo. That is bad. If the "desk stuff" is something not related your current tasks, it is clutter: a distraction at best, it should be elsewhere. Where do we put all this stuff? We have a cubby section where each developer can safely store their stuff like bike clothes, business cards, notebooks, and other things we use but don't need constantly to do our work. We also use cubbies to store our keyboards and mice when we leave, so that others can use the vacant desks.

Cubbies were about the only addition we needed to make this work. We provide everyone on the team with the best equipment available, so there is no inequality; no good desk or chairs to fight over. If you were implement this policy in an office where there were several different types of desks and chairs you would want to make sure there are enough setups of each type to make everyone happy.

The Results

We've been running this experiment for several months now. The no assigned seats policy allows everyone the chance to switch things up and get to know each other better. The policy also allows team members to have more control over their environment. When there are a lot of people in the office, desk selection can be slim, but so far no one has been without a desk, or has vocally complained about having to sit at a desk they don't want to. Some people still sit in the same desk everyday. Other people sit at a different desk every time they come into the office. The point of ROWE is that we are all responsible adults, perfectly capable of making our own decisions. We decide when it is time to work and time to play. The next logical step was to remove assigned seating for those who decide to work in our office. Doing so has given each member of our team the more control over their work environment, and has also made for a more open and collaborative workplace. What steps have your team taken to make your office more awesome for everyone?

Tech Talk on Brakeman

2012-01-27T00:00:00-05:00

Dave's Tech Talk this week is on security on web applications, focusing on Ruby on Rails applications, and using the static analysis security scanner called Brakeman (brakeman on github).

In this talk, Dave looks at how static security analyzers work, and how we used it to find some very tiny (already fixed within a few minutes of finding) possible security weak-points in an application we built for a client.

Static analysis looks at your source code and looks for known vulnerabilities. As opposed to dynamic scanners, or intrusion "detection" scanners that scan a live site from the outside -- these are run on the code itself.

In Ruby this is hard because it's a dynamic language. Because of this, Brakeman (and any static analysis scanners) will identify "false positives" (code that isn't necessarily a vulnerability) but they're easy to track down, and safely ignore.

Dave notes, that because of Ruby's dynamic-ness, if you are doing a lot of method_missing meta-programming, it won't be able to analyze your code. The practices and places it does identify are easy to see, and you can apply Brakeman's other recommendations in your meta-programming code, to secure your web application.

Check out the talk on Vimeo: Tech Talk on Brakeman

Image Credit: fishywang on Flickr

Hack Night at Highgroove

2012-01-26T00:00:00-05:00

Highgroove hosted our monthly Hack Night, and with 20 attendees, this was our largest yet.

To program a computer in a clever, virtuosic, and wizardly manner. Ordinary computer jockeys merely write programs; hacking is the domain of digital poets. Hacking is a subtle and arguably mystical art, equal parts wit and technical ability, that is rarely appreciated by non-hackers. See hacker. -- Urban Dictionary: hack

Hack Nights are a chance for Highgroovers to simply "hack" which means: to experiment, learn, and play with technologies we might not get to during our day-to-day. So, what did we hack on?

the io language using Day 1 of the io chapter in the Seven Languages in Seven Weeks book
game programming (the wolfbrain) on the Sifteo Game Cubes in C#
a remote music playing server with voting and control from our chat room for the office
adding functionality to some of our favorite ruby gems and libraries including brakeman, and a yet-unreleased crazy-math backend algorithm solver
an iOS app for awarding Highgroovers the coveted Highgroove Award
a search-across-the-company app using some new search technologies powered by Ruby
and much, much more....

Thanks to all who came out. Stay tuned for our next Hack Night in February by liking us on Facebook and Twitter.

Maintaining Clients (keeping everyone happy)

2012-01-24T00:00:00-05:00

Far too often there exists a chasm between the client paying for and the developer working on a particular project. One party has a longterm vision; a vision that will ultimately impact his/her financial future. The other party has a much closer view of the project, which creates the risk of getting tunnel visioned. The client worries about when the project will be completed; the developer worries about how the project will be completed. When this happens, communication becomes difficult and frustration builds. How can this be avoided?

At Highgroove, we know that there is no replacement for good communication. This perhaps requires a bit more work from our clients than they might initially anticipate, but by maintaining frequent contact, we prevent frustration and confusion from creeping into the process. By keeping our clients actively involved (i.e. forcing them to accept/reject stories in Pivotal Tracker), they know exactly where the project stands and what work remains. This helps the project stay on track and gives us, the developers, quick feedback about whether or not the work being done is acceptable.

When we are able to maintain this level of communication with our clients, we can develop a comfortable working relationship that allows us to honestly question features or approaches that would complicate projects and potentially delay launching. At Highgroove, we earnestly want our clients' projects to succeed. We hope that our clients choose us not for only our technical skill, but our dedication to giving them the best version of their vision. By striving to cultivate a successful working relationship, we go beyond maintaining happy clients: we promote successful enterprises.

Tech Talk on Riak

2012-01-20T00:00:00-05:00

Will's Tech Talk this week is on Riak. "Riak is an open source, highly scalable, fault-tolerant distributed database." -- Riak Overview on Basho.com.

The main use-case is to use in web applications that have heavy requirements for data, where lots of data must be "written" or "read" in a distributed fashion, but needs to be high-availability.

Will also went over one of the main Ruby libraries named ripple that is a rich Ruby modeling layer for Riak, Basho's distributed database that contains an ActiveModel-based document abstraction.

Riak works great when your database gets painful. If you have an application that just has tons of writes, or tons of reads happening, and the database is just dragging, Riak is a great choice for storing and reading data quickly. Adding Riak along-side your database to alleviate performance is a great way to scale out and make your web application more performant.

Check out the video on our Tech Talks page: http://vimeo.com/highgroove/videos

Git Treeishes Considered Awesome

2012-01-19T00:00:00-05:00

As developers, we love having quality tools like git, available to us.

Using git, we can easily navigate through all the code ever "committed" or added to a project throughout its history. We do this via git treeishes. Treeishes are git's way of referencing commits and relations between commits. Treeishes can improve your workflow immensely if you're a frequent git user.

In this post, we'll cover some of the more basic treeishes and then work into the advanced ones, with some real-world examples.

The Basics

You'll probably be familiar with the types of treeishes in this first section if you've ever used git before. The following section will cover some more advanced uses of treeishes that even long-time git users may not know about.

SHAs

Every commit in git is identified by its SHA — a unique hash. These hashes are pretty long (40 characters in fact), but git also allows you to reference any commit with any truncated version of its hash so long as the truncated portion is unique (generally, this means at least 5 characters long).

Branches, Remotes, and Tags

Branches, remotes, and tags are another kind of treeish. Each of these is actually just a pointer to a git commit.

Advanced Treeishes

Now we're getting to the really good stuff!

It's important to have a good understand of the git reference log (reflog for short) which enables most of the more advanced treeishes.

The git reflog is a running log of recent changes to tips of branches. In practice, this means that every time you commit to a branch, pull commits down from a remote repository, or checkout a new branch, your reflog will update. It's important to recall that reflogs are specific to individual checkouts of a repository so the reflogs for the same project can differ across machines.

Here's an example reflog from one of my personal projects:

This reflog shows that some of the more recent changes to this project include:

Cloning the project from the remote repository
Making a few commits
Checking out a new branch from the tree at 0a4faaa (which just happens to be master)

Using the reflog, git can infer some interesting relative context for commits that will allow us some more flexibility in what we can get at with treeishes.

Date Specs

We use the date spec here at Highgroove as part of our weekly code reviews. With the date spec, I can easily look at the activity on a feature branch over the course of the last week:

Tilde Spec

Similar to the ordinal spec, the tilde spec allows you to reference the Nth grandparent of a commit.

This is useful just to go back 2 or more commits in time in one command.

Caret Parent Spec

Unlike the tilde spec, the caret parent spec allows you to reference the Nth parent of a commit. The distinction here is important as the caret parent spec can walk backwards through commits that have more than one parent (merge commits), while the ordinal spec cannot.

The caret parent spec looks like:

The caret parent spec and the tilde spec can also be chained together, so master^^ would actually point to the same commit as master~2.

Range Spec

The range spec allows you to express a range of commits as a first-class object in git. These are frequently used for things like diffs where you may want to view the changes across a number of commits.

For example, the following command would show a diff of all commits between master and my current feature branch:

You can also refer to the range from the current commit through all commits after it by just leaving off the other ref after the "..".

All of these treeishes are available from the git command line interface -- though if you're using a GUI to interact with git your particular application it may support some or all of these. I know that Github's compare view can be used with treeishes as well. Hopefully if you're not already using treeishes you'll find that they're as useful to you as they are to us.

How are you using git treeishes to improve your workflow?

Image credit: @moonlightbulb

Simple Design, the Highgroove Way

2012-01-16T00:00:00-05:00

Highgroove Studios has taught me a lot about Software Development, Consulting, and building new web applications. Apart from the myriad of technical skills I've added since I came on board, Highgroove is a fantastic company to learn how to build your own web apps, how to design them, and how to remain focused on the most business critical aspects of the system. Highgroove taught me these things by adhering to a process which manages Agile development, responds to changing software requirements and business needs, and encourages constant communication. While these are all noticed by clients, there is one part of the Highgroove process that goes largely unseen; however, it is just as integral as the former three. That behind-the-scenes aspect of the Highgroove development process is keeping the software as simple as possible to meet current demands.

Keeping software systems simple is fundamental to the development style at Highgroove. What I've seen a lot of developers do (not the Highgroove developers of course!) is create an application that is too complex for the current applications scope, which wastes precious time in the process. The unfortunate side effect of creating apps that are too complex for what is actually needed is ultimately an app that performs poorly under load, is prone to errors, and is very difficult to change. All of those things affect a projects budget and could have been avoided. At Highgroove, we leverage the Open Source community and Ruby on Rails to build scalable web applications. This web application development framework comes with a core set of plugins but also enjoys a host of plugins and libraries all available for free. We don't write custom loggers. We don't write google apps or Facebook authentication from scratch. We hardly even have to write SQL. We leverage the network of freely available, well tested and trusted plugin library to accomplish these basic tasks for us. Once the common system functions are handled, that allows Highgroove to focus solely on business logic and building the app. That really means we get more done, in less time, and on budget too!

At Highgroove, we do not believe in creating Rube Goldberg machines…we just want to build the best possible app with the least amount of complexity that does exactly what you need. Its because of this ethos that Highgroove's customers succeed (and I grow as a developer and a consultant too!).

How do you build apps?

Image Credit: Wikipedia

Getting into the Groove

2012-01-16T00:00:00-05:00

Going in to my third week here at Highgroove, I already feel like my first couple weeks have been some of my best in any work environment, both in terms of my personal productivity and my happiness at work.

I'm still completing my undergraduate degree at Georgia Tech. As any student at Tech will be quick to tell you, the school is often very demanding of its students. This can often make a good balance between work and school difficult, and juggling the two over the past couple years of school has been challening.

Enter Highgroove, a ROWE (Results-Only Work Environment).

I've been lucky enough to call a good portion of the Highgroove staff friends for a while, and I'd known that Highgroove works on amazing projects with even more amazing clients. This was more than enough to get me excited to work with Highgroove. But hearing more about how Highgroove is a ROWE during my interview process really interested me.

A ROWE sounds pretty self-explanatory — it's all about results. Focusing on results rather than time spent "working" (who hasn't checked on Reddit at work before?), allows us to work in a fashion that's all about personal responsibility and trust. I can take a day to study for a big exam if I know I'll have some time later on in the iteration to make sure I'm hitting my results. Reading the ROWE book (ironically, the one requirement we have at Highgroove besides meeting results) and working in a ROWE has already taught me a lot: mostly I don't think I could go back to working any other way.

Highgroove gives me control over my time and allows me to focus on getting things done — whether for work, school, or me — in a completely guilt-free environment. And I'm loving every minute of it.

Using MySQL Spatial Extensions for Range Queries

2012-01-10T00:00:00-05:00

We help one of our clients operate a high traffic API that, among other things, returns data that is associated with an IP address range. It's similar to querying for information from WHOIS.

The IP address ranges are specified in a MySQL database with ip_start and ip_end columns. The IP addresses are first converted to their representation as 32-bit integers.

For instance, the range 1.1.1.1 to 1.1.3.3 would look like:

ip_start	ip_end	...
16843009	16843523	...

Read on for more about how we helped optimize the queries that search for an IP within these ranges.

Loading new information into the API required that we test whether a new IP range overlapped with a current range in the database.

It is relatively easy to write a query that tests whether one range intersects another in SQL:

Unfortunately, even with indexes, this query requires a full table scan in MySQL. On a large table, the query took around 0.25 seconds. While insignificant for one-off queries, loading thousands of new ranges made the time really add up.

We scoped out various solutions, but the one that ended up working best was modeling the problem using MySQL Spatial Extensions.

While IP ranges are not a typical use case for spatial extensions, they can be modeled as 1-dimensional lines. And given that spatial indexes are very efficient at doing queries like Intersects(), we found the solution to be really performant.

The following SQL snippet creates a column called ip_range on the table that is then populated with rectangles in 2-dimensions that represent the IP ranges. We only care about 1-dimension, so the height of the rectangles is unimportant:

With that column and the spatial index, queries to test for range intersection can be written as:

This query achieves the same thing as the earlier query, but runs much faster.

Have you modeled a problem with the geospatial features of a database?

Hadoop Across Availability Zones

2012-01-10T00:00:00-05:00

Hadoop is a great solution to the big data problem and with the instant access to servers and storage in the cloud, it's easier than ever to spin up and manage your own cluster. If you haven't heard too much about it yet, hadoop provides access to a distributed file system along with a framework for running map reduce jobs over the data. It takes care of replicating chunks of data to each node and running jobs in parallel for you. However, when you want to expand your hadoop cluster across availability zones you can run into some unexpected problems. So lets dig into the ideas we tried and the final solution that worked the best for our configuration.

Communicating across data centers is a pretty old problem with a lot of different solutions so we wanted to start out with the simplest thing we could think of. First, we decided to allow access by adding an elastic ip to each node and adding those addresses in the security group of the opposite availability zone. So this would basically just allow open communication from certain public addresses on the internet, which isn't a bad solution. But if you ever need to replace or spin up a new node, it requires a decent bit of manual intervention to reassign the elastic ip or to create a new one and add it back to the security groups. Another downside is that, due to hadoop's fancy hostname resolution, we can only utilize tasktrackers within the same availability zone as the namenode.

So next we tried using OpenVPN to create a bridge across the two networks, but there was one small problem. Because we didn't have full control over what ip address our servers were assigned, there was the potential for two machines with the same ip on separate networks which would cause all kinds of weird problems. Amazon's VPC can help solve this problem by giving you more control over the network but we decided to go a different route.

Instead of using a classic VPN solution, we gave tinc a try. Tinc is still considered a vpn daemon but instead of using it for a point to point connection, because it is so lightweight, you can install it on all of your servers and expose a new network interface to route traffic over. This allowed us to easily configure the vpn on the production system and then just switch the interface hadoop was listening on once everything was up and running. So here's how we set it up.

Three files make up the basic configuration. The tinc-up script is responsible for creating the new network interface. The tinc.conf names the local node and keeps track of all the other nodes it needs to talk to. We also need a hosts file which is named after the local node we set in the tinc.conf. This file includes attributes for setting up the connection to the vpn like the external ip, vpn ip, vpn subnet and public key. Since we have everything already set up in chef we just set up a cookbook that ran a few searches across our servers and generated all the configuration files we needed. We also took advantage of hadoop's rack awareness feature to make sure data is always streamed from the closest node and also makes sure replicas are spread across different data centers. There was only 1 problem that we came across which again involved hadoop's hostname resolution. Map reduce jobs were periodically failing because the tasktracker would sometimes use the ec2 hostname and resolve the ip of a datanode incorrectly. So, we had to setup a tiny dns server to make sure that hostnames always pointed to correct tinc address. After fixing that everything worked great. We also had the added benefit of being able to connect through the vpn and access the files from hdfs directly using hue.

Since we quickly covered a lot of new technology, here are some useful resources for when you want to try this out yourself:

What other tips do you have on breaking the single availability zone barrier?

Tech Demo on RescueTime

2012-01-06T00:00:00-05:00

Stafford is demoing on productivity hacking with RescueTime. "It's like server analytics for you and your time," says Stafford.

We all like to think of ourselves as productive, but we can all use some help being more efficient. One tool Stafford likes using is RescueTime that tracks where you're spending your time while using your computer.

RescueTime runs in the background, and tracks applications that are open. It then tries to help you classify which applications are productive and which are non-productive. It can show you how much time you're spending browsing Facebook, versus writing code. You can even help it learn which tasks for you are more productive -- for some people, email is productive, for others, maybe not.

There are some very cool charts that show where time is spent, in your browser, in your code editor, in your email client, etc. It can show you when you're most productive, just by knowing. It also has a competition aspect, you can see how your productivity numbers stack against others, and even yourself each week.

Stafford goes in once a day to clean up his RescueTime and reclassify events like: "Chatting" which may be with a client, or with a friend.

Stafford also showed off the project view, which helps him realize what tasks he's worked on for a client/project.

Stafford ended saying: "It's a good gauge of when I'm being productive. If I'm not being productive, it's time to leave and get away, go do something else, and come back when I'm productive to work."

Check out the video on our Vimeo channel, in our Tech Talks.

Accept/Reject as you go!

2012-01-04T00:00:00-05:00

At Highgroove, we like forward momentum. This means that we know that every delivery cannot be perfect, so instead of worrying about perfection, we worry about progress.

By tracking stories in Pivotal Tracker, we have an easy interface to shoot stories back and forth with our clients. As each story is delivered, the client can either accept or reject it. Acceptance allows us to move on to other stories; rejection requires us to restart the story.

But unlike in some other situations, this rejection does not send us crying into a bathroom stall, diary in hand. In fact, this rejection simply helps refine what the story should actually accomplish.

Sometimes clients seem a bit hesitant to reject a story, opting to instead leave a comment that informs us that while they expected to send an email when they clicked "send email," the system actually logged them out of the site completely. Then the client will proceed to accept the story based on the fact that the button did in fact do something. This helps no one! Reject this!

The constant feedback of accept/reject keeps a developer on track and on pace. When we are waiting for feedback on prior stories, the stories we work on in the meantime lack certainty. We depend on the process to produce and progress.

Acceptance feels great! ...when it is for the right reasons. But the important part is not whether you check "yes" or "no" but that you send the note back before class ends!

Using Git Hunks

2012-01-02T00:00:00-05:00

As a developer, I always try to follow the "Boy Scout Principle" when it comes to the code I'm working with. Simply put:

always leave the code cleaner than when you found it

based on the idea that Boy Scouts always leave a campground cleaner than when they found it.

It is tempting to make sweeping changes and clean up lots of code while implementing a small new feature, however, it's also good practice to separate commits that don't include any unrelated changes. Occasionally this means that a file that has a few small changes that clean up code, and others that are dealing with some functional changes, needs to get committed. Luckily, git has an awesome feature that allows us to get exactly what we want: hunks.

Many of the git subcommands can be passed --patch or -p for short. When used with git add, we can compose a commit with exactly the changes we want, instead of just adding whole files. Once you hit enter, you get an interactive prompt where you're presented with a diff and a set of options.

Some of the more important options are y (yes), n (no), q (quit), and s for "split into smaller hunks". Using this we can easily create a commit with only the changes I made to clean up a file, instead of any other changes I made. When this isn't enough, you can also use e (edit) which will open $EDITOR and allow you to modify your diff and get it just right. Once you save, git will make sure that it still applies to the files you're editing.

Using git add -p allows us to make nice small self-contained commits that only change 1 thing at a time. However, we can use hunks in other places too. git reset -p will allow us to do the exact opposite and unstage things we don't want to commit by hunk. We can also use it with git checkout to completely get rid of changes we don't want and may have accidentally added.

This small feature changed how I work, allowing me to crank through a bunch of code and then go back and compose good commits. Are there any small git features that you really enjoy?

Using Block Scope for a Fancy DSL

2011-12-30T00:00:00-05:00

In Ruby, blocks are kind of a big deal. We use them for everything from basic iteration to executing callbacks. They are also really handy for writing Domain Specific Languages, or DSLs for short. For example, checkout how blather uses blocks to respond to an XMPP message.

We call the message method with some options and a block to execute when a message with those options is received. Blocks make this code a lot more fun to write because the event handler is hooked up behind the scenes. All we really have to worry about is what happens when we receive a specific message. So how would we implement this?

Without including the extra boiler plate to actually receive messages, it's pretty simple code. We just register a callback and call it when the event happens. But, I want to take this a step further. If you've used sinatra, these examples involving blocks should look pretty familiar. Here's an example of a simple get request definition to refresh your memory.

Cool! But how is this different from the blather message example? Look at the block we passed in. Notice anything different? We didn't pass any parameters to the callback, but we used a params variable to get the beverage for the request. We never defined it in the scope before the block in our code. So, where did it come from? Since we always want to have a params hash in the callback of a request it makes sense that it would already be defined for us, but sinatra is actually doing some interesting things behind the scenes to make this happen. Let's look at another example of how we would implement this DSL.

Now I wouldn't use this all the time, but for cases like sinatra's get requests, it can really make for an awesome DSL. Before you implement it in your own project, I would recommend reading over the docs for UnboundMethod and read though the sinatra source code just so you know exactly what's going on. Since this article just skims the surface of what is important to know when building a DSL, here are a few resources that dig a little deeper:

What are some other resources and techniques you've used to make a great ruby DSL?

Using Rails localization outside of your views

2011-12-21T00:00:00-05:00

When I went to Europe for the first time, first to England and then to France, I was deeply obsessed with track cycling. Since I was (kind of) writing my Masters Thesis while my wife was (actually) doing serious research for hers, I had plenty of spare time to hunt down every concrete or wood-banked oval in all of France. My process started out by simply looking for the famous ones like Roubaix and Vélodrome d'Hiver. I quickly realized that every town in France has a great website and they nearly always listed the address of their velodrome if they had one! With that in hand, I would then search for it in Google Earth and put a push-pin there. This was a strange and silly obsession but it taught me a few things. It taught me to read a bit of French, to provide correct accents to Google France's search site, and to do my best to search like I was actually a French speaker. I'm not sure if I succeeded, but this is the exact opposite of what we want our users to have to do with any of the sites we create. Luckily Ruby on Rails provides extensive support for Internationalization (i18n for short) via mechanisms such as Translation and Localization. The prior simply creates mappings between variables (and parameters) and some copy in the language to be displayed. The latter does things like format numbers, times and dates, and currencies correctly according to the language to which we are localizing. This alone should make us ecstatic but we can leverage the tools provided in even more exciting ways.

More after the break.

Recently I have worked with a customer on accomplishing a very simple task: Create a CSV template for download by their customers. The content was mostly static headers with a little bit of dynamic content to boot:

Since we like to do a little something, show it to the customer, and see what they think, this was a completely sane first step. The static headers were provided by them via a story in Pivotal Tracker, so we simply copied them into the code and added a little business logic in the model.

The client was happy and accepted the story but immediately created a second story because they wanted a few more headers that weren't there. Clearly, I can fix this very easily: I just get the new headers, fire up the editor, add them, and push:

Great. Done. Next project.

Actually there are much cooler ways to do this, but first a short aside. Early on at Highgroove, during our weekly code-reviews, one of the team came across a short and easy commit I had made. The only change was to update some copy in the view that was provided by the copy-writer at the client shop. I had created some generic text explaining what I thought things did just as a place-holder, knowing full well (and in fact creating a story asking for) new text which would replace it later. Actually this text was in the i18n file (en.yml) file and looked something like:

And my commit actually updated config/locales/en.yml rather than the view itself. My reviewer said "Have you let the client know they can simply edit the translation file themselves when they want the text changed?" To be honest, I had no idea. I got back a quick skitch that summed it up nicely:

Github provides so much great functionality it can be hard to describe it to someone without running out of breath in the process. One of those features is the ability to edit some files in Github without checking out the repository. This is great for people who aren't developers but are actively involved in the development process in some way, like say, copy-writers!

Great, so now we can work on models and edit files in the browser, thanks to the folks at Github, but how does this work with CSVs? Well, what if we can store the headers themselves somewhere that the client can edit them without asking us? Can we access the internationalization file from within Models and not just views? It turns out we can with the #human_attribute_name method provided by ActiveModel::Translation.

Note that to utilize the "human attributes" we define a new section of our en.yml file per the Translations for Active Record Models section of RailsGuides.

This provides many advantages over the previous implementation: * We separate model logic and data storage. This is the same natural data abstraction that MVC is intended to provide. * Multiple methods can now access the static headers and do whatever they may need to do with them (such as validate them later). To do this in the model we would require a constant, a class variable, or some other mechanism to bind it to the model directly.

Since the localization files are written in YAML we can leverage its flexibility to embed simple objects within our files which leads to another application of i18n functionality in our code. If a client has a complicated external collection of test-cases that is updated often, perhaps based on their own agile or iterative process, then we can leverage i18n in our testing regime.

Consider a test-suite like the following:

In this case the list of the three expected countries (line 6) are provided by the customer, as is the country file. If they have a new and better test fixture that handles some complicated edge case, then they have to provide us with the new file. We can then analyze it, do the right TDD thing by updating the test-suite, etc. Or, the customer could update en.yml with the new test data and the necessary fixture, and we can keep the test-suite exactly as it was. It could potentially break at this point, which is a great place to start our red-red-green TDD process:

Leveraging i18n within all aspects of a rails app, including testing and models, rather than restricting ourself to our views empowers developers and clients in our Agile process. We can work on delivering high-quality code which is also extremely flexible to the needs of the customer. This in turn allows the customer to make necessary changes without a back-and-forth via a medium such as Tracker or a chat-room. Instead 'text-needed' or 'data-needed' stories can be created and delivered without developers being involved unless they are necessary, making the entire team more productive.

Tracking Time the ROWE way

2011-12-15T00:00:00-05:00

ROWE means getting results done and not worrying about when it happens or how long it takes. This is a great way to get things done, but the amount of time spent on tasks can be an extremely useful metric.

At Highgroove, we're strictly ROWE: it doesn't matter what hours we work or how much time we spend on things, as long as the work gets done and the customer is happy. Some people work the 9 to 5 while others work off and on 24 hours a day. Some people manage how they get things done with tools like paper or Things.app, and everyone has a different workload and different set of customers they interact. This is great because tons of work gets done and we don't have to continually interrupt ourselves to track how much time we are spending on projects.

That said, tracking time comes in very handy for us in two ways. First up: new hires that are new to ROWE and not used to having the freedom and flexibility to work however they want don't always pick it up right away. Imagine that you're used to someone telling you what to do every minute of your day and suddenly:

The office is regularly half-full due to people working crazy hours and/or working remotely (from home, other cities, and other states)
All meetings are optional and people regularly skip things that feel they don't need to go to
Someone leaves work early because they got a beta invite to Star Wars: The Old Republic and want to start playing
Beer is poured right after lunch on Friday

Sounds like, uh, time to just go sit by the pool and read comic books! That works for a day or few, but when your project falls on it's face and your customer is unhappy, you probably won't have a job for long.

Until new developers at Highgroove feel comfortable that they have ROWE figured out, they are encouraged to track their time with Freckle or TicToc or something similar. This isn't used for firing people or judging performance or billing, but when combined with good metrics like customer satisfaction it's extremely useful: 20 hours a week and your 100% project's customer is happy means that you get more responsibilities if you want (and more pay). 60 hours a week and an unhappy customer means that something isn't working right, and that something (project management, responsibilities, development practices, customer communication and expectations, etc) needs to change.

Once developers get a feel for how to get work done where only results matter, the time tracking can go away which tends to make people happier more and productive.

However, the second useful case for time tracking is that developers track their time because they want to and they find it to be useful for gauging their efforts. For some, knowing that 40 hours of work done makes it easier to skip out early on Friday. For others, it's a handy way to divide up effort between multiple projects.

A handful of us have started using RescueTime which sits in the background and figures out if your computer usage is productive or not. This is a fantastic tool because it requires no effort or data entry and gives you similar metrics about time spent on projects. I was able to see that those two weeks that I spent playing what seemed like far too many hours of Kingdom Rush may have affected other things, but didn't prevent me from spending the usual amount of time on work and getting all of my work done.

Do you track your time even if the numbers are only for you? Is the mere thought of time tracking enough to send you running in the opposite direction?

Refactoring Regular Expressions with Ruby 1.9 Named Captures

2011-12-14T00:00:00-05:00

I've often felt like Ruby Regexp captures are a bit clumsy.

Let's say we need to break apart phone numbers:

After executing this match, we might do something like this with the parsed number:

What's up with the dollar signs and the sequential numbers?

I feel like I'm writing assembly code and referring to registers or memory offsets or something.

If I'm a new Ruby programmer reading this code, I might have no idea what is going on here.

We can do better if we upgrade from magical variables to the Regexp.last_match method:

At least this is a bit more readable than the magic variables.

And it's probably easier for a newcomer to find documentation for Regexp.last_match than $1.

But there's an even better way in Ruby 1.9 - named captures:

Now we've got readable code, and better documentation built into our regular expression.

Are you using Ruby 1.9 yet? If so, have you had a chance to use named captures?

Lessons Learned at Global Day of Coderetreat

2011-12-13T00:00:00-05:00

cc licensed ( BY ) flickr photo shared by khanb1

In case you missed it, the awesome Globay Day of Coderetreat occurred on December 3rd. The amount of fun I experienced was unexpected and impressive! I learned some things too. Read on to find out what.

(Also, don't worry if you missed the code retreat, sad kitten has some good news for you at the end of this post.)

I managed to make it out to Knoxville where they held the most awesome code retreat. Congrats to Brian Friesen for putting together such an excellent event.

For those not familiar with code retreats, you pair program with six different people under different constraints during each session. You delete all code between sessions. And you only work on one problem, Conway's Game of Life.

Lessons learned:

You can't code Game of Life in forty-five minutes. Or you can, if you're really really good. Or if you've done it five times before in the same day and you skimp on tests and still probably not even then.
Sometimes you're the teacher, sometimes you're the student. Enjoy both!
C# is looking cooler and cooler. But not cool enough to pull me away from Ruby. (Sorry .NET recruiters. You can stop emailing me now. Seriously. Stop.)
How to make proper use of equality in ruby to generate a unique collection of items.

Let's talk about number four!

Most of the attendees were C# developers. I was the lone ruby hold out and convinced a couple of my fellow attendees to give ruby a try. They were enjoying the ruby exploration and implementation until we hit a little snag.

I wanted to get a Set of a Point class instances. Given an array of Points, I wanted ruby to spit back out only the unique items. Overriding == didn't work. Overriding .eql? didn't work either. Neither did any combination thereof. Gah, this was extremely frustrating.

After a precious wasted 10-15 minutes I, mildly embarassed, threw in the towel. We moved on.. and of course, did not finish implementing GoL in ruby.

The C# devs were still impressed with ruby but I couldn't let it go. During the lunch break, I spent some time (another 10-15 minutes) googling and came up with:

See it in action here. I based the code on a nice module implementation of equality and further discussion of equality in ruby.

The key is to override both #hash and #eql?.

Upon further reflection, my chagrin at ignorance of implementing equality in arbitrary Ruby classes was misplaced. The point of a code retreat is to practice, to get better. To learn from one another. Sometimes, even when you're the teacher, you're still the student. That's one of the reasons we love code reviews so much at highgroove).

Have you gone to a code retreat? If so, what did you learn?

p.s. Oh, I almost forgot. The great news for those of you who missed it, or want to again, is that I'm hosting a code retreat in Athens in 2012. Come check out it!

cc licensed ( BY) flickr photo shared by rikkis_refuge

Breaking Bad Software Habits

2011-12-13T00:00:00-05:00

I have only been a professional software developer for a few years, but I feel like I have already acquired bad habits.

One of the hardest for me to break is the desire to create software in a cloistered environment where the requirements are stable and the best technical approach always wins over other constraints like schedule and cost.

Thankfully Highgroove's iterative software development process forces me to flee from these broken methods of building applications.

Read on for more of my thoughts.

I have a formal degree in computer science, and I am glad I have it. However, all the assignments I completed in school had requirements that were set in stone. These requirements would be published, often on physical paper, and given out well in advance of the due date. This works well when attempting to be fair to all students, but customers in the real world rarely have requirements as stable as a classroom assignment.

The first company I worked for out of college claimed to do iterative development, but the process really felt like waterfall with a few small modifications. The solution to customers changing their requirements was to try to nail down those requirements sooner. Code was delivered in big releases every 6-9 months.

These experiences started to shape my instincts: sometimes I feel that uncertain or unclear requirements are a problem that must be solved before development can start in earnest.

That is so foolish. I am learning that developing interesting software often means it is impossible to know the "right answer" up front ... and that is OK.

The answer to uncertainty in software is not to view it as a problem, but to embrace it.

How am I learning to do that .. practically?

Bias towards action: if the "right answer" to a problem seems unclear, bias towards implementing some solution (even if you know it's not the best). Get it out there for the client (or even the client's customers!) to use as soon as possible. The best solution will come by iterating on the non-ideal solutions you unabashedly tried at first.

Test-driven development: Coming up with solutions quickly is good, but not at the expense of bad code quality. The biggest wins of TDD are not code correctness (though that's a nice byproduct), but the design feedback it gives. If the tests are hard to write, you are probably writing smelly code.

SOLID design principles: Learning about--and attempting to follow-- the principles of the SOLID mnemonic have lead to improvements in the code I write. SOLID code is better able to handle the changes that will inevitably come when doing iterative development. For Rubyists specifically, I highly recommend Gregory Brown's article and Jim Weirich's talk.

In short, bad software development habits are easy to pick up, especially in formal education and in companies that have not truly adopted an agile/iterative approach. What bad habits have you broken--or are still breaking--when developing software?

Fake It

2011-12-06T00:00:00-05:00

One of the more complicated Ruby/Rails projects we work with at Highgroove has many points where it interacts directly with the filesystem.

Writing tests for an application whose code requires reading from or writing to the filesystem presents challenges, especially if done naively.

While it's tempting to simply use the real filesystem during unit tests, this presents a few problems:

The tests may be brittle, breaking on systems that are not setup just like the initial developer's local environment.
Setting up fixtures on the real filesystem may not be plausible; for instance, if the code interacts with system files (such as in the example coming below!).
Test code must be careful to clean up afterwards, even in error cases. Otherwise, the file system could be left polluted, dirtying the developer's machine and possibly breaking tests on the next run.
Tests running in parallel may interact with one another, causing random failures (e.g., on a continuous integration server or with parallel_tests).
The filesystem is slow; when attempting to make unit tests as fast as possible, the time to write, sync, and/or read from the filesystem may become significant.

So what's the solution? Fake the filesystem during unit tests.

More after the break.

Consider the following (contrived) piece of code:

Now, what's the best way to test it? The tests can't touch /etc/passwd due to permissions, and even if they could, writing to /etc/passwd would be destructive to the system.

How about mocking File.read?

Assuming we're using RSpec and mocha for mocking:

This works, but tightly couples the test code. What if someone comes along and refactors the production code, making use of File.readlines instead of File.read.split?

The code still works, but the test breaks because File.read is stubbed, not File.readlines. In fact, it's now reading from the real system's /etc/passwd. The code still works, but the test broke. Ugh!

A better solution is to stub the entire filesystem at a higher level, giving a blank slate each time a new test runs. Nothing is ever actually written to the filesystem.

Enter the fakefs gem.

It's trivial to get up and running. If using bundler, I recommend adding it to the :test group:

Next, configure RSpec to include the fakefs helpers to automatically activate and deactivate fakefs whenever a test is tagged with fakefs: true:

And finally, write tests that are tagged with fakefs: true. But this time, test code can manipulate any part of the filesystem, knowing it's completely emphemeral and isolated:

Voila! While there is a bit more test code, the test is verifying behavior, not the specific implementation. Do you have any other tips for testing code that interacts with the filesystem?

Scratching Your Own Itch in the Badgerworks

2011-12-05T00:00:00-05:00

"With all of the people doing this, why hasn't anyone ever just made a ..."

"I really like doing this, but everytime I get to this point I struggle. I need to make that better."

"Man... what I really need is something that makes this job less tedious!"

Those sentiments have reared their ugly heads in some way or another with every crafty endeavour in which we at Highgroove have gotten involved. Sometimes those crafts are working on bikes (crafty and artisanal as well as mechanical), brewing beer (crafty, artisinal, and after some patience, tasty if you're careful), working on rails apps (crafty and often fun!), or pulling shots of espresso (crafty, tasty, and caffeinated. Sometimes with latte art.) Instead of simply wishing there was a solution, Highgroove encourages us to solve the problem, not just for ourselves, but for others.

Highgroove's team is expected to be made up of craftsmen and -women in our work not only by asking us to be such, but also providing us with time to work on our own projects, contribute to open-source projects, and build internal tools to help the team work better. As a result of this expectation, the Highgroove Badgerworks has not one but many exciting projects we not only developed to use internally but also release for your use and enjoyment! In the cases of our open-source projects and internal tools, we start to eat our own dogfood as quickly as possible, often before the developer is "ready" to do so (as was the case with Reflecticle when we started using it to track our work internally). This bleeding-edge mentality has advantages. It reminds us that often 85% correct is "correct enough to ship" because things will undoubtably change (and an over-sharpened sword is often brittle). It also encourages flexibility in our code; the more flexible our code is, the easier it is to respond to the wants and needs and likes and dislikes of early adopters.

Continuing education of our team is also incredibly important, and building our own projects allows us to do that by "finally" getting to play with cool technologies we've come across during research for our work but have never had a chance to use. Want to write a mobile app using Sencha touch? No problem. Want to finally build a Rails 3.1 app, or a Sinatra app, or somehow integrate Hadoop into something you do? Go for it. Tired of not knowing enough about Sunspot and Solr even though there is expertise in house? Please do something cool with it, and then, please ask questions or submit patches.

After you've honed your craft, built your toy, or added functionality to an existing project with a patch, it's time to give back to the rest of the team by showing them what you've done. If the project or tool is just plain neat and you want to show it off we encourage you to step up and do a tech-talk for the team and any visitors we may have. These tech-talks are an informal break from the rest of our work during which we can all take some time to learn something new together in 15-45 minutes. This is a great forum to get feedback on your work to date, ask for help on those niggling details that are still to be worked out, and get everyone else excited about your pet-project as well as using your new tool in their own project.

If, in addition to being interesting and fun, your tool can contribute to Highgroove's day-to-day workflow, as our soon to be released ROWEapp and Reflecticle do, we encourage the team to start using them, and even developing them. The ROWEapp is a great internal project for tracking your progress each iteration of a project. It ties in with the other tools we use (PivotalTracker, LetsFreckle, etc.) and allows us to see the progress the entire team is making on each of the various on-going projects Highgroove is involved in. It has also provided a fantastic place for the team to work together on a cutting-edge rails app where breaking things or making mistakes is not the end of the world. In the past month so much progress has been made by one of our developers that it has gone from being a great 'scratch your own itch' application to a product nearly ready to ship to some of our other friendly ROWEs including ripple, Matchstic, and Dealer Ignition. For us this is extremely exciting!

If you have the itch, it's pretty likely others do, too. If you've done your search of all the likely sites to find your solution and have come up empty, it may be time to scratch your own itch. It's likely you are solving a problem many people have. Once you've done so, Congratulations on contributing to the diverse and useful software ecosystem!

Don't Settle on Just One Test Framework!

2011-11-29T00:00:00-05:00

At Highgroove Studios, we're hyperspecialists in Ruby on Rails web application development. Not only do we write amazingly complicated code on time and on budget, but we follow up all our work with tests so we know our code works.

This is not without its problems however, as a test suite can become just as bloated and difficult to update as the code base it tests! This week, I'll talk about some of the ways I've learned to approach writing tests.

Since I began working for Highgroove Studios back in April, I have been innundated with the importance of thorough and reliable testing. Given the culture of performance and quality at Highgroove Studios, I knew that I had to find a way through the maze of test frameworks and options.

After struggling with some very large, brittle test suites, it dawned on me that good test design, just like designing the application itself, was the key to success. Maybe challenging some of my own assumptions about testing was in order as well!

Once I realized that I needed to focus on writing better tests, instead of writing them faster and getting them to pass, I turned to Google and found some really good stuff that I wanted to share:

What all of this information made me realize was that I was trying to accomplish all of this with just a single test framework and certain ones align to Test Driven Development (TDD) and Acceptance Testing (BDD - Behavior Driven Development) better than others do. (Figure 2 in the AgileData link explains visually why I was wrong!)

After spending the time pondering my own practices, I decided to split apart the two tasks (BDD, and TDD) into Cucumber and Rspec tests, respectively. After making the switch from an all-or-nothing, single test suite mentality to a "use what works best" mentality I've seen both the quality and ease of coding of my test increase. Its amazing what happens when you stop trying to cram a square peg through a triangle hole!

I prefer both Rspec and Cucumber for TDD and BDD, but you might have a different method. If so, what is it and how does it make your day-to-day test writing simpler and more efficient?

Tech Demo on Getting Things Done with Things.app

2011-11-28T00:00:00-05:00

This week's Tech Demo was on Monday because of the holidays. This is a recurring Tech Demo we've done before, and one of my favorites.

We're all productivity junkies at Highgroove, and this week's talk was chock full of useful advice, tips, and tricks from our Methodologist, Chris.

One of the key points in Chris' talk is whatever system you're using, be it Things.app, Pivotal Tracker, or just pen and paper -- there are 5 key habits that highly effective folks use:

Capture Everything in a trusted system
Keep your Inbox Zero
Start Your Day Right
Review Everything Weekly
Do Due Dates Right

This Tech Demo wasn't as technical as usual, but ended with some real live question and answers, and a demo of how to use some of these habits in practice.

For Highgroovers, the demo is on the Time Flux Capacitor (our Drobo-powered network share) for download. In other news, we're thinking of opening up our Tech Demos, like other tech companies do. If we did, would you (our readers) attend or watch in a live-streaming fashion?

Classifying Data with Discriminant Analysis

2011-11-22T00:00:00-05:00

Cluster analysis methods have been gaining popularity as a way of Relating pieces of data in large datasets with one another. Examples in social networking are obvious: friends on Facebook cluster into cliques and communities, which cluster into even larger groups. Demographics and other marketing research can also be aided by sorting prospective customers into groups based on preference.

When the clusters are known, and ample training data is available, discriminant analysis is particularly effective at classifying new data. Discriminant analysis methods are built into the R programming language (something we've discussed a bit in the past) with a standard package. However, R can be cumbersome to use by itself (and the syntax still seems a bit bizarre to me personally), so I used Rinruby, a gem which gives direct access to R methods and data, to put a nice Ruby wrapper around it. Below is some example code that analyzes a well-known clustering test dataset, the Fisher iris data.

Say our iris data is contained as an array ("training_rows") of hashes that each look like

(where the species is given as 1, 2, or 3). We can load our data into a DiscriminantAnalysis instance and start predicting in just a few lines:

Every prediction comes as a hash with a confidence score to check the quality of the classification. This code uses a linear analysis (i.e., it separates classes by lines or planes); the pretty scatterplot up top was generated using the iris data with quadratic analysis, which can be achieved in the code above by simply replacing "init_lda_analysis" with "init_qda_analysis".

R allows for tons of manipulation of the analysis once it's loaded, some of which has been built in to the DiscriminantAnalysis class (scatterplots, accuracy and significance testing, etc.).

Since we're all about contributing to the open source community at Highgroove, I've packaged these methods into a gem called harlequin (cheesily enough after one of the irises in Fisher's data). If you have a working copy of R on your machine, you can get the code above running by just adding the proper requires for the gem. Feel free to check it out and help to make it more awesome!

Have you used clustering methods before? How do you deal with heavy-duty data processing in Ruby?

(New) Team Work

2011-11-22T00:00:00-05:00

The first couple of weeks in a new position for a new company is usually peppered with stressful stretches of striving to become useful and relevant mixed with hours of boredom working on a project of little importance. At Highgroove, we are immediately thrown into the mix and are expected to perform.

Jason F. and I shared a start date. We reveled in our goodie bags filled with organic candy and stickers, fiddled with the settings on our new Macbook Pros, and giggled (at least I did) over how much caffeine I was consuming from our office's Espresso machine.

Then we got to work. I met with my client, talked about what he expected for our first iteration, then was released to make it happen. Harsh reality has never been so pleasant. Where in many environments, asking questions feels discouraged to avoid cause for doubt of your abilities, at Highgroove, not asking soon enough is more discouraged. The team at Highgroove is more than eager to not only answer quick questions, but savor the more difficult ones. Even as a new member, my time is valued by the whole team; they want me to succeed with them. And because of that, I will.

Tech Demo on Merging Conflicting Code

2011-11-18T00:00:00-05:00

Dave Worth's Tech Demo today is on merging codebases, using some workflow tricks and the GitFugitive vim plugin ("a Git wrapper so awesome, it should be illegal") by Tim Pope.

We regularly work with other software development teams, and merging codebases and features is routine work for us. Dave shows us his workflow so we can get faster, and more efficient at merging lots of features all at once, together.

We love sharpening our axes at Highgroove. One way is through slick tools that can save us typing, time, and just make things easier. Using git through an interactive rebase with conflicts can be daunting. Dave shows us how to use vim, coupled with git, along with Git Fugitive to make quick work of a nasty rebase.

For more info on some of the tools, here are some links:

Git Fugitive plugin - https://github.com/tpope/vim-fugitive
Vimcasts - http://vimcasts.org/ (lots of good stuff including basically Dave's demo over n parts)
MacVim - http://code.google.com/p/macvim/ - (like Textmate only different) Zing!

Debugging Best Practices

2011-11-16T00:00:00-05:00

At rubyhoedown, the inimitable Jim Weirich gave an awesome presentation on using the debugger in ruby. Before his new found respect for the ruby debugger, Jim told us that puts statements worked just fine for him.

And this is true. You can get by with puts. But, you can get by much faster using the debugger. Read on to find out when to use the debugger and how.

Firstly, when should you use the the ruby debugger? Simple.

Actually, there is one case in which puts is better than the debugger but let's touch on that later. For now, let us look at how to use the debugger.

Require ruby-debug in your source code. Use ruby-debug19 for ruby 1.9+. If in rails, add ruby-debug / ruby-debug19 to your Gemfile.
Wherever you would put a puts statement, replace with 'debugger; true'.^[0]

Once you're in the debugger, you can revert to your puts using ways. puts any expression to evaluate it. Protip: or type in 'set autoeval' so you don't have to use puts!! You can even set this in an .rdebugrc file. Example

The sweet spot of debugger commands.

n - step over the next line (don't step into a function)
s - step into the next line (into a function or block)
c - continue until another breakpoint or execution completes
v l - view all locally scoped variables

For a more comprehensive overview of commands, check out errtheblog's cheatsheet.

Now, as I mentioned before, there is one time where I like puts over the ruby debugger. When you want to view the change of data throughout the execution of a program, puts to a log file will give you a higher level overview.

Can you think of any others? How do you use the debugger?

[0] 'debugger' does not act like a traditional breakpoint. Instead of halting execution before the 'debugger' line, the ruby-debugger halts execution after. Therefore, we drop a '; true' to ensure that we don't step out of a function or into a method.

The Great Campfire Experiment

2011-11-15T00:00:00-05:00

At Highgroove, we like using the right tool for the job, but we also don't like getting too bogged down in thinking about what we should use. We recently switched our internal company chat from Skype to Campfire, but it wasn't a just simple switch to flip.

We like to use Skype to talk to our customers, since it allows us to be in constant contact with our customers, and it makes it easy to have face to face conversations with them when we can. Since we were already using Skype to talk to customers, we also made a group chat for Highgroovers to talk to one another.

It wasn't a perfect solution, but it was an easy one. As we've been growing, the issues have started to bother us more and more, but it's conveience was impossible to beat. So how do we switch? It's hard to give up something that's already so integrated in our day to day lives. We weren't even sure if the benefits of switching outweighed the convienence of the current solution.

We decided the best way was to run an experiment. We had heard great things about Campfire, and it seemed nice, but would it fit our needs? Our null hypothesis was that Campfire was just as good as Skype, and our alternative hypothesis was that Campfire was better and would fit our needs better. To test this, we decided to use Campfire for 1 week and then evaluate it afterwards.

Starting on a Monday, we all opened up Campfire as well and didn't talk at all in our old Skype group chat. We quickly found things we enjoyed, like inline images, flexible notifications via Propane, and Pivotal Tracker integration, along with things we missed, such as direct messages. Anytime we ran into something we liked or hated, we were vocal.

I collected these opinions, and at the end of the week, asked how they had changed or if they were still issues. As a team we saw that the benefits outweighed the drawbacks and switched. While we would have eventually switched to something better if we sat around longer, we're already enjoying Campfire, and working to improve our experience (like setting up Hubot) with it now, something that would have had to wait for us to decide on the best chat system if we didn't just bias towards action.

How does your organization make decisions? Do you try to decide on the best solution and then use it, or do you try to throw spaghetti against the wall and see what sticks?

One Week In

2011-11-15T00:00:00-05:00

After a week of working at Highgroove I am impressed at our efforts to provide everything a developer needs to start creating the best software possible through the right hardware, tools, and support. For someone like myself, who had lots of programming experience but knew little about Ruby on Rails, this was invaluable in getting me into the Rails trenches and helping my client.

Although I received lots of cool things on my first day of work nothing stood out like the shiny, new, top of the line Macbook Pro sitting on the table. It put the other work laptops I have received, a Windows XP Dell and a worn out Apple Powerbook, to shame. Our policy on tools also separates Highgroove from the rest of the pack. In other places I would have to do a sales pitch to negotiate my way into the sea of paperwork in order to get permission to use a non-standard tool that might help me get my work done. At Highgroove it is not only encouraged but required that I get whatever I need to do my best. If there is a tool unknown to the team that I think will help, I am required to show it off in case it might help them in their work.

The most powerful tool in getting a new developer up to speed costs nothing but time. I've never had an employer that used pair programming, but we encourage and require it. Having someone with more Rails knowledge available to bounce ideas and problems off of gives more insight into the development process than any book or tutorial ever could. Several of the software bugs that caused me to spend too much time banging my head into my desk were quickly resolved through pair programming. No two developers think alike, and having a second thought process in the mix helps avoid lost time and benefits both partners in seeing how the other operates. Now I wouldn't work anywhere that didn't encourage it.

Combining Many Customer Databases into One

2011-11-08T00:00:00-05:00

Here at Highgroove we like a good challenge, and we love it when we can meet a challenge with a simple and elegant solution.

Recently we had a customer with a great problem to have - they had signed up lots of new customers and were experiencing some growing pains.

A typical Ruby on Rails app has one database. This app had one database per customer.

Routine tasks like launching servers, deploying code, or migrating databases were taking much longer than normal because they had to be performed per customer instead of just once.

What this app needed instead was Multitenancy - a fancy word for combining the customers into one database.

To combine safely, we'd need to ensure each customer's unique keys no longer were in conflict with each other.

Here is an example of unique keys before the merge:

Customer A		Customer B
1	Bob	1	Frank
2	Alice	2	Eleanor

And after the merge:

Customer A & B
1	Bob
1	Frank
2	Alice
2	Eleanor

You can see the unique keys are no longer unique.

How do we fix this? Time to look at some code:

This bit of code will give us the maximum id across all tables in all customer databases.

Once we know the maximum id, we can reassign each customer's unique keys with enough offset to ensure there are no conflicts:

This will give us the combined table:

Customer A & B
3	Bob	customer_a
4	Alice	customer_a
5	Frank	customer_b
6	Eleanor	customer_b

Now we're off to the races - with a single database containing records for all of our customers - and no unique key conflicts!

Have you ever had to combine conflicting database records? How did you tackle the problem?

Using Active Shipping for Shipping Rate Estimates

2011-11-01T00:00:00-04:00

Active Shipping is a nifty Shipping API extension for Active Merchant. It provides methods for interacting with common shipping carrier APIs. Recently we used Active Shipping on a client's e-commerce site to provide customers with many shipping options from several carriers.

Getting Rates

To get shipping rates for a carrier, you first need to "package" up some information about what you are shipping. Using the Package class you can build an array of packages. Use each package's dimensions, weight, and the unit system you are measuring with (this will default to metric) to create a new package. Here in an example package:

From working with Active Shipping and various carrier APIs we have learned that all carriers require a package weight. Some will return results for a package with no dimensions. However, rates will vary greatly depending on the dimensions of your packages, so it is a good idea to always include them so that your rate estimates are accurate.

Along with the array of packages, Active Shipping needs to know some information about the origin and destination of your shipment. Using the Location class you can create an origin and destination location for your shipment. Here is an example of a destination (the origin is built the same way):

The Location class takes country, state, city, and zip. Working with Active Shipping and various carrier APIs we have found you can get away with just using country and zip code if needed.

Once we have an origin, destination, and packages we are ready to make request to the carriers' APIs. Initialize the carrier you want to you use, then call its find_rates method:

Getting access to carrier APIs.

Each carrier requires its own set of credentials. You can sign up for each carrier's service on their respective websites. While the UPS codes seem to always "just work", you will need to pass :test => true as a param to get Fedex to work while you are testing your application. We were unable to get the USPS API to provide rates in test mode). To get the USPS API to provide rates you will need to contact and have your production API access turned on. The best way to do this is the call the number provided in the confirmation email they sent. Requests made via email seem to take a couple days to get processed.

Active Shipping is just one of the great extensions of Active Merchant. What other Active Merchant extensions are you using in your e-commerce sites/development?

Tech Demo on Sencha Touch Mobile HTML 5 Framework

2011-10-28T00:00:00-04:00

This week's Tech Demo was on Sencha Touch, a cross-platform framework aimed at next generation, touch enabled devices.

Brandon went over a small sample Sencha Touch and Ruby on Rails app that we're using as a "spike" on the Beltline for Bikes Project for the Matchstic "On the House" community service project.

It's currently compatible with Apple iOS 3+, Android 2.1+, and BlackBerry 6+ devices. Sencha Touch is the world's first app framework built specifically to leverage HTML5, CSS3, and Javascript for the highest level of power, flexibility, and optimization. We make specific use of HTML5 to deliver components like audio and video, as well as a localStorage proxy for saving data offline.

Basically, it plugs right into a Ruby on Rails application, and even has a similar MVC style structure. Sencha can help build out mobile applications that can be submitted into the app store (or just bookmarked on the device). Because they use HTML5 and are backed by a web application, there's a great use-case for data-driven sites that don't require native graphics or access to hardware.

Another neat tool for our tool belt.

Starting Off Right

2011-10-27T00:00:00-04:00

I'll be honest. My reaction to my first morning in the Highgroove office was one which I thought I would not be sharing. After talking to a couple other Highgroovers, however, I realized it was a pretty common reaction. To us, our first day felt a little bit like Christmas morning.

I fully realize how cheesy this sounds, but showing up to a new desk covered in a goodie bag and developer treats must have brought out the giddy 10 year olds in us. My previous jobs have usually provided me with some inherited file folders, a pile of training and maybe a dying fern on the first day. With Highgroove, we got launched head long into developer bliss with a new macbook pro (!), luxurious extra monitor and all the amenities right off the bat.

Possibly more important to the new employee experience than coming into a sweet new workspace is coming in to a ROWE. There is no stressing out to appear busy on your first day, no waiting for someone to tell you when to go to lunch. There's only learning about Highgroove and getting up to speed so that you are ready to work magic for your first client.

Our experience was that even on our first day, we knew we will only be judged by the results of our work and that we will be provided the tools to reach our goals. That has made being a new employee for Highgroove different from any other shop around.

my second annual ruby conf report

2011-10-26T00:00:00-04:00

RubyConf, ahhh Rubyconf.

How was it this year you ask? To answer that question we must first address the question:

How was my first ruby conf?

To which an honest answer is, I couldn't really know at the time (but you can read about my impressions).

When you've done something only once, you don't have enough perspective to judge effectively. Thank goodness Matt and I wrote that post together.

So, how did my initial impressions hold up? First, let's see what happened this year.

Matz's Keynote

Matz covered the continual maturation of ruby. Companies hire ruby core committers full time these days! Awesome implementations annihilate adversaries!

Big Data Enterprisy Analytics App and Ruby

Phusion rails ruby backed by Oracle and MS SQL Server for large healthcare analytics? Done! Ruby conquers the enterprise!

Parallel GC Marking

The first attempt was too fine grained in its approach.

Narihiro Nakamura attained great efficiency improvements by allowing threads to grab tasks from one another when a thread 'stalled' on a large node (typically a large array or hash).

How Getting Buff Can Make You a Better Rubyist

Rob Sanheim reviewed the importance of sleep and exercise, the power of pomodoro, and the value of good long breaks away from the computer.

LocalCommunity.Build, :through => Code Retreats

Here's one of the talks I looked forward to the most. I love geeking out over the internals of ruby or hearing war stories of processes and organizations conquered but learning how to better spread the love? Priceless.

Rails services in the walled garden

Map services to business verticals and keep them self contained and decoupled for the win. Centralize roles but federate the rules. Keep centralized services fast (~40ms) and implement caching.

That concludes my notes for the first day.

...

Where are the rest of my notes you ask? Even though I was super inspired, I found myself strangely drained of energy the second day.

So good yet, so tired

RubyConf 2011 Summary: Come for the talks, stay for the people.

Map Reduce Jobs in Ruby with Wukong

2011-10-25T00:00:00-04:00

I recently got to work on some really interesting, big data problems at Highgroove. One of our clients needed to record every api call and analyze specific time periods for averages and usage metrics. Hadoop fits this use case pretty well with a distributed file system and map reduce framework built in. But, I'll be honest, I wasn't too happy about having to write map reduce jobs in Java. While looking for alternatives I found Wukong; a gem that adds a Ruby wrapper around the Hadoop Streaming utility. Here's an example of how easy it makes writing map reduce jobs.

Now there are always trade offs. Streaming queries, especially to a Ruby script, are definitely slower than their pure Java counterparts. But, between the base startup time of any map reduce job (about 30 seconds) and the speed up from running all jobs in parallel across a cluster of workers, it's really not that bad and totally worth the improvements in readability and maintenance. Also, because this script runs over the entire cluster, you have to make sure that each machine has the right ruby version and and all necessary gems installed, but we easily solved this with a few lines in a Chef recipe.

We've really only touched the surface of what is included with Wukong. There are plenty of different mapper and reducer classes depending on how you have your data structured and how you want to query it. With the script class provided by Wukong, you can run scripts across an entire Hadoop cluster with a single command. It even includes support for writing pig latin scripts in Ruby. So if you know Ruby and want to get involved with some big data projects, Wukong + Hadoop is a great way to get started.

Tech Demo on the R Project for Statistical Computing

2011-10-21T00:00:00-04:00

This week's Tech Demo was on the R Project for Statistical Computing within Ruby. The RinRuby project integrates the R interpreter in Ruby, exposing R's statistical routines and graphics (charting, plotting) from within Ruby.

What would you do with R from within Ruby....

Brian Stanwyck showed off some sample uses using Linear Discriminant Analysis for attempting to "guess" whether or not you are male or female, just by asking your height and weight. It uses known "data" to learn and then extrapolates a guess based on the algorithm.

This comes in super handy for some of our big data projects. Doing a mean, standard deviation, or scatterplot is now a one-liner, and super fast.

Another neat tool for our tool belt!

Email Marketing for your Rails App Made Easy With Mailchimp

2011-10-18T00:00:00-04:00

So you've got an awesome web application up and running. You want to keep your users engaged. You could try to handle collecting emails, writing your own newsletters, managing the list, keeping content fresh, tracking who read your messages, and so on (and so on) -- all within your own app; however, there's a better way. We recommend using Mailchimp. It's actually very simple to integrate Mailchimp into your own Ruby on Rails application. Quick. Easy. DRY. Just the way we like it.

At Highgroove, we're notorious for not wanting to re-invent the wheel. We LOVE using plugins, open-source libraries, ruby gems, and other external integrations in our apps. Mailchimp.com is one of those external services we regularly rely on.

A fairly common scenario is to collect user emails and send them a subscriber newsletter. We need all the basics: collect the email itself, ensure the email is real, send the actual newsletter on a certain schedule, and allow users to opt-out or change their email profile settings.

The first step to completing this online subscriber newsletter is to signup for Mailchimp.com. Once that is done, provide your API Key to us.

The second step is to create a List within Mailchimp. This List could be called "My Sites Newsletter", for example.

The third step is to install the Gibbon gem. This is essentially a Ruby wrapper for Mailchimp.com's own API.

Now that we've taken care of all the setup of Mailchimp, the next couple of steps will show you just how amazing Mailchimp.com and Gibbon really are:

On a basic level, all we need in the Rails application's code is an HTML form that collects an email address and some code within a controller method to save the submitted email address to Mailchimp. We do this in two steps:

First, create your form (this example is a HAML template):

Two, in your controller, add a line like this:

Once you've done the above, your Rails app is fully integrated with Mailchimp! The content, frequency, and other settings that have to do with your newsletter are all controlled in Mailchimp's very simple interface.

The only question you'll have left at this point is "What cool feature can we add next?"

Scaling Searching

2011-10-14T00:00:00-04:00

Search is a hard problem, thankfully a lot of really smart people have spent a lot of time on it and come up with some awesome tools. Most of our projects involve some kind of search functionality, and often tuning search indexes on the database server will get us enough performance to launch the minimum viable product. But say we have 350,000 things that need fulltext search with ordering and boosting, or 50,000 things with extremely complex access controls in a high traffic environment that needs to search quickly and scale up to 10s of millions of things? Here's a low-level look at some of the specific steps we've taken to make Rails scale. Can rails scale?. We disagree. Onward!

Once we've exhausted the ability of the database layer, there are two paths to take. Even though people call me 'old school' for it, I like to start with caching things in memcache. I'll add dalli to my project, set it up as the Rails.cache store, and get started. This begins as a pretty simple method; Here's an example of caching something expensive:

This works great for things that are expensive to calculate, but has two problems. It doesn't speed up actual searching, and when the cache is cold, the expensive calculation is inline and we have to start to do crazy things like this:

'.delay' is a magic method provided by delayed_job which enqueues this expensive computation as a background job. All those 'force' parameters are there to force not using the cache which makes debugging and testing a lot easier, because this get's very crazy very fast when you have 10s of different things you're caching, some of which include results from other cached values. Typically it makes sense for some cached things to be stored in the cache permanently, with code that updates them nightly or whenever their values change, as an empty cache is pretty useless when you're counting on it.

Unfortunately, it's very hard to get caching right and you will have page loads that take 10s of seconds due to things not being cached when you need them. Also, queuing up 1000s of delayed jobs with .delay (due to cache being cold) means 1000s of database inserts which isn't quick either! Something like resque will help a bit, but it's a tiny bit more involved to get running than delayed_job. Perhaps most annoying, things can get deleted from under you so you have to handle the possibility that an id you are using may no longer exist. Here's one way of doing that:

So you've gotten to this point, caching isn't enough, and searching needs to happen and be fast. Enter Solr and sunspot (Which our very own Andy Lindeman works on!). Adding these to your project is explained very well in the sunspot documentation, and if your app is deployed on Heroku like many of ours are, there's a reasonably priced websolr addon that Just Works with the sunspot gem. Adding it to a model is very straightforward

Searching is also pretty straightforward, but this usage is a little more complex in the basics we want to make sure that all of the connection ids return both match the parameters and are in a potentially huge set of IDs that we are allowed to access.

Solr searching is very fast, especially when compared to bigger datasets in postgres, and using it's search_ids method (instead of search) prevents loading all of the objects so that we can rely on the controller/view pattern above for loading only the objects we are displaying (pagination of the resulting array happens in the actual use case). This, combined with Rails 3.1's Identity Map feature or something similar means that we can serve pages and pages of search results very quickly (200ms or so), doing fulltext matching, and displaying information from within the objects, all without having to hit the database.

Everything Just Works on Heroku, and for running things locally, theres a Procfile and foreman for that:

Whats your secret to scaling up?

The Sales Playbook

2011-10-11T00:00:00-04:00

Today, I attended a workshop from the Atlanta EO chapter led by the charismatic Jim Ryerson of Sales Octane on improving your sales process, through the use of a Sales Playbook. I'll admit, I've always been pretty good at selling (it helps to have an awesome team and product behind you), but the Sales Playbook we have (and that went through a ton of editing today) makes it even easier.

What's in a Sales Playbook? Ours contains things like:

our Elevator Pitch and an extended (longer) version
our Sales Map -- a flowchart of all the steps we take to close a deal
explanations, scenarios, scripts, and stories from items on the Sales Map
how we prospect -- what associations and networking events we attend
how we research potential clients
a list of the common objections (and some fodder for consideration)

Just simply writing down the process provides many benefits. I have a hard time saying no, so having a process to abide by, makes it easy to qualify (or not qualify) incoming leads. Also, since everyone at Highgroove can see the process, they know exactly how it's done, and can help improve it. It provides a fantastic answer to a client's question "what's the next step?" It provides clarity to everyone at Highgroove on how projects end up ready to start iterating. It takes the pressure off the "sale" and makes it fun and enjoyable.

We even make part of our Sales Process completely transparent to clients. At the bottom of every email, we bullet point out the next steps in the process, so everyone knows what's next.

Do you have a Sales Playbook, and if so, what's in it?

If you aren't getting burned...

2011-10-05T00:00:00-04:00

Highgroove's "bias towards action" rallying cry is no secret, and we try to abide by that rule whenever we can, whether we're deploying or choosing where to go for lunch. An important corollary is that we try to bias towards making mistakes earlier rather than later, too. A fellow Highgroover captured it well by saying: "If you aren't getting burned, you need to play with fire more."

Learning what breaks a new system helps developers get a feel for when heavier infrastructure might be required (e.g., implementing background jobs if server-side code starts becoming too intensive). And it's part of why we encourage new developers to deploy code on the first day: it's better to learn how to fix production while you're getting situated than when there might be no one around at the moment (especially in a ROWE).

But finding out when your system doesn't break can be just as important. I've discovered security issues in the past by using unsafe code I "knew" shouldn't work as part of an initial naïve solution or experiment.

Well-developed test suites, of course, are what make all of this possible, by telling you exactly what has broken and where. It's also what made me initially come around to TDD; writing tests first means you don't bias your own thinking to the code you've written, making it easier to consider what "breaking" a model or controller (for example) should look like in a test.

Do you encourage experimentation and "failing fast"? How?