Jon McCartie

How To Delete All Your Tweets

2019-01-21T10:32:26+00:00

There may come a time in your Twitter existence when you realize that you’ve said a lot of dumb stuff in your past. Facebook is especially good at reminding you of how immature and annoying you were 10 years ago. So while the reasons may differ from person to person, it’s important to know how to flush your Twitter account.

Unfortunately, Twitter doesn’t make this easy. And the 3rd party online tools have some severe limitations: 1) the Twitter API only exposes a limited number (few thousand) of your most recent tweets, and 2) you’re giving a 3rd party tool all permissions to your account.

Well, if you have a few minutes and have Ruby installed, you can do this whole process by yourself.

Create an Oauth App

Head over to: https://developer.twitter.com/en/apps If you already have an existing Oauth app, you’re good to go. If you need to create one, you’ll need to sign up for a Twitter developer account.

Once you create (or have) an Oauth app handy, view the app details and click on “Keys and Tokens”. You’ll need 4 pieces of data:

Consumer Key
Consumer Secret
Access Token
Access Secret

You may need to generate the Access Token. You can do this by clicking “Create” under “Access token & access token secret”.

Hang onto this info – you’ll need it in the Ruby script later.

Download Your Data

Because Twitter doesn’t expose all of your tweets in the API, you’ll need to get all of the IDs yourself. Luckily, Twitter allows you to download all of your Twitter data.

Head to: https://twitter.com/settings/your_twitter_data
Under “Download your Twitter data”, click “Request Data”

Give it a few minutes and you’ll get a link to download a ZIP file. Inside this, you’ll want to grab two specific files:

tweets.js (this will give you all your tweet ID’s), and
like.js (this will give you the tweet ID’s of your likes)

Do two things to these files:

Remove window.YTD.like.part0 = from the first line. Now it should only be [ {
Save the file to your desktop (or somewhere you can find it easily) and rename the file extension to .json

Delete Your Tweets

Ok. You ready? No turning back, my friend.

Create a file in the same directory as your two JSON files. Call it … twitter.rb ?

require "twitter"
require "json"

client = Twitter::Streaming::Client.new do |config|
  config.consumer_key        = "YOUR_CONSUMER_KEY"
  config.consumer_secret     = "YOUR_CONSUMER_SECRET"
  config.access_token        = "YOUR_ACCESS_TOKEN"
  config.access_token_secret = "YOUR_ACCESS_SECRET"
end

tweets = JSON.parse(File.read('tweets.json'))
tweets.each do |tweet|
  begin
    client.destroy_tweet(tweet["id"])
    print "."
  rescue => exception
    puts "\n>> #{exception.message}"
  end
end

To run this, simply open a terminal window and run ruby twitter.rb.

Common issues:

Unauthorized. Your keys are wrong. (I accidentally had a space after my access token)
Random other twitter messages exposed in exception.message. (I have about 40 @-replies directed at suspended accounts. Twitter won’t show them, but they also won’t let me delete them)

Delete Your Likes

Done with your tweets? Let’s get rid of those likes too. Change your twitter.rb file to this:

require "twitter"
require "json"

client = Twitter::Streaming::Client.new do |config|
  config.consumer_key        = "YOUR_CONSUMER_KEY"
  config.consumer_secret     = "YOUR_CONSUMER_SECRET"
  config.access_token        = "YOUR_ACCESS_TOKEN"
  config.access_token_secret = "YOUR_ACCESS_SECRET"
end

likes = JSON.parse(File.read('like.json'))
likes.each do |like|
  begin
    client.unfavorite like["like"]["tweetId"]
    print "."
  rescue => exception
    puts "\n>> #{exception.message}"
  end
end

Same running process: ruby twitter.rb.

Conclusion

The hardest part of this is creating the Oauth app. I had some old apps from back in the day that I was able to repurpose. Just remember: you can’t put these tweets back. So give it a second thought before you run these scripts.

Happy Twittering!

How Teams Grow, Thrive, and Fail

2018-03-30T09:49:57+00:00

As your team or business grows, things change – they have to! A $10 billion business cannot operate the same as a 1-2 person startup. And since no businesses start at that scale, it’s important to understand what helps a team grow without breaking. How can your team continue to innovate, take on new customers, provide quality service, and maintain a healthy work/life balance for your staff?

A key difference between large and small teams is their amount of process and organization. As new problems arise from growth, successful teams adapt. And they do so without losing what made them great to begin with. But as you’ll see, a team can easily cross an imaginary line into bureaucracy. Their processes can become cumbersome, and begin to suck the life out of them. They lose themselves as they keep adding layers of process in order to accomodate for their growth.

The goal of this discussion is to help you understand the different stages of growth, identify where your team is at, and learn how to take on the next stage with confidence. But first we have to talk about … Problems.

Problems

If your team is lucky enough to grow, you’re bound to experience growing pains. What worked for your team when it was small is no longer working. Communication is breaking down, quality has declined.

But guess what? If you can solve the problems that you’re currently facing, you’ll be handsomely rewarded with … new problems! And they won’t be the same ones – they’ll be larger and more complex. So if dealing with growth problems is a constant, what can we do to prepare for them?

Your role is not to prevent problems. Nor is it to slow the pace of change. Instead, your focus should be on increasing your team’s ability to both recognize and resolve problems. One of these is not enough. High recognition with low resolution is chaos. Low recognition with high resolution is a team with its head in the sand. But if your team can do both – quickly recognize and then resolve problems – you have a distinct advantage.

But how can you possibly foresee the future like this? Thankfully, most of the issues you will face are common to other teams. Understanding the different stages of growth will allow you to see the road ahead and prepare for what’s next.

So let’s take a look at what these different stages of growth look like.

LifeCycles

I was first introduced to the idea of “Lifecycles” through the work of Dr. Izach Adizes. His book “Corporate Lifecycles” is well researched and is a wonderful, detailed exploration. His model starts with the “Courtship” phase, progresses through the apex of “Prime”, then descends to “Bureaucracy” and finally “Death”.

The Adizes model is more detailed than the other model we’ll look at and provides several “short-circuit” scenarios like “Affair” and “Founder’s Trap” that explain certain ways where teams can cut their lives short through negative behaviors.

The second model was created by Les McKeown in his book “Predictable Success”. It takes the highly academic work of Adizes and simplifies it greatly. We’ll explore this one in more depth because it provides a good entry-level overview of the concepts.

Early Struggle

“Early Struggle” is your team or business’ first steps into the world. It’s an exciting, yet terrifying time for your team. For a business, the stakeholders are simply focused on two things: cash flow and establishing a market. For a team within an organization, you might have been newly tasked with creating and building a team and feel the need to prove yourself or even prove that your team should exist! Unfortunately, most new businesses never make it out of this stage. But if you’re lucky enough to do so, it’s time for “Fun”!

Fun

Congrats! The frustration and difficulty of “Early Struggle” is behind you. You’ve established a market, you have a few customers, and you’re not so terrified about revenue. Your team has proven itself and you’re quickly transitioning from cash to sales. Things are growing and you’re having a huge impact because you’ve got your hands in everything. You may not how why you’re growing, but it sure is fun!

“Fun” is also where you get company myths. If your team is past this stage, you’ll hear about the “old timers” that came before you. At Heroku, people fondly reminisce of when we crossed $10 million or that major incident where people were all-hands-on-deck. The people on the team that came before me were gods. These myths are built because at the time, these people were part of small team that did incredible things!

Whitewater

Mo’ customers, mo’ problems. In “Whitewater”, you start to notice that the structure and minimal processes you built during “Early Struggle” and “Fun” aren’t fully adept at handling the growth of your business. If you’re on a support team, you’re finding yourself unable to keep up with all the tickets. So you start to hire. But hiring requires training. What about those feature requests and bug reports? How should we organize them and report them back to product or engineering?

You may be able to come up with ways to change things, but are they the right decisions? And you may have good ideas, but getting these new policies to actually stick is an entirely other issue. Your team is having an identity crisis (and you probably are too).

The focus used to be on cash, then sales, but now it’s on profitability. And profitability requires new systems and policies to bring your team towards “Predictable Success”.

Predictable Success

“Predictable Success” is the perfect balance between process and innovation. Predictable Success is healthy and sustainable. Your team is structured and adaptive, but your team is also given the freedom to work efficiently without being bogged down by process. Your team is able to consistently set and achieve their goals.

The big difference between “Fun” and “Predictable Success” is that you actually know how and why your team is growing and are able to adapt to changes in the business.

Treadmill

You might consider “Predictable Success” a perfect balance between your cowboys and your bureaucrats; between your highly innovative and highly organized types. If that’s the case, “Treadmill” is what happens when your bureaucrats begin to have too much say in things. Policies begin to be too burdensome. Your team is spending more time trying to cross every T and dot every I, and as a result, they’re not getting much done.

The metaphor of a treadmill is perfect for this stage because while a lot of energy is being applied, there’s little to no forward momentum. Risk-taking, creative thinking, and initiative are in decline. We’re now in a world where we second guess everything. Why bother taking a risk?

You see where this is going…

The Big Rut

If “Treadmill” is what happens when process begins to squeeze out innovation, “The Big Rut” is what happens when this environment begins to solidify. “Treadmill” can be reversed – processes can be trimmed back and innovation can be restored. But left unchecked, “The Big Rut” takes over. The team now firmly values process and systems over action and results. The team has also lost its self-awareness and can no longer diagnose problems as they come up. “A problem?! Well, is there a playbook to follow?”

Sadly, companies can stay here for a very long while, but eventually they’ll decay.

Death Rattle

If you love bankruptcies and corporate sell-offs, then you’ll love “Death Rattle”! This is the end of the line. People are jumping ship left and right and things are falling apart. If you were lucky enough, you got out of the burning building with only 2nd degree burns. This is a sad stage for any company, but far too common.

Application

Where is your team?

But the good news is, this doesn’t have to be your story! If you’re reading this, chances are your team isn’t in one of the last two stages (by that point, those organizations have lost the self-awareness to read an article like this!). So how can we apply this understanding of these stages to our teams and businesses?

First, it’s crucial to identify your team along this arc. Is your team small and struggling? Mid-size and cruising? Growing beyond your capacity? Second, once you identify what stage you’re in, you can see what it will take to move closer to the next stage and what will be waiting for you when you get there.

Implementing Process

If there is only one thing you remember from this discussion, it should be this: how you decide on and implement systems and processes is crucial to your team’s success. Create too much process and you’ll snuff out innovation and start your team on the path to The Big Rut. Create too little process and your team will buckle under the weight of its own growth and new responsibilities.

When deciding to implement process, remember two things: 1) only implement new processes when necessary. Don’t create a new policy because it’s a Monday and you feel like you haven’t done much work as a manager lately. And similarly 2) don’t create policies for one-off issues. Wait to see if a pattern arises. Nothing is worse as an innovator on your team than to have Mike from accounting screw up everything for the rest of us because the team manager felt like every decision now has to have multiple layers of approval and now things are stupid and suck and I’m going to start interviewing somewhere else tomorrow.

Finally…

We’ve quickly covered the ideas of a team lifecycle. Hopefully, you’re able to identify where your team is at and you’re now able to see what’s behind and ahead of you. And you’re now able to see where this is all headed – towards a “prime” position. Balancing on the tip of this arc, your team has found middle ground between too much and too little process. Innovation is fostered, but is given healthy boundaries that encourage growth. And you’ll be able to see what it looks like when your team begins to tip too far into systems and administration.

“Your people are doing their best, but their efforts cannot compensate for your inadequate, dysfunctional system.” Peter Scholtes, The Leader’s Handbook

Take some time to self-reflect on where your team is and where it’s headed; what challenges await you around the next corner. Doing so will enable your team to attack the problems – instead of each other.

How Not To Squash Your Team With Red Tape

2018-02-09T16:18:30+00:00

One of the common tenets of growth theories is that companies require process and structure as they grow. And if they’re lucky, they’ll be around long enough to reach a critical tipping point where too much structure will begin to squash innovation. It’s easy to find examples of companies that have crossed this threshold — they’re slow, bureaucratic, behemoths. How did they get there? At some point in time, someone decided they needed a new process or policy without considering the cost.

I’ve been thinking a lot about this on the Heroku Support team. We’re at a critical point in our evolution where we’re growing fast and trying to keep up. New policies and procedures are probably necessary to streamline our operations. However, we’re also dangerously close to that important tipping point of flexibility versus bureaucracy.

It’s also very tempting as a manager to want to try to fix everything for your staff. You’ll spot an inconsistency or ineffective workflow and immediately want to throw on a cape and get to work. But too often we place even greater burdens on the team that we were trying to help in the first place.

For example, your team and ticket load is growing and you’re seeing more and more examples of unhappy customers demanding refunds. Your gut instinct may be to implement a hard-and-fast policy: “All requests for account credits much be approved by 2 managers!” Instead, trust your staff and give them guidance on how to deal with these situations. If you’re worried about it, place a soft limit and have them consult you if the amount is more. Then continually train and guide your staff to be more self-sufficient.

Over-bearing policies can also cast too wide of a net and affect the wrong people. It’s all too easy to try to affect the behavior of a single team member by creating a policy that inadvertently affects the entire team. One agent handles a ticket poorly and suddenly an entire new process is enacted to fix one isolated incident. As a result, the entire team suffers.

Instead, I’d encourage you (both manager and team member) to think long and hard as you add additional processes and structures to your team. Before you do so, ask yourself these important questions:

What am I trying to solve here? Do I have a clear understanding of the specific problem?
What negative side effects might this change trigger? Could other positive behaviors be negatively affected?
Have I run this idea by other members of the team? Do they agree on both the problem and the solution?

Implementing new processes is far easier than removing them. And often times, many issues can be resolved at the individual level. So before you decide to change out your support tool, change up a playbook, or require a new process in handling tickets, stop and consider the full cost and effects of a new policy. Your team will thank you!

If you’re interested in hearing more about this: I’m really excited to be speaking at Elevate Philadelphia this spring! My talk is called “How Teams Grow, Thrive, and Fail” and we’ll be mostly talking about corporate growth theories and how they can apply to our teams.

Building Healthy Teams - Elevate Summit, Denver

2017-10-07T18:21:55+00:00

In preparation for speaking at Elevate Summit in Denver, I prepared this script for my talk. I wasn’t able to memorize it fully, so wanted to share it in text since there were a few key points that may be easier to digest in this format. The video of the talk is embedded at the end. 👍

Intro

After working in corporate-land right out of college, an opportunity came up to join a hot new startup in San Francisco. I just bought an outrageously overpriced first home (thanks housing bubble) and was very excited for my massive startup payday. So we packed up the moving truck and headed to the Bay.

The first few months were very exciting. Our small team was a powerhouse and we were getting tons of interest and new customers. Fast forward a year and things … were different. We had raised a few rounds of capital and we had grown to fill our needs. But our team was no longer … a team.

Here’s an example of what I mean:

Our CEO had demanded a 9am start time from everyone, so each of us filed in on time. Those that were late made sure to stop by the CEO’s office (near the front door) to give him our excuse for our tardiness that day. We mumbled our “hello”’s to each other at the coffee pot. The person who arrived the latest that day could usually be found grumbling at the lack of coffee when they arrived.

We all worked in relative silence in our open office. Then – around 11:45am – came the day’s moment of truth. Your stomach would start to grumble, announcing that it was time to make a decision: do I really want to eat lunch with anyone in here? The answer, of course, was “no”. But you couldn’t just throw your coat on and head to the elevators. Someone would know what’s up. And the only thing worse than a group lunch was a one-on-one lunch with someone you didn’t like.

So, the trick was to not take your coat. Stand up. Stretch. Maybe add in a yawn. Then slowly walk towards the front of the office. If anyone stopped to ask you if you were headed to lunch, you quickly respond: “Me? No. Just headed to the bathroom.” If caught in this lie, make sure to hang a left towards the bathroom. Maybe stand in there for a minute to really sell it. Stare at yourself in the mirror and wonder how the hell you ended up working somewhere like this. Then slowly re-open the door and sneak down the hallway to the elevator. Head to some place along Market St where you know no one else will eat. Finish your lunch, then stare at your watch and watch the minutes tick by until it was time to return to your desk.

This was my life in San Francisco. It’s no wonder I left after only a year. But sadly, this is what happens when teams are not cultivated: when growth consumes the culture of a team and focus is not given to a team’s health. And sadly, this is an all too common experience.

According to a 2015 Gallup survey, only 32% of people say they are “engaged” at work. Aside from these toxic work environments being a drain on the individual, they have serious business impacts as well. Disengaged individuals tend to do the bare minimum at work. Low morale breeds low morale and it spreads across entire organizations. It costs the U.S. an estimated $450 billion to $550 billion per year of lost productivity, stolen goods, and missed days of work.

So if working somewhere like this sucks so bad, why aren’t we doing more? Why aren’t we actively engaged in building healthy teams instead?

Frankly, nobody taught us how.

Now, there are a litany of managerial and leadership books you can pick up. Each has their own unique perspective on this topic. Each one may help a different team in a different stage in its growth. But for the teams I’ve had the pleasure of leading, I’ve found 3 strategies most effective.

Clarify the Why

First, we’re going to clarify the “why.” Pat McMillan said: “A clear common compelling task that is important to the individual team members is the single biggest factor in team success.”

If you’ve ever worked on a crappy team, odds are that there was no “clear, common, or compelling task”. Was it “make a bunch of money”? Sell the most ads? For lots of people, business metrics are hard to get excited about. So what’s the goal here? What is your team doing that’s exciting or compelling?

But before you call for a team offsite to layout your corporate mission statement, your best bet is to start with some introspection. Find a quiet space and begin to really think through some questions. What drew you to this company? Why are you working there? What makes you get up each morning and go to work? What’s the problem your team is solving, or the opportunity it has before it?

As a manager on the Heroku Support team, I love helping people. Heroku is a platform that runs over 6 millions apps, receiving over 13 Billion requests per day. Our customers are unique in that most are programmers. I’ve been programming for almost 20 years and so I feel a unique connection to our customers. Sure, we get the usual “how do I reset my password?” tickets, but most of the time we’re there to help fellow programmers solve a difficult problem and get on with their day. This is incredibly rewarding for me and everyone on my team (who are also programmers).

So for me, I tell my team that I love helping fellow programmers get un-stuck. Programming is hard and I want every customer that opens a ticket to be amazed at how great our team is in answering even the toughest of questions. And as a manager, this is an easy idea to rally my team behind. It elevates the cause of our team and gives us a clear goal. In addition, when we have a clear goal ahead of us, it often helps solve many other problems the team faces. “Should we invest more time in better documentation?” Well, does it help our fellow programmers get un-stuck? Yes. Then do it. “We’ve seen a lot of tickets about this new feature.” It looks like we’re causing unnecessary issues for our fellow programmers. Fix the damn bug.

So find the common goal for your team. Don’t start with “what” your team does – it’s important, sure. But instead, start with “why”. Why are we all here? And once you have that, tell your team. Then repeat it. And repeat it again. Focus has a tendency to drift, but if you continually remind each other about your “why”, you have a better chance at building a healthy team.

Find and Develop Individual Strengths

Have you ever heard of Rudy? If you’re too young to remember, the movie “Rudy” was about a kid who grew up near Notre Dame. All his childhood he dreamed of playing football. When we grew up, he was accepted to Notre Dame, but not on the football team. But he worked hard, practiced, and helped out on the team. And one day … at the final game of his senior year, the crowd begins to chant: “RU-DEE! RU-DEE!” Finally, the coach turns to him in the final few minutes of the game and sends him out on the field. He makes a few plays and when the game was over, his teammates carried him out of the stadium on their shoulders.

Wow!

But here’s the thing. Rudy sucked at football. But we love the underdog. It’s a great story to watch, but at some point you have to wonder: what was Rudy really good at? How would his younger years have looked different if someone told him the truth and helped him discover his natural talents.

On my teams, we work to find and develop strengths. We don’t ignore weaknesses, but we certainly don’t dwell on them for too long. Instead, we work together to find the things that come naturally to you and focus on developing those.

During one-on-one’s, I’m constantly looking for a glimmer in someone’s eye – that look of excitement when they’ve finished a certain ticket or task. The look or tone of voice that tells me that this thing is something they loved doing. Or I look for times when a task is done in half the time as expected. “Was this hard for you?” Nope, it came easy.

Bingo. Now we have a conversation starter. Why was it exciting to do that type of ticket? What about that task was easy for you? If it makes sense, we now have a path towards growth. Over and over on our teams at Heroku, we’ve grown people out of the Support team. Retention is not a metric for me. I am actively working with my team to grow them. “What?! What if you lose them?” I’m ok with that. I’d rather grow them and lose them, then force them into stagnation and keep them around.

Chances are, you have some incredibly bright and talented people on your team, but no one has told them not to play football. Sit with them, invest in them, and see what incredible things they’re capable of.

Create a High-Feedback Culture

Finally, we get to talk about one of the hardest things about leading a team: feedback.

There are many long and leather-bound books about this topic, so I don’t want to stay here too long. But I think there’s an unhealthy focus on this topic around critical feedback. Don’t get me wrong – providing critical or negative feedback is absolutely necessary and it’s something that’s a learned skill.

But we far too often ignore the fact that feedback must come from all directions. A healthy team not only extends feedback up and down the org chart, but members accept feedback. No, they crave it.

Two jobs ago, the CEO of our company popped his head into my team’s office. He scanned the room, and called out to me: “Can I borrow you for 30 minutes?” Oh crap. I headed to his office, where he had assembled people from all over the company.

“I’m giving this presentation next week and want your honest feedback. Don’t sugar-coat it. Tell me what I’m missing or what things I need to take out.”

It was terrifying to tell your CEO what was wrong with his presentation. But it was incredible. Not only did this guy preach “high feedback”, but he lived it, too!

And when your team isn’t worried about how you’ll take it, you’ll be amazed at how helpful that feedback can be. Ask for it! Don’t shy away. And learn to take it well – even the crappy ideas. Thank them, digest the feedback, then pick out anything you want to keep. But in doing so, you’re creating a valuable level of trust amongst your team.

Let’s talk about positive feedback. It’s the easiest type of feedback to give, but we rarely do it. Why is that? Why is it so hard for us to acknowledge someone for doing a good job?

According to a American Psychological Association survey, 93% of employees who reported feeling valued, said that they are motivated do their best work. And yet, we seem to only dole out praise when people leave the company.

Let’s start using the Homeland Security procedure: “see something, say something.” If you see something good on your team, just say it! Call it out! Some people don’t like public praise – that’s fine. But encourage people when they’re doing the right thing.

And when we do express praise, we tend to only focus on the work – not the person. We need to move from individual accomplishments to wholistic praise and gratitude. We need to move from “Nice work on that ticket” to “I’m grateful for you and your work on this team.” Do you hear the difference?

We don’t have workers on our team – we have people.

Conclusion

Dr. Henry Cloud said: “As a leader, you are always going to get a combination of two things: What you create and What you allow.”

Building a healthy team is an act of intentionality. You cannot turn it on and let it run by itself with no guidance, and expect a positive result.

And we all have an obligation to create positive, healthy work environments. This isn’t just a management thing. Think about the work you’re doing. If we’re honest, a lot of the work we’re doing won’t be around in 15 or 20 years. But the relationships, the friendships, the people’s lives on your team who you choose to pour into … those will last.

If you’re a manager, you’ve been entrusted with a few short years of someone’s life. It may one day end up as a bullet note on a resume. Or it could be one of the happiest and life-giving moments they’ll ever have. Choose to invest in the people around you.

Rails 5.1: Default Primary Keys

2016-12-05T13:34:31+00:00

Friends don't let friends use INT as a primary key.
— Richard Schneeman (@schneems) May 13, 2016

A Bit of History…

Remember the Twitpocalypse? Back in 2009, Twitter ran out of Integers. Like most of us, they set up the primary key in their tweets table to be a 32-bit signed integer. Once they hit the magical number (2,147,483,647), servers were all but guaranteed to catch fire in the most spectacular fashion.

Just in the nick of time, however, they managed to migrate to using 64-bit integers and bought themselves a nice chunk of time (the max is now 9,223,372,036,854,775,807).

Now, I’m sure you’re thinking to yourself: “Self, should I be worried about this?” And some days, it may feel kind of silly to worry about ever having more than 2.1B rows in a single table. But Twitter was like you once and didn’t worry about it … until it was too late!

Well, worry yourself no more. Primary keys in Rails 5.1 are now big integers by default! 🎉

But why not UUID?

Hey, I love me some UUID’s. But their usage as primary keys is still debated in the community. So for this change, we opted for making a more sane default that was more widely accepted.

But don’t give up hope! You can default to using UUID’s for primary keys in generators if you want. I submitted that patch last year: https://www.mccartie.com/2015/10/20/default-uuid’s-in-rails.html

Migrate

Already have a table with 32-bit integer primary keys? It’s easy enough to migrate them:

$ rails g migration change_primary_key_to_bigint_on_users

Then…

change_column :users, :id, :bigint

Richard Baptist mentioned to me that he needed to run the following migration for MySQL 5.7:

change_column :users, :id, :bigint, unique: true, null: false, auto_increment: true

The usual warnings apply when doing this to a large table running in prouduction…consult your local DBA for side effects.

NOTE: If you perform this upgrade, make sure to update foreign keys as well. If your primary key and foreign keys don’t match, Postgres won’t give you any warnings – MySQL will break.

Rails Core Team FTW

This was a big change to make. A fellow Heroku co-worker (“Herokai” Pavel Pravosud ) kicked this off back in May with a change for Postgresql. Unbenknowst to me that Pavel had started with this work, I made my own change to include both MySQL and Postgresql (sorry, SQLite doesn’t do bigint primary keys). I was able to merge Pavel’s work into my own. Then began the difficult (yet satisfying) task of hashing out the finer points with the Rails core team. But looking back on it all and seeing the quality of the tweaks that were suggested, I’m really grateful for the back and forth.

So find yourself a Core Team member today and give them a hug. Tell them thanks for their tireless efforts. And stop worrying about whether or not your primary keys are going to run out.

Cheers!

Token-Based API Authentication

2016-11-03T11:02:09+00:00

A friend recently came to me wondering how he could add token-based authentication to his API.

I used Devise for my app, but it looks like they removed token auth. I’ve found a few gems, but they all look to do more than I need.

I’ve been critical of Devise for a long time. I used it exclusively on my earliest Rails sites because of how popular it was and how powerful it was out of the box. But trying to customize Devise is its biggest downside. In this instance, Devise is percevied as your authentication gatekeeper – and homage must be paid.

The good news is: feel free to keep Devise around, but you don’t need it for your API. We can use some relatively unknown methods built right into Ruby and Rails. Here’s how…

Migrations

We’re going to assign a token to each user. Once the client signs in, they’ll receive a token that can then be used for all authenticated API calls. Let’s get that onto the users table.

$ rails g migration add_auth_token_to_users auth_token

This creates the following migration for us:

class AddAuthTokenToUsers < ActiveRecord::Migration
  def change
    add_column :users, :auth_token, :string
    add_index :users, :auth_token
  end
end

We’re going to be looking up users based on that auth_token, so that add_index method is very important here.

Run the migration:

$ rake db:migrate

The Base Controller

I’ve built plenty of Rails-based API’s in my time. The first step for me is usually to make sure I have a good “base” controller for all API controllers. This allows me to set things up apart from ApplicationController, which typically is heavily bloated with methods that I don’t need in my API. In this case, it will allow me to set custom authentication methods outside of Devise.

class Api::ApiController < ActionController::Base
end

There’s a lot to digest here, so let’s break it down:

First, we need a way to protect our controllers:

  def require_login!
    return true if authenticate_token
    render json: { errors: [ { detail: "Access denied" } ] }, status: 401
  end

We’re stubbing out a authenticate_token method here for now – we want to encapsulate that logic in a separte method. But if it returns a user, we’re all set. Otherwise, we return an error message and a 401 status.

We know that the vast majority of our controllers are not accessible without authentication, so we can run the require_login! method as a before_action in this controller. If we need to skip it for certain endpoints (like sign-in), we can use a skip_before_action there.

So now we can move on to the authenticate_token method:

def authenticate_token
  authenticate_with_http_token do |token, options|
    User.find_by(auth_token: token)
  end
end

Rails has the authenticate_with_http_token method built-in. It’ll handle all the details for us here – we just need to know how to lookup the user with the token that’s passed in as a header.

Next, we’ll finish this up with some helper methods like current_user and user_signed_in?.

The finished controller:

# app/controllers/api/base_controller.rb

class Api::BaseController < ActionController::Base
  before_action :require_login!
  helper_method :person_signed_in?, :current_user

  def user_signed_in?
    current_person.present?
  end

  def require_login!
    return true if authenticate_token
    render json: { errors: [ { detail: "Access denied" } ] }, status: 401
  end

  def current_user
    @_current_user ||= authenticate_token
  end

  private
    def authenticate_token
      authenticate_with_http_token do |token, options|
        User.find_by(auth_token: token)
      end
    end

end

Signing In

So how do we get the token for the user? Just like a normal sessions controller – but we’ll return the token instead of handling setting session info and performing redirects.

First, we’ll add some routes:

# config/routes.rb

  namespace :api, :defaults => {:format => :json} do
    as :user do
      post   "/sign-in"       => "sessions#create"
      delete "/sign-out"      => "sessions#destroy"
    end
  end

And the controller:

class Api::SessionsController < Api::BaseController
  skip_before_action :require_login!, only: [:create]

  def create
    resource = User.find_for_database_authentication(:email => params[:user_login][:email])
    resource ||= User.new

    if resource.valid_password?(params[:user_login][:password])
      auth_token = resource.generate_auth_token
      render json: { auth_token: auth_token }
    else
      invalid_login_attempt
    end

  end

  def destroy
    resource = current_person
    resource.invalidate_auth_token
    head :ok
  end

  private
    def invalid_login_attempt
      render json: { errors: [ { detail:"Error with your login or password" }]}, status: 401
    end
  end

We’re using two Devise methods in the create method here: find_for_database_authentication and valid_password?. If you’re not using Devise, you can easily replace these with your own authentication system. The key here is to load and verify the user. Once this is done, we’ll ask the User model to give us a token. Note that we’re delegating this responsibilty to the User model – this is not the job of the controller!

Likewise for destroying this token, we provide a sign out method. We’ve stubbed out invalidate_auth_token on the User which we can build next.

Please also note that we don’t immediately error if the user is not found – we’ll load User.new and still check the password even if it’s blank. This prevents attackers from timing our response times to determine if an email is valid or no.

User Model

In the sessions controller, we stubbed out two methods on the User model – one to generate a token and one to invalidate this token. Let’s fill those in:

# app/models/user.rb

  def generate_auth_token
    token = SecureRandom.hex
    self.update_columns(auth_token: token)
    token
  end

  def invalidate_auth_token
    self.update_columns(auth_token: nil)
  end

We’re using SecureRandom to generate a random hexadecimal string. This will generate a 32 character string. For example:

irb(main):001:0> 10.times { p SecureRandom.hex }
"c46890f205b82c1b74c750c3dce43223"
"0cda5c4fd3b4e9536b823696fbd8874b"
"8d8d96a8dc575b7d3659acf8e37aee64"
"60d5f9a3586da5290bdbfab88ec6e47a"
"25a6c19009805f89c71b4a9079a3585f"
"336b75f35e60f987b477b852c29989f3"
"849dcecdf685712517d9f5d0a7793138"
"554992f0b9cabb68b9aa8a90070da259"
"e5159948971bf835deb59df0f5ac3efe"
"5dd10aca5fea289027228f422898f48f"

generate_auth_token will generate a new token, update the database, and return the token to be used by our SessionsController. invalidate_auth_token will simply set this value to nil and disallow the User to be authenticated with this token in the future.

API Usage

Now that we’ve got all the moving parts, let’s test things out with curl.

# Initial Authorization
$ curl -X POST --data "user_login%5Bemail%5D=jon%40mccartie.com&user_login%5Bpassword%5D=please1234" https://localhost:5000/api/sign-in.json
{"auth_token":"a88601e054ba3df53bf9c9ff2d0d24f9"}

# Protected Calls
$ curl -H "Authorization: Token token=a88601e054ba3df53bf9c9ff2d0d24f9" https://localhost:5000/api/people.json
[{"id":3,"name":"Lonzo McDermott","profile":...

# Example Failed Authentication
$ curl -H "Authorization: Token token=abc" https://localhost:5000/api/people.json
{"errors":[{"detail":"Access denied"}]}

# Sign out (returns blank 200 response)
$ curl -X DELETE -H "Authorization: Token token=a88601e054ba3df53bf9c9ff2d0d24f9" https://localhost:5000/api/sign-out.json

Expiration

Maybe you don’t like the idea of these auth tokens living on indefinitely. We can easily add some logic to expire these tokens.

First, we need to know when the token was generated at. So we add a token_created_at field to User:

$ rails g migration add_token_created_at_to_users token_created_at:datetime

Note: now we’ll be looking up users now not just by auth_token, but by auth_token and token_created_at. Let’s make sure to add a compound index:

class AddTokenCreatedAtToUsers < ActiveRecord::Migration
  def change
    add_column :users, :token_created_at, :datetime
    remove_index :users, :auth_token
    add_index :users, [:auth_token, :token_created_at]
  end
end

Next, let’s make sure we touch this attribute when we create and destroy tokens:

# app/models/user.rb

  def generate_auth_token
    token = SecureRandom.hex
    self.update_columns(auth_token: token, token_created_at: Time.zone.now)
    token
  end

  def invalidate_auth_token
    self.update_columns(auth_token: nil, token_created_at: nil)
  end

Now in our base controller, we can check to make sure the token is still “fresh”. Should we validate the user, then check the token_created_at value? Gadzooks, no! Let’s make our database do that.

# app/controllers/api/base_controller

def authenticate_token
  authenticate_with_http_token do |token, options|
    User.where(auth_token: token).where("token_created_at >= ?", 1.month.ago).first
  end
end

The API client will get the unauthorized response and can attempt to sign in again to fetch a fresh token.

Conclusion

As Ruby developers, we have a littany of incredible Gems available to us. And when faced with the question “Should I build this myself? Or use a gem?”, there’s usually incredible value to not re-inventing the wheel. But make sure your reliance on other libraries never blinds you to simpler solutions that can be fully customized to your needs. Sometimes “rolling your own” can actually save you time.

Happy coding!

Rails and Slack: Creating a Slash Command App

2016-09-15T13:49:52+00:00

Back in my Campfire days, I remember being blown away when Github released Hubot. Our team tinkered around with our install for days – hoping to make Hubot do our bidding. It was a great little tool (and still is!), but I’ve recently fallen in love with Slack’s “Slash Commands”. They’re a handy way to send commands to a service and get a response. And the best part? You can write one in Ruby!

The Slack Webhook

Slash commands work in a simple manner: you start a command with a forward slash, then the command name. For this basic example, we’ll ask for weather for a particular zip code:

/weather 83864

The entire command, along with any additional text, is sent as a webhook to your application.

token=gIkuvaNzQIHg97ATvDxqgjtO
team_id=T0001
team_domain=example
channel_id=C2147483705
channel_name=test
user_id=U2147483697
user_name=Steve
command=/weather
text=83864
response_url=https://hooks.slack.com/commands/1234/5678

The token is a secret setup when you configure the command inside your Slack instance. You’ll be checking this to ensure the command is actually from Slack.

The response_url is where we’ll send our data once it’s ready.

Receiving the Command

The webhook from Slack comes as a POST request, so we’ll create a Rails controller with a create method:

class CommandsController < ApplicationController

  def create
    return render json: {}, status: 403 unless valid_slack_token?
    CommandWorker.perform_async(command_params.to_h)
    render json: { response_type: "in_channel" }, status: :created
  end

  private

    def valid_slack_token?
      params[:token] == ENV["SLACK_SLASH_COMMAND_TOKEN"]
    end

    # Only allow a trusted parameter "white list" through.
    def command_params
      params.permit(:text, :token, :user_id, :response_url)
    end

end

We’re doing a few simple things here:

We immediately return a 403 (forbidden) unless the Slack token matches. We do that with the private method valid_slack_token? which compares the token in the params with the one Slack gave us (and is stored in an ENV var).
Next, we’ll send off our entire hash of params to a Sidekiq background job. (I had some trouble serializing the params hash in Rails 5, so I’m using .to_h to do this for me) We’re using a background job here since we’ll be communicating with a 3rd party API. That way, we can return a response quickly to Slack and avoid a timeout.
Finally, we return a 201 (created) and send back some JSON.

The { response_type: "in_channel" } JSON tells slack to leave the original command (/weather 83864) in the Slack chat history. If you’re sending something back that only the user will see privately, you’ll want to return { response_type: "ephemeral" } at this step.

Returning Data

Next, we’ll have our Sidekiq worker fetch the data and send it all back using the response_url that Slack sent us in the params.

To fetch the Weather data, I’m using Weather Underground. I’m going to create a basic class to encapsulate some of this behavior.

class Weather

  attr_reader :zip

  def initialize(zip)
    @zip = zip
  end

  def city
    data["full"]
  end

  def temperature
    data["temperature_string"]
  end

  def icon_url
    data["icon_url"]
  end


  private

    def data
      @data ||= HTTParty.get("https://api.wunderground.com/api/#{ENV['WUNDERGROUND_KEY']}/conditions/q/#{zip}.json")["current_observation"]["display_location"]
    end
end

And then, the Sidekiq worker which uses this class:

class CommandWorker
  include Sidekiq::Worker
  sidekiq_options :retry => false

  def perform(params)
    weather = Weather.new(params[:text])

    message = {
      text: "The temperature in #{weather.city} is #{weather.temperature}",
      response_type: "in_channel"
    }

    HTTParty.post(params[:response_url], { body: message.to_json, headers: {
        "Content-Type" => "application/json"
      }
    })

  end
end

Here’s what’s happening:

Our worker starts up with the params it was sent by Slack.
We initialize a Weather object and pass in the text from the message (zip code)
The Weather instance fetches the data from the API, then exposes it back to the CommandWorker to be used in generating the text.
We post back our text string to Slack, using the response_url value it sent us in the first step.

Attachments

To make our message a little prettier, we can send along the image URL as an “attachment” to Slack. This requires simply modifying our JSON a bit like this:

{
    "attachments": [
        {
            "title": "Weather for Sandpoint, ID",
            "title_link": "https://www.wunderground.com/US/ID/Sandpoint.html",
            "text": "Optional text that appears within the attachment",
            "fields": [
                {
                    "title": "Temp",
                    "value": "68F",
                    "short": false
                },
        {
          "title": "Wind",
          "value": "16mph",
          "short": false
        }
            ],
            "image_url": "https://icons.wxug.com/i/c/k/clear.gif"
        }
    ]
}

Slack provides a nice demo tool of message formatting. It’s very handy when getting your messages to look just right.

Extending

Next, we may want to send different types of commands to the same app. For instance, I could specify what I want from the weather app, like this:

/weather forecast:tomorrow 83864
/weather temp:f 83864

To do so, you may choose to separate out pieces of the app into separate modules or sub-classes, then dispatch them from one place. In an app I’m working on at Heroku, we pass commands like this:

/slack-app-name task:subtask command

…where slack-app-name is one of a few different Slash Commands apps. In order to properly parse the task, subtask, and command, we have a Processor module:

module Commands
  module Processor
    extend self

    def init(params)
      task = task(params).titleize
      "Commands::#{task}".constantize.new(params)
    end

    # Given `task:sub message`, return task
    def task(params)
      params[:text].scan(/\w+/).first
    end

    # Given `task:sub message`, return subtask
    def subtask(params)
      params[:text].scan(/\w+/)[1]
    end

    # Given `task:sub message`, return message
    def message(params)
      params[:text][/\w+:\w+\s(.*)/, 1]
    end
  end
end

We use this to initialize the correct class to handle the command:

class CommandWorker
  include Sidekiq::Worker
  sidekiq_options :retry => false # job will be discarded immediately if failed

  def perform(params)
    Commands::Processor.init(params).run
  end
end

Now we can send something like /weather forecast:tomorrow 83864 and have a class Commands::Forecast process it by looking at subtask (tomorrow) and message (83864). By doing so we can extend and expand our app to process multiple different tasks.

Conclusion

I hope this gives you a glimpse into what’s possible with Slash Commands! Happy coding, Rubyist!

Added "not_in?" to Rails

2016-07-21T18:01:59+00:00

UPDATE: DHH yanked this patch. Ah well. Open source giveth, and open source taketh away. I sure wish we could restart this discussion and get this change made, but I’ll probably pass on fighting over this one.

Here’s the (now defunct) celebratory post…

While helping a friend today with their Rails project, I found I wanted to add a simple helper method to an object. In the existing view I found something like this:

- if @item.tag != "free" && @item.tag != "extras"

This is a code smell and I could solve it in two ways: either write a view helper or add a helper method to the Item class. In this case, I want to ask the object a question, so this makes sense to me on the object itself.

So first we change the view:

- if @item.for_sale?

And we implement on the model:

class Item < ActiveRecord::Base

  def for_sale?
    self.tag != "free" && self.tag != "extras"
  end

end

Ew. We’re checking a string attribute agaisnt two other strings. Let’s clean that up.

def for_sale?
  !["free", "extras"].include?(tag)
end

We’re basically saying “this item is for sale if the tag isn’t either ‘free’ or ‘extras’”. To shave off some cognitive load, we use Rails’ in? method and flip things around. Since we’re really talking about the tag (and not the array up front), this makes it easier to read.

def for_sale?
  !tag.in?(["free", "extras"])
end

That’s all fine and good, but that bang (!) out front is still making my mind do some gymnastics. Surely, if Rails has in? it has an opposite. But Google gave me nothing, and some other Rails friends I asked hadn’t heard of such a thing.

Turns out, Rails doesn’t have an opposite for in?. Ruby has include? and Rails implements exclude?, which is nice. But Rails implements in? without an opposite.

Putting aside the fact that I think these methods (exclude? and in?) should be added to Ruby core, I would love to see Rails go that extra step and have a not_in? method.

… and now it does!

As of this commit, Rails now has a not_in? method. It works like this:

Replace this:

[1,2].exclude?(user_id)

…into this:

user_id.not_in?([1,2])

So in our previous example, we can now lead with the object in question. So this:

def for_sale?
  !tag.in?(["free", "extras"])
end

… turns into this:

def for_sale?
  tag.not_in?(["free", "extras"])
end

Nice and clean! Enjoy!

Heroku National Park Name Generator

2016-07-05T13:08:54+00:00

In the past, the Heroku API has done some fun stuff on certain holidays. Whenever you run heroku create to make a new app, the API usually generates a random name for your app. But on these holidays (like Halloween), we’ve changed up the API to return some fun names specific to the day.

In celebtration of the National Park Service’s 100th anniversary, the Heroku API did something special on July 4th (American Independence Day) this year. Whenever you ran heroku create, we populated your app’s name with National Park names.

Well, July 4th has come and gone, but the National Park centenenial continues! So here’s a quick script for you to run locally whenever you want to add a new app. Just download and run ruby script.rb and it’ll create a new Heroku app for you.

NATIONAL_PARK_ADJECTIVES = %w[rugged beautiful scenic peaceful pristine majestic
  tranquil historic serene lovely dazzling alluring marvelous stunning gorgeous]

NATIONAL_PARK_NOUNS = %w[acadia arches badlands big-bend biscayne
  gunnison bryce-canyon canyonlands capitol-reef carlsbad-caverns
  channel-islands congaree crater-lake cuyahoga-valley death-valley
  denali-preserve dry-tortugas everglades gates-of-the-arctic glacier-bay
  glacier grand-canyon grand-teton great-basin great-sand-dunes
  great-smoky-mountains guadalupe-mountains haleakala hawaii-volcanoes
  hot-springs isle-royale joshua-tree kenai-fjords kings-canyon kobuk-valley
  lake-clark lassen-volcanic mammoth-cave mesa-verde mount-rainier north-cascades
  olympic petrified-forest redwood rocky-mountain saguaro sequoia shenandoah
  theodore-roosevelt virgin-islands voyageurs wind-cave yellowstone yosemite zion]

system `heroku create "#{NATIONAL_PARK_ADJECTIVES.sample}-#{NATIONAL_PARK_NOUNS.sample}-#{rand(10000..99999)}"`

A few sample names:

historic-lake-clark-26105
rugged-joshua-tree-84943
alluring-grand-canyon-67671
majestic-badlands-30198
tranquil-joshua-tree-92886

Enjoy!!

Letting Users Add Custom Domains On Heroku

2016-04-04T11:50:52+00:00

Consider the following use case: you’d like to run a blogging service on Heroku that allows users to sign up and bring their own custom domain. We’ll call your new idea “Me-dium” (that’s probably not taken).

So you’ve got www.me-dium.com setup on Heroku. And your users automatically get a subdomain like username.me-dium.com. But they also want to bring their own domain and CNAME it to username.me-dium.com.

The Heroku Platform requires that any custom domains be registered with their corresponding app. That way, when a request comes into your custom domain, the Heroku router knows where to send the request. This is a pretty simple process when you have 1 or 2 domains, but when you want to give users the ability to add their own domain, you need a way to tell Heroku about these new domains.

We’ll be adding a simple background worker to interact with the Heroku API. We add a column and form field to the user’s profile for domain. Then, when they change this value, we update the Heroku API with the new value.

Let’s first start by adding the Heroku API gem to our project.

gem 'platform-api'

Next, we need an API token to use with our app.

$ heroku plugins:install git@github.com:heroku/heroku-oauth.git
$ heroku authorizations:create -d "Platform API example token"
Created OAuth authorization.
  ID:          2f01aac0-e9d3-4773-af4e-3e510aa006ca
  Description: Platform API example token
  Scope:       global
  Token:       e7dd6ad7-3c6a-411e-a2be-c9fe52ac7ed2

Add the token to your .env file:

HEROKU_API_KEY=e7dd6ad7-3c6a-411e-a2be-c9fe52ac7ed2

Let’s set up the worker first. Since this is an external API call, we’ll send this work off to a ActiveJob worker.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
require 'platform-api'

class HerokuDomainJob < ActiveJob::Base
  queue_as :default

  def perform(domain, action)
    heroku = PlatformAPI.connect_oauth(ENV['HEROKU_API_KEY'])

    begin
      case action
      when "add"
        heroku.domain.create(MY_HEROKU_APP_NAME, "hostname" => domain)
      when "remove"
        heroku.domain.delete(MY_HEROKU_APP_NAME, domain)
      end

    rescue Heroku::API::Errors::RequestFailed => e
      Rails.logger.error "[Heroku Domain Worker] ERROR: #{e}"
    end
  end

end

So this one worker will take two params: the domain in question, and what to do with it (add or remove it). We’ll trigger this worker using an ActiveRecord callback. This method utilizes dirty tracking and we expect a domain column on the users table.

1
2
3
4
5
6
7
8
9
10
11
12
13
# app/models/user.rb

after_save :update_heroku_domains, if: "domain.present?"

...

private
  def update_heroku_domains
    if self.domain_changed? && self.domain_was.present?
      HerokuDomainJob.perform_later(self.domain_was, "remove")
    end
    HerokuDomainJob.perform_later(self.domain, "add")
  end

First, we check to see if there was an existing value and that it has been changed (line 9). If so, we send off a worker job to remove the old value (line 10). Then, we add the new value (line 12).

That’s it! Your app is now setup to interact with the Heroku API and can easily add/remove domains as needed. Good luck on your new blogging platform!

Copy Heroku Database To Local

2016-03-22T14:49:30+00:00

I’ve had a few Heroku projects where it’s been really nice to copy production data to my local database for development. Obviously, there are a few security concerns with this method (make sure the data in your project should be on your local machine – don’t do this if you’ve got any sensitive data!).

If you’re ok with having that data locally, you can add this script at lib/tasks/database.rake

namespace :db do

  desc "Copy production database to local"
  task :copy_production => :environment do

    # Download latest dump
    system("wget -O tmp/latest.dump `heroku pg:backups public-url -q`")

    # get user and database name
    config   = Rails.configuration.database_configuration["development"]
    database = config["database"]
    user = config["username"]

    # import
    system("pg_restore --verbose --clean --no-acl --no-owner -h localhost -d #{database} #{Rails.root}/tmp/latest.dump")
  end

end

With that in place, you can run the task with

$ rake db:copy_production

This will grab your latest Postgres backup and save it locally to tmp/latest.dump. It then grabs your local DB credentials and performs a pg_restore to restore the dump file to your local database.

Enjoy!

Modify Your Ruby App's DNS Lookup

2016-03-14T08:34:50+00:00

We recently had a Heroku support ticket where a user needed to modify their DNS lookup inside their app:

We’re connecting to an external API that requires a custom DNS entry. Locally, we modify our /etc/hosts file. How can we do this inside our app?

Modifying your /etc/hosts file is a common process for tweaking your local DNS lookup. Your machine will first check this file for any custom entries before proceeding to find the host address on your remote DNS server (usually provided by your ISP).

However, while modifying the /etc/hosts file on a remote server will work as well, this decreases the ability for your app to remain portable. In addition, this may create confusion for other team members that aren’t aware that you’ve modifed this file. Instead, let’s make the change inside our app using the Ruby standard library.

resolv.rb is a pure-Ruby DNS implementation, and it’s actually much better performance-wise when app concurrency grows. The resolv-replace.rb part replaces the default Ruby DNS resolution (e.g. system DNS) with resolv.rb.

To implement, we first require “resolv-replace”, then create a Resolv::Hosts resolver and a regular DNS resolver. That way, the app will will check the hosts first, then fallback to real DNS.

Let’s start by adding our entry to a custom_hosts file within our app:

$ echo 127.0.0.1 my.customserver.com > custom_hosts

You can now test this out in an irb session:

require 'resolv-replace'

hosts_resolver = Resolv::Hosts.new('custom_hosts')
dns_resolver = Resolv::DNS.new

Resolv::DefaultResolver.replace_resolvers([hosts_resolver, dns_resolver])

require "net/http"
require "uri"

You can now test the URL’s and ensure the DNS lookup behaves as necessary:

> Net::HTTP.get_response(URI.parse("https://google.com/"))
=> #<Net::HTTPFound 302 Found readbody=true>
> Net::HTTP.get_response(URI.parse("https://my.customserver.com/"))
Errno::ECONNREFUSED: Failed to open TCP connection to my.customserver.com:80 (Connection refused - connect(2) for "127.0.0.1" port 80)

As you can see above, the DNS for my.customserver.com resolves correctly as 127.0.0.1.

We now have custom DNS information within our app, instead of needing to modify your system’s /etc/hosts file. This allows you to keep this vital information contained within your app and under source control.

NOTE: using resolv.rb can cause some unwanted side effects like this one. Please make sure to test your implementation thoroughly!

Rails' Tagged Logging + Heroku Dyno Metadata

2016-01-23T15:33:27+00:00

Rails added “tagged logging” awhile back, but it’s never seemed like a particularly useful thing to me. I care far more about the log output than a request ID or other data.But the more and more I debug odd edge-cases in other people’s Rails apps, the more I see the need the potential benefits.

“request_id” may be what most people think of with Rails’ tagged logging. Adding this to your production.rb:

config.log_tags += [ :uuid ]

Gives you a unique ID for each request:

[d50fcd66-97c9-4f9e-9a76-d828e1485a8d]
[d50fcd66-97c9-4f9e-9a76-d828e1485a8d]
[d50fcd66-97c9-4f9e-9a76-d828e1485a8d] Started GET "/" for ::1 at 2016-01-13 09:22:54 -0800
[d50fcd66-97c9-4f9e-9a76-d828e1485a8d]   ActiveRecord::SchemaMigration Load (0.5ms)  SELECT "schema_migrations".* FROM "schema_migrations"
[d50fcd66-97c9-4f9e-9a76-d828e1485a8d] Processing by PostsController#index as HTML
[d50fcd66-97c9-4f9e-9a76-d828e1485a8d]   Post Load (0.6ms)  SELECT "posts".* FROM "posts"
[d50fcd66-97c9-4f9e-9a76-d828e1485a8d]   Rendered posts/index.html.erb within layouts/application (20.0ms)
[d50fcd66-97c9-4f9e-9a76-d828e1485a8d] Completed 200 OK in 283ms (Views: 276.8ms | ActiveRecord: 4.4ms)

This can be very handy to use when filtering a noisy log stream. (FYI: logging request ID’s is about to become a default in Rails)

But the full power of this feature is that you can shove whatever data you want in there. UUID, server name, or even a Git SHA.

In a recent Heroku support ticket, a customer mentioned he was using Heroku’s Preboot feature. With Preboot, Heroku will deploy your code to a new dyno, then cut over the requests to it, thus avoiding any request queueing while your app restarts.

The customer wanted to be able to see which version of his code was serving incoming requests at a particular time. This is a great use-case for Rails’ tagged logging, and Heroku’s new “dyno metadata”.

“Dyno metadata” adds a ton of information on your running dyno like: app id, dyno id, release version, and Git SHA. This information is saved as an ENV var for easy reference. Here’s a sampling:

HEROKU_APP_ID:                   9daa2797-e49b-4624-932f-ec3f9688e3da
HEROKU_APP_NAME:                 example-app
HEROKU_DYNO_ID:                  1vac4117-c29f-4312-521e-ba4d8638c1ac
HEROKU_RELEASE_VERSION:          v42
HEROKU_SLUG_COMMIT:              2c3a0b24069af49b3de35b8e8c26765c1dba9ff0
HEROKU_SLUG_DESCRIPTION:         Deploy 2c3a0b2

This feature is currently as “Labs” feature, so add it to your app with this command:

heroku labs:enable runtime-dyno-metadata -a <app_name>

You can then pass the ENV["HEROKU_SLUG_COMMIT"] into your Rails’ logging:

config.log_tags += [ ENV["HEROKU_SLUG_COMMIT"] ]

Now you’ll know exactly which Git version your app is running. Very handy for Preboot or otherwise! (Alternatively, your might choose to use ENV["HEROKU_RELEASE_VERSION"] instead, since the Git SHA length might be bothersome)

2016-01-13T17:41:16.857263+00:00 app[web.1]: [f8ce293494e4fa6634cdb30995bade8817047a7f] Started GET "/" for 70.209.200.115 at 2016-01-13 17:41:16 +0000
2016-01-13T17:41:17.084326+00:00 app[web.1]: [f8ce293494e4fa6634cdb30995bade8817047a7f]   Post Load (6.7ms)  SELECT "posts".* FROM "posts"
2016-01-13T17:41:16.984597+00:00 app[web.1]: [f8ce293494e4fa6634cdb30995bade8817047a7f] Processing by PostsController#index as HTML
2016-01-13T17:41:17.119678+00:00 app[web.1]: [f8ce293494e4fa6634cdb30995bade8817047a7f]   Rendered posts/index.html.erb within layouts/application (108.8ms)
2016-01-13T17:41:17.146112+00:00 app[web.1]: [f8ce293494e4fa6634cdb30995bade8817047a7f] Completed 200 OK in 161ms (Views: 140.3ms | ActiveRecord: 16.5ms)

Here’s an example of one of my Heroku apps cutting over from one version of my code to another. Note the change from adc997c to 7be3dfd:

2016-01-13T17:53:42.704148+00:00 app[web.1]: [adc997cad5e8941b1a085ae3875e2370824cffb6] Started HEAD "/" for 70.209.200.115 at 2016-01-13 17:53:42 +0000
2016-01-13T17:53:42.711699+00:00 app[web.1]: [adc997cad5e8941b1a085ae3875e2370824cffb6]   Post Load (0.9ms)  SELECT "posts".* FROM "posts"
2016-01-13T17:53:42.713734+00:00 app[web.1]: [adc997cad5e8941b1a085ae3875e2370824cffb6] Completed 200 OK in 5ms (Views: 3.0ms | ActiveRecord: 0.9ms)
2016-01-13T17:53:42.708938+00:00 app[web.1]: [adc997cad5e8941b1a085ae3875e2370824cffb6] Processing by PostsController#index as */*
2016-01-13T17:53:42.712343+00:00 app[web.1]: [adc997cad5e8941b1a085ae3875e2370824cffb6]   Rendered posts/index.html.erb within layouts/application (2.1ms)
2016-01-13T17:53:45.379444+00:00 app[web.1]: [7be3dfd2f62e3a876d74212be6a7c9dab8128f1d] Started HEAD "/" for 70.209.200.115 at 2016-01-13 17:53:45 +0000
2016-01-13T17:53:45.477517+00:00 app[web.1]: [7be3dfd2f62e3a876d74212be6a7c9dab8128f1d]   Post Load (3.0ms)  SELECT "posts".* FROM "posts"
2016-01-13T17:53:45.495110+00:00 app[web.1]: [7be3dfd2f62e3a876d74212be6a7c9dab8128f1d] Completed 200 OK in 43ms (Views: 29.6ms | ActiveRecord: 7.3ms)
2016-01-13T17:53:45.498012+00:00 heroku[router]: at=info method=HEAD path="/" host=whispering-eyrie-6721.herokuapp.com request_id=11b40ae8-2cf4-4915-89af-0cecad83005a fwd="70.209.200.115" dyno=web.1 connect=1ms service=123ms status=200 bytes=684
2016-01-13T17:53:45.452038+00:00 app[web.1]: [7be3dfd2f62e3a876d74212be6a7c9dab8128f1d] Processing by PostsController#index as */*
2016-01-13T17:53:45.492969+00:00 app[web.1]: [7be3dfd2f62e3a876d74212be6a7c9dab8128f1d]   Rendered posts/index.html.erb within layouts/application (31.9ms)
2016-01-13T17:53:48.316089+00:00 heroku[router]: at=info method=HEAD path="/" host=whispering-eyrie-6721.herokuapp.com request_id=3f08fd7d-4b40-4c3f-8f25-3f245a1e3188 fwd="70.209.200.115" dyno=web.1 connect=3ms service=9ms status=200 bytes=684
2016-01-13T17:53:48.309340+00:00 app[web.1]: [7be3dfd2f62e3a876d74212be6a7c9dab8128f1d] Started HEAD "/" for 70.209.200.115 at 2016-01-13 17:53:48 +0000
2016-01-13T17:53:48.315024+00:00 app[web.1]: [7be3dfd2f62e3a876d74212be6a7c9dab8128f1d]   Post Load (1.7ms)  SELECT "posts".* FROM "posts"
2016-01-13T17:53:48.316491+00:00 app[web.1]: [7be3dfd2f62e3a876d74212be6a7c9dab8128f1d] Completed 200 OK in 4ms (Views: 2.2ms | ActiveRecord: 1.7ms)
2016-01-13T17:53:48.312092+00:00 app[web.1]: [7be3dfd2f62e3a876d74212be6a7c9dab8128f1d] Processing by PostsController#index as */*
2016-01-13T17:53:48.315465+00:00 app[web.1]: [7be3dfd2f62e3a876d74212be6a7c9dab8128f1d]   Rendered posts/index.html.erb within layouts/application (2.7ms)

Tagged Logging is an incredibly powerful feature in Rails that gets very little attention. Here’s a few other articles I found that give it a more in-depth explanation. Enjoy!

Removing Distractions For Your Team

2015-11-24T10:48:11+00:00

I’ve known far too many people who reach the coveted level of “middle management” and believe they’ve finally reached Easy Street. “Time to sit back and let the minions do my bidding!” Ok, I’m sure that’s not exactly verbalized, but delegation is easily abused by middle managers who think their primary job is to ensure everyone is typing away happily all day. (these are all the same managers you find tapping their watch when you come in too late or leave too early)

On the contrary, great managers accept the responsibility of what it is to lead. And great leaders don’t bark out orders and head out for a long lunch break. Great managers help their team by serving them.

The thing to remember here is that moving into management usually means you’re no longer a direct producer. Your coding/designing/selling skills are usually taking a back seat to your newly found management skills. So you must realize that your team is now the only ones capable of producing a win. The prevailing thought cannot be “Who can I get to do this work for me?” but instead should be “My job is to make sure my team members can succeed.”

The best way to make sure your team succeeds is to ensure they’re productive. So you must keep an eye out for things that destroy productivity: useless meetings, directional ambiguity, and interruptions.

Useless Meetings. For some reason, middle managers seem to think that creating and attending meetings is their gift to the team. And yet, there are countless studies that prove that meetings destroy productivity. Yes, there are some meetings that must be had, but please stop thinking that everything must be done synchronously. Most times, the discussion you’re wanting to have can be done over email.
Directional Ambiguity. When the team doesn’t know what’s expected or where they’re headed, they will attempt to fill in the gap. This leads to team members going “off the reservation”, building features that aren’t needed, and needing to re-build things because someone up the chain has changed their mind (again). Make a plan. Work the plan. Remind your team what the plan is and do it frequently.
Interruptions. There’s no bigger productivity killer than a distracting work environment. Most people think music in the office is the main offender, but sometimes the worst distraction is you. You walk around the office during peak productivity times, stopping by desks to smile and shake hands … to “see what you’re working on”. And your team will stop what they’re doing, take off their headphones, and engage. But you’ve just cost them their focus. Instead, leave “check-ins” for stand-ups or 1-on-1’s. Say hello in the morning while getting coffee, but fight the urge to constantly interrupt your team throughout the day.

Like a good Scrum Master, a great manager excels at removing obstacles for their team. Cut through the red tape, get the answers your team needs, and do your very best to provide a distraction-free work environment. Your team will thank you.

Measure, Release, Measure Again

2015-11-03T13:16:15+00:00

In the world of product development, there is far too little mentioned about the need and use of data in decision making. We’re easily pleased with ideas that seem logical to us, but may be terrible for the users of our product. We add new features, but we never check to see if the investment we made (in resources and hours spent) have actually paid off.

Instead of assuming, let’s take a look at how data can be easily gathered and utilized to inform our decision making process.

The Problem

“Why aren’t we gaining more users?” someone asked at our monthly meeting with management. “Do we need to spend more money on marketing?”

A simple “yeah, sure” would have been easier for me. But blowing tons of money on marketing usually masks other issues. Sure, it has its place, but why invest that kind of money if another issue is the root of the problem.

“I don’t know,” I said. (the 3 hardest words in the English language) But I said it with confidence. I honestly had not yet considered this question and knew I didn’t have data one way or the other. But I assured management I would spend the week trying to see if we had a leaky funnel somewhere – were users not coming to the site? Or were they coming but not signing up?

Google Analytics was an easy tool to reach for here. I spent some time digging through user patterns – seeing where they went after hitting the homepage. After only about an hour, I found the problem: a 65% conversion rate on our signup page. I knew we could do better.

Hypothesis

Now that we had a baseline set of data, we could begin to experiment. The first thought from our team was that the form was too long. We had our entire terms of service document as an iframe that was visible before clicking the submit button.

“Let’s just chop off that entire section and place a link to our TOS instead.” But we had another issue at hand – we wanted to capture more information about the user on sign up. So for our first test, we decided to split the difference: we added a few more fields, but we completely removed the TOS iframe.

I certainly had my doubts about this theory. But that’s just it, they were my doubts. It’s easy to sit in a meeting and argue about subjectivity. Making product decisions from the gut can be a great move, but you have to remember that it’s impossible for you to fully and accurately represent your entire user base.

After a week, we checked the numbers. Our conversion rate had dropped from 65% to 57%. The addition of the extra fields were scaring off more people than the TOS iframe had been. So we implemented another change: a simpler form to validate the theory that our original form was too intimidating. Our form needed to say: “Just give us the bare mimimum and get in here!” So we pulled it all back and had the form only ask the bare minimum to use the site successfully.

Lo and behold, our numbers jumped to 71% the very next week. We continued for 2 more weeks to make changes that eeked out another 4% to get us to a respectable 75% conversion. The need for the extra user info was still there – we still wanted to capture that data somehow. But we all agreed that increasing our conversion was more critical than getting that extra data at this step.

Next Steps

After seeing how easy it was to perform these tests, we feel in love with Optimizely – a great tool made for A/B testing. We looked for other ways to make small but meaningful changes to our marketing sites: smaller headers, larger headers, different copy in the navbar, different colored buttons. All of these were small changes to make, but we saw huge improvements in our metrics.

For you, it may not be a marketing homepage. Acquisition may not be the target – maybe it’s retention. So where’s the drop-off? Why are customers leaving? What pain points might your users be having that they’re not telling you about, but your data can reveal to you? Use the data you have – or create more through logging – but make your data work for you. Invest your time and money in features that you know will pay off; not just ones that “feel” right, but ones backed by solid data.

Delegation in Rails

2015-10-27T16:22:19+00:00

After spending a year or two in Rails, I started to plateau a bit. I knew my code wasn’t clean, but didn’t know how to make it better. I began to ask more-senior Rails programmers about thoughts and ideas that helped them improve.

One of the more influential lessons I learned was about the Law of Demeter (and loose coupling). Let’s start with defining what it is, then anti-patterns, and an example on how Rails makes it easier to avoid some costly technical debt.

Law of Demeter

Here’s the best definition I’ve found for the Law of Demeter: “Principle of Least Knowledge: Only talk to your friends”

Basically, we want our models to have very little knowledge of the models it interacts with. “Loose coupling” means we can swap out models and/or make changes without requiring a complete re-write. You’ll see a bit later what can happen when your models are tightly coupled, but for now, let’s look at this bad example.

Example Anti-Pattern

@post = Post.find(params[:id])
@post_date = @post.body.posted_on

In this (contrived) example, we’re wanting to get the date for a blog post. We have a relationship between post and a body, like this:

class Post < ActiveRecord::Base
  has_one :body
end

class Body < ActiveRecord::Base
  belongs_to :post
end

So the Post’s body has the date we want, but we have to hop through Body to get there. This isn’t bad now, but will get bad once we try to refactor this. This is an example of tight coupling.

Ever seen (or written) something like this?

@address = @invoice.client.preferences.billing_address

We’re taking the @invoice we have, but we’re storing an address on the client, perhaps in a preferences hash. If we ever want to tweak anything in the middle, we’re gonna have a bad time.

Delegate

Rails (via ActiveSupport) has a great little helper to solve our problem: delegate. Let’s see how we fix the “post” example above:

class Post < ActiveRecord::Base
  has_one :body
  delegate :posted_on, to: :body
end

That’s it! Now we can call posted_on directly on our @post instance. Sprinkle @post.posted_on wherever you need it, and if ever the delegation changes, just change the implementation in one spot. Much better than a massive find/replace on @post.body.posted_on

Here’s how we solve the other example:

class Invoice < ActiveRecord::Base
  belongs_to :client
  delegate :billing_address, to: :client
end

class Client < ActiveRecord::Base
  has_many :invoices

  def billing_address
    preferences.fetch(:billing_address, nil)
  end
end

Now we can call @invoice.billing_address and Rails will handle the details.

Thinking Ahead

So how do you know when to implement delegate in your own project? What’s the “code smell”?

The best way to catch it in the act it to be on the look out for multiple “dots”. If your object needs to reach across more than one other object to get its data, you’re probably violating the Law of Demeter. Instead, start by writing the method as you want it to be without worrying about implementation first.

For instance, if you wrote @post.body.posted_on, an alarm should go off in your head: “Why am I asking another object for data that makes perfect sense for me to know myself?” Instead, write @post.posted_on, then move down to the model layer to implement.

@invoice.client.preferences... “OH LAWD! No good.” @invoice.billing_address – “Ahhhh. Much better.”

The more you get in the habbit of smelling out bad code like this, the faster you’ll get at implementing them right the first time. Staying ahead of technical debt like this will save you hours (sometimes days) of headache in the future.

Happy delegating!

Default UUID's In Rails

2015-10-20T17:27:56+00:00

🎉 I added a new feature to Rails: Default to uuid as primary key when generating database migrations. Enjoy!

UUID’s

I love UUID’s. You should use them. Rails 4 makes it simple to get setup and use UUID’s throughout your project.

First, we enable UUID’s:

rails g migration enable_uuid_extension

This creates the following migration:

1
2
3
4
5
class EnableUuidExtension < ActiveRecord::Migration
  def change
    enable_extension 'pgcrypto'
  end
end

Next, create a model

rails g model Book title

The Problem

Unfortunately, this creates an integer-based primary key migration:

1
2
3
4
5
6
7
8
class CreateBooks < ActiveRecord::Migration
  def change
    create_table :books do |t|
      t.string :title
      t.timestamps
    end
  end
end

To move forward, you must fiddle with that migration to add id: :uuid to the create_table method:

1
2
3
4
5
6
7
8
class CreateBooks < ActiveRecord::Migration
  def change
    create_table :books, id: :uuid do |t|
      t.string :title
      t.timestamps
    end
  end
end

After spending a few weeks on a new project doing this, I figured we could make a change to Rails to allow this to happen. Here’s how.

The Solution

I recently had a change merged into Rails you should know about. Here’s a look at the original commit, then a follow-up to make a few minor modifications.

In application.rb, simply make the following addition:

config.generators do |g|
  g.orm :active_record, primary_key_type: :uuid
end

Now, whenever you generate a migration, we’ll tag on id: :uuid to the create_table method.

Warning

You must have already added a UUID extension to your database. If not, running this migration will fail. So don’t forget the rails g migration enable_uuid_extension migration up front.

Enjoy!

SMS Verification With Rails

2015-10-20T14:27:56+00:00

While working recently on a side project, I came across the task of “SMS validation”. The project allows users to sign up with their mobile phone number, and have certain text messages sent to their phone on a schedule they determine. Here’s how the feature request came in:

As a user, I need my phone number verified before being allowed to send out SMS messages to myself

Database

First off, we need to figure out what part in our sign-up process this shows up in. We don’t want the user to be able to create any SMS jobs without making sure their phone number is a) valid, and b) their own. So let’s do two things:

Make sure the user is marked as “unverified” until they confirm their number
Ensure they are not able to leave the sign up process and return to subvert this security check

So we start with the basics: the database. We know we’ll want the “verified” boolean, but we’ll also want two other bits of data: storing the pin andwhen was the pin sent at (so we disallow old pins from being used).

$ rails generate migration add_verified_to_users pin:integer pin_sent_at:datetime verified:boolean

Then we modify the resulting migration to set the default for “verified”:

1
2
3
4
5
6
7
class AddPinAndVerifiedToUsers < ActiveRecord::Migration
  def change
    add_column :users, :pin, :integer
    add_column :users, :pin_sent_at, :datetime
    add_column :users, :verified, :boolean, default: false
  end
end

Controller

Now that we have our data fields, let’s make sure our controllers route the user to the correct place based on whether or not they’ve been verified. First, let’s look at applciation_controller.rb:

1
2
3
4
5
6
7
before_filter :redirect_if_unverified

def redirect_if_unverified
  if logged_in? && !current_user.verified?
    redirect_to new_verify_url, notice: "Please verify your phone number"
  end
end

For boolean fields, Rails provides a handy ? method on the column name. So if you’re boolean was awesome, you now have user.awesome? availabe to you. So we tap into that and check to see if the user has yet been verified. If not, we redirect them to our new_verify_url. Here’s the routing for that:

1
2
3
get "/verify" => 'verify#new', as: 'new_verify'
put '/verify' => 'verify#update', as: 'verify'
post '/verify' => 'verify#create', as: 'resend_verify'

NOTE: you can create similar routing with resource, but I wanted to name the POST method as resend_verify. We’ll see that next.

We’re still missing two things at the controller level:

Making sure we visit new_verify_url after sign up, and
The verification form

Here’s the basics of users_controller.rb:

1
2
3
4
5
6
7
8
9
10
def create
  @user = User.new(user_params)

  if @user.save
    auto_login(@user)
    redirect_to new_verify_url
  else
    render :new
  end
end

We’re simply changing the redirect after user creation from something like redirect_to root_url to redirect_to new_verify_url.

Next, we create a separate controller for verification. It’s temping here to shove all of this logic into users_controller.rb, but resist the urge. The behavior of verification is its own thing, and as such, belongs in its own controller (single responsibility principle).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
class VerifyController < ApplicationController
  skip_before_filter :redirect_if_unverified

  # GET /verify
  def new
  end

  # PUT /verify
  def update
    if Time.now > current_user.pin_sent_at.advance(minutes: 60)
      flash.now[:alert] = "Your pin has expired. Please request another."
      render :new and return
    elsif params[:pin].try(:to_i) == current_user.pin
      current_user.update_attribute(:verified, true)
      redirect_to root_url, notice: "Your phone number has been verified!"
    else
      flash.now[:alert] = "The code you entered is invalid."
      render :new
    end
  end

  # POST /verify
  def create
    current_user.send_pin!
    redirect_to new_verify_url, notice: "A PIN number has been sent to your phone"
  end

end

First off, we make sure we skip our redirect_if_unverified filter or else we’ll end up in a redirect loop. def new is the basics of our form. def update is the business.

In this method, we’re checking to make sure that the user’s pin was sent less than an hour ago. We return early from this method since this is a non-starter for us. Next, we check to see if the pin exists (using try) and that it matches the user’s pin. If not, we return an error immediately.

You could probably rework this method a few different ways, but I’m happy with how readable it is. Having the last else block also allows for a clean “fall-through” in case we add other validation checks above it.

Finally, def create will be used for a “Send me another PIN” link in the view:

1
<%= link_to 'Resend code', resend_verify_path, method: :post %>

Model

For the User model, we want to make sure we cover the following:

Set the PIN upon user creation
Create a method where the PIN can be reset
Fire off a background job to send out the SMS to the user

For number 1, let’s use ActiveRecor’s callbacks to fire a method when we create a user:

1
after_save :send_pin!, if: "phone_number_changed?"

Next, let’s create two helper methods and the send_pin! method.

1
2
3
4
5
6
7
8
9
10
11
12
13
def reset_pin!
  self.update_column(:pin, rand(1000..9999))
end

def unverify!
  self.update_column(:verified, false)
end

def send_pin!
  reset_pin!
  unverify!
  SendPinJob.perform_later(self)
end

Whenever we send the PIN, we need to make sure we update the pin column and reset the user to be unverified. send_pin! relies the other two helper methods to do these tasks, then enqueues a background job to send out the SMS message.

“You really shouldn’t enqueue a background job with an object. Use an ID.” Meh. Rails has a great new feature called “Global ID” which allows you to enqueue jobs with the ActiveRecord object itself.

“But what about pin_sent_at?” Well, let’s let our upcoming background job handle this since we don’t want the model setting this in case there’s a delay in actually sending out the job.

Background Job

Easy peasy.

1
2
3
4
5
6
7
8
9
class SendPinJob < ActiveJob::Base

  def perform(user)
    nexmo = Nexmo::Client.new(key: ENV["NEXMO_KEY"], secret: ENV["NEXMO_SECRET"])
    resp = nexmo.send_2fa_message(to: user.phone_number, pin: user.pin)
    user.touch(:pin_sent_at)
  end

end

We’re using Nexmo here, but regardless of the SMS provider, the basics are the same: send the message and update pin_sent_at. Rails’ touch method is a good fit here.

Note that I’m assiging the response from the Nexmo API as resp but not doing anything with it yet. One of my next TODO’s is to only touch pin_sent_at if the response is successful.

Re-Send SMS

So what happens if the user doesn’t get their PIN or if they’re too late in entering it? No problem! As seen earlier, the “Resend Code” link makes a POST to the verify_controller.rb, which re-uses our send_pin! method:

1
2
3
4
def create
  current_user.send_pin!
  redirect_to new_verify_url, notice: "A PIN number has been sent to your phone"
end

This will then ensure the user is unverified and is sent a new pin.

Conclusion

Overall, I really enjoyed working on this task. It took awhile to make sure we covered all our use-cases (user leaving mid-verification, expired PIN, re-send, etc), but the process is seamless now. From a security standpoint, creating a 4-digit pin is not the most secure thing here. However, the user must already be logged in to verify their phone, so we’re relying on our session security already. Furthermore, we’re just checking to make sure the phone number was entered correctly so that we’re not sending out messages to bad numbers.

I hope you enjoyed this exercise. Have you ever implemented something similar? See any changes I could make in the samples above? Let me know!

User Time Zone On Signup

2014-10-22T00:00:00+00:00

Knowing a user’s time zone can be incredibly important to your application. At SproutMark we utilize this information in numerous places to enhance the user’s experience.

But it’s difficult to know when to ask for it. If you ask during sign up, you expand the signup form an can reduce your signup’s because of the increased barrier to entry. If you wait until after sign up, you create an experience for the user that’s less than ideal.

Instead, let’s determine the user’s time zone based on their IP. We can set a sane default for the user and then allow them to change it. Let’s take a look at how we can do that with rails.

(I’ll be taking you through a pure Ruby implementation first … then the final JS-only version at the end)

The Guts

We’re going to use a combo of two gem’s to determine a user’s time zone.

# Gemfile
gem 'geocoder'
gem 'timezone'

We’ll use the Geocoder gem to turn our user’s IP into coordinates. Now Geocoder is awesome, but it doesn’t give us any time zone data. So we’ll then need to pass these lat/long coordinates to something else to get the time zone. Enter the Time Zone gem. (Clever name, eh?)

module SignupHelper

  def time_zone_from_ip
    begin
      ip_address = Rails.env.production? ? request.ip : "108.245.160.41"  # In development, our IP will be 127.0.0.1 ... not useful
      geocode = Geocoder.search(ip_address).first
      Timezone::Zone.new(latlon: [geocode.latitude, geocode.longitude]).active_support_time_zone
    rescue Exception => e
      Rails.logger.error "ERROR WITH GEOCODING: #{e.message}"
      "UTC"
    end
  end

end

…and the view…

.form-group
  = f.label :time_zone, "Time Zone"
  = f.collection_select :time_zone, ActiveSupport::TimeZone.all, :name, :to_s, { selected: f.object.time_zone || time_zone_from_ip }, class: "chosen-select"

In short, we’re defaulting to the object’s value (nil by default, or set in case this form was submitted with other errors). If the value is nil, we’re calling our helper to determine the user’s time zone.

This is pretty slick, but I’m not too happy with how this form appears to the user. Even though we’re setting their default and the user can easily skip this select box, it’s mentally taxing to the user to see a form field and process whether or not they need to change the value. How can we show the default, but still allow the user to change it? Let’s make it a link that toggles the select box.

.form-group
  = f.label :time_zone, "Time Zone"

  p= link_to f.object.time_zone || "UTC", "javascript:;", data: { "time-zone" => "toggle" }

  .hide
    = f.collection_select :time_zone, ActiveSupport::TimeZone.all, :name, :to_s, { selected: f.object.time_zone }, class: "chosen-select"

Now for the JS. When a user clicks the link, we’ll hide it and then display the select box.

$("a[data-time-zone='toggle']").click ->
    $(@).hide()
    $(@).parent().next(".hide").removeClass("hide")
    $(".chosen-container").css("width", "340px")

Non-Blocking

One more problem we’ve got: a blocking HTTP request inside our signup form. In order to have the best chance at converting a visitor, we need this page to load FAST. And waiting for this IP check is hurting us.

Instead of making the form wait on this request, we can background it by loading the form with sane defaults, then making another AJAX request back to the server for the time zone info.

# SignupController.rb

  def determine_time_zone
    render json: { time_zone: time_zone_from_ip }
  end

private
  def time_zone_from_ip
    begin
      ip_address = Rails.env.production? ? request.ip : "108.245.160.41"
      geocode = Geocoder.search(ip_address).first
      Timezone::Zone.new(latlon: [geocode.latitude, geocode.longitude]).active_support_time_zone if geocode
    rescue Exception => e
      Rails.logger.error "ERROR WITH GEOCODING: #{e.message}"
      "UTC"
    end
  end

And now the JS. If our new div ID is on the page, let’s use jQuery’s getJSON to fetch this user’s time zone as JSON. We’ll then update the default link text, show the link, update our select box, and reset Chosen.

if $("#signup_time_zone_group").length > 0
  $.getJSON "/determine_time_zone", (json) ->
    $("#signup_time_zone_group").find("p a").html(json.time_zone)
    $("#signup_time_zone_group").removeClass("hide")
    $("#signup_time_zone").val(json.time_zone)
    $("#signup_time_zone").trigger("chosen:updated")
    return

There’s Got To Be A Better Way

After having this implementation in production for a few days, I wrote up this post touting how simple this Ruby implementation was, and how terribly awful trying to do this in pure JS was. Then, magically, a simpler solution showed itself.

While researching a JS version, I ran across jstz – a little library that grabs a user’s time zone from their browser. The problem with this was the fact that the library returns a IANA zone info key (aka the Olson time zone database). This is not the same key that Rails uses. I scrapped this route based on this information.

However, today I managed to find a neat little piece of JS that converts from IANA to the Rails time zones: rails-timezone-js. I dropped in this converter along with jstz, and we have a much simpler solution:

  if $("#signup_time_zone_group").length > 0
    rails_timezone = window.RailsTimeZone.to(jstz.determine().name())
    $("#signup_time_zone_group").find("p a").html(rails_timezone)
    $("#signup_time_zone_group").removeClass("hide")
    $("#signup_time_zone").val(rails_timezone)
    $("#signup_time_zone").trigger("chosen:updated")

You’re notice there’s now no AJAX call back to the server. We use jstz to grab the time zone name, then pass it off to the converter helper method: rails_timezone = window.RailsTimeZone.to(jstz.determine().name()). BOOM!

Conclusion

After getting stuck on a pure JS version, I went down the path of using Ruby and a little AJAX… but then managed to find a solution to the JS version. Regardless of how the time zone was determined, the more interesting part of this process for me was the UX.

My first version didn’t solve the main problem of reducing cognitive load. So I hid the dropdown. This solved the user-facing issue, but the process of determining the user’s lat/lng was creating slow load times. We fixed this by “backgrounding” this part of the process, but that left us with a lot of Ruby for a problem that I was sure had to have been solved before. Sure enough, rails-timezone-js to the rescue!

To see this in action, view SproutMark’s signup form. If you have any tips, let me know in the comments!

Much thanks to Jeff Berg for helping me with this. He and his team at Planning Center use a similar implementation.

Digital Ocean: Ubuntu, Nginx, Unicorn, Rails

2014-08-28T00:00:00+00:00

UPDATE

I ran this setup from August 2014 to December 2014. It served me pretty well. However, I’ve since moved it back to Heroku.

Honestly, the setup was fun and I certainly enjoyed the smaller bill, but keeping up with security patches, the cost of server monitoring, worrying about firewalls, etc. just got to be a pain.

That being said, if you’re into the Ops side of things, I hope this post serves you well.

I recently decided to move SproutMark off to a VPS. I’d heard great things about Digital Ocean, so decided to make the switch. Although there are a handful of tutorials out there (this one was VERY helpful) on how to setup the Nginx/Unicorn/Rails stack, none had everything I needed. So here are my learnings and experience on how I got set up on Digital Ocean.

Here’s what we’ll be covering

Ubuntu
Nginx
Unicorn
Rbenv
Rails 4.1
PostgreSQL
Redis
Memcache
Mina (for deployments)

Get That VPS Built

Sign Up for Digital Ocean
Click “Create Droplet”
Choose your size. A typical Rails app can run about 150-200mb per process. Multiply that by how many Unicorn processes you want to run to figure out how much RAM you need. For my little app, I’m running 2 unicorns + Sidekiq and am hovering around 800mb so the 1GB plan works fine for me.
Under “Select Image”, choose “Ubuntu 14.04 x64”
Click “Create Droplet”

That’s it? Yes. Well, yeah, so far.

Set Up A User For Deployments

It’s usually not good practice to use your root user to handle deployments, so let’s first start by adding a “deployer” user.

$ ssh root@YOUR_NEW_DROPLET
$ sudo adduser deployer
$ su deployer

SSH Keys

Stop using that root password. Setup and upload an SSH key.

Digital Ocean has a fine document on creating a SSH key and getting it installed. The only part I’d never seen before was ssh-copy-id. It’s quite handy and easy to install with Homebrew.

Rbenv

We all have fond memories of RVM. Well, I have some fond memories, but mostly memories or RVM acting crazy and putting crap all over my system. “Is there a better way?”, you ask. Yes. Yes, there is. Please use Rbenv. Let’s get it installed for your deployer user.

First, some dependencies:

$ sudo apt-get update
$ sudo apt-get install curl git-core build-essential zlib1g-dev libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libcurl4-openssl-dev libxml2-dev libxslt1-dev python-software-properties

Now, we’ll install rbenv into your home directory and add some commands to your .bashrc for completution and shims.

$ git clone https://github.com/sstephenson/rbenv.git ~/.rbenv
$ echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
$ echo 'eval "$(rbenv init -)"' >> ~/.bashrc

Now, let’s restart the shell and make sure Rbenv is install:

$ exec $SHELL
$ type rbenv
#=> "rbenv is a function"

Time to install Ruby!

$ git clone https://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
$ rbenv install 2.1.2
$ rbenv global 2.1.2

This part may take awhile. Go grab an apple.

Ok, done?

Let’s make sure all is well…

$ ruby -v
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]

If it’s not working, you’ll probably see this:

$ ruby -v
The program 'ruby' can be found in the following packages:
 * ruby
 * ruby1.8
Try: apt-get install <selected package>

^ That’s not good. Re-visit the instructions above and don’t proceed until you get Ubuntu figuring out where Ruby is.

One last step that always trips people up: install Bundler real quick.

$ gem install bundler

You’ll thank me later…

PostgreSQL

Use MySQL if you must, but I ain’t helping you there. Let’s set up PostgreSQL.

Don’t try and get fancy with other DB users. Just use the built-in ‘postgres’ user.

$ sudo apt-get install postgresql postgresql-contrib
$ createuser --pwprompt

Create your database:

$ su postgres
$ psql
psql (9.3.5)
Type "help" for help.

postgres=# CREATE DATABASE yourapp_production;

Memcache, and Redis

These two are a piece of cake. Thanks, ‘apt-get’

Memcache

$ sudo apt-get install memcached

And Redis (for Sidekiq and other fun stuff)

$ apt-get install redis-server

Boot ‘er up and make sure you can get to the Redis console:

$ redis-server /etc/redis/redis.conf
$ redis-cli
127.0.0.1:6379>

If you’re just using Redis for Sidekiq, you’re done. However, Redis is a pretty powerful datastore. For more, check out Redis in Action.

Nginx

Here’s the part where I started getting scared. I don’t know Nginx well. There. I said it. I’m not ashamed. I got a big head start using Esther Hazzard-Strong’s post, but still had some problems. Here’s the config I ended up with:

Install Nginx:

$ sudo apt-get install nginx
$ sudo service nginx start

You should now be able to view Nginx’s welcome page by visiting your Droplet’s IP in your browser.

You can also check to make sure Nginx is running:

$ ps ax | grep nginx

Now we can get Nginx set up to point to our (upcoming) Rails install. Let’s open up the nginx.conf file:

nano /etc/nginx/nginx.conf

Here are my settings:

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
  worker_connections 768;
}

http {

  ##
  # Basic Settings
  ##

  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;
  keepalive_timeout 65;
  types_hash_max_size 2048;

  server_name_in_redirect off;

  include /etc/nginx/mime.types;
  default_type application/octet-stream;

  ##
  # Logging Settings
  ##

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  ##
  # Gzip Settings
  ##

  gzip on;
  gzip_disable "msie6";

  ##
  # Virtual Host Configs
  ##

  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

So that’s the overall Nginx setup. Now let’s set up the conf file for our Rails site. The cheater way to do this is place a file directly into /etc/nginx/sites-enabled. The better way is to place the file into sites-available and then symlink into sites-enabled.

But I’m a cheater…

nano /etc/nginx/sites-enabled/default

Here’s my setup. I’m running SSL, so there are two server blocks. The “listen 80” block takes any non-http request and redirects to the other server block. The second block is for “listen 443” (SSL). If you don’t want/need SSL, you can remove the first server block and swap out 443 to 80 in the second block. You’ll want to remove the ssl_certificate lines, too.

upstream app {
    # Path to Unicorn SOCK file, as defined previously
    server unix:/home/deployer/YOUR_APP_NAME/shared/sockets/unicorn.sock fail_timeo
ut=0;
}

server {
    listen         80;
    return 301 https://$host$request_uri;
}

server {

    listen 443;

    # Application root, as defined previously
    root /home/deployer/YOUR_APP_NAME/current/public;

    ssl on;
    ssl_certificate /etc/nginx/ssl/SSL.crt;
    ssl_certificate_key /etc/nginx/ssl/YOUR_APP_NAME.key;

    server_name www.YOUR_APP_NAME.com YOUR_APP_NAME.com;

    try_files $uri/index.html $uri @app;

    access_log /var/log/nginx/YOUR_APP_NAME_access.log combined;
    error_log /var/log/nginx/YOUR_APP_NAME_error.log;

    location @app {
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass https://app;
        proxy_set_header   X-Forwarded-Proto https;  # <-- don't need this if you're not running SSL
    }

    error_page 500 502 503 504 /500.html;
    client_max_body_size 4G;
    keepalive_timeout 10;
}

I probably spent more time on that one file above anything. Thank you, internet, for providing tons of random tutorials on setting that up.

Ok, so Nginx is set up. Time to get into the fun stuff.

If you’re looking for a deep dive into Nginx, check out Mastering Nginx.

Unicorn

This was another one that took me awhile, so I’m excited to share this setup with someone. I’m going to assume you know what Unicorn is and why you should use it. Here’s my unicorn.rb Rails initializer (config/unicorn.rb)

# Set your full path to application.
app_dir = File.expand_path('../../', __FILE__)
shared_dir = File.expand_path('../../../shared/', __FILE__)

# Set unicorn options
worker_processes 2
preload_app true
timeout 30

# Fill path to your app
working_directory app_dir

# Set up socket location
listen "#{shared_dir}/sockets/unicorn.sock", :backlog => 64

# Loging
stderr_path "#{shared_dir}/log/unicorn.stderr.log"
stdout_path "#{shared_dir}/log/unicorn.stdout.log"

# Set master PID location
pid "#{shared_dir}/pids/unicorn.pid"

before_fork do |server, worker|
  defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect!
  old_pid = "#{server.config[:pid]}.oldbin"
  if File.exists?(old_pid) && server.pid != old_pid
    begin
      sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
      Process.kill(sig, File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

after_fork do |server, worker|
  defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection
end

before_exec do |server|
  ENV['BUNDLE_GEMFILE'] = "#{app_dir}/Gemfile"
end

Mina

If you’ve ever struggled with the Capistrano DSL, then you’ll be quite happy when you see Mina. It’s quite lovely. And VERY fast.

Add it to your Gemfile.

gem 'mina'

Let’s also add two other gems that will help us manage Unicorn and Sidekiq

gem 'mina-sidekiq', :require => false
gem 'mina-unicorn', :require => false

Create the necessary “deploy.rb”:

$ mina init
Created config/deploy.rb.

Make sure to add your project’s folder inside ‘/home/deployer’. For instance, ‘/home/deployer/YOUR_APP’.

Let’s tweak that deploy.rb file. Here’s my setup:

require 'mina/bundler'
require 'mina/rails'
require 'mina/git'
require 'mina/rbenv'
require 'mina_sidekiq/tasks'
require 'mina/unicorn'

# Basic settings:
#   domain       - The hostname to SSH to.
#   deploy_to    - Path to deploy into.
#   repository   - Git repo to clone from. (needed by mina/git)
#   branch       - Branch name to deploy. (needed by mina/git)

set :domain, 'YOUR DROPLETS IP'
set :deploy_to, '/home/deployer/YOUR_APP/'
set :repository, 'YOUR GIT REPO URL'
set :branch, 'master'
set :user, 'deployer'
set :forward_agent, true
set :port, '22'
set :unicorn_pid, "#{deploy_to}/shared/pids/unicorn.pid"

# Manually create these paths in shared/ (eg: shared/config/database.yml) in your server.
# They will be linked in the 'deploy:link_shared_paths' step.
set :shared_paths, ['config/database.yml', 'log', 'config/secrets.yml']


# This task is the environment that is loaded for most commands, such as
# `mina deploy` or `mina rake`.
task :environment do
  queue %{
echo "-----> Loading environment"
#{echo_cmd %[source ~/.bashrc]}
}
  invoke :'rbenv:load'
  # If you're using rbenv, use this to load the rbenv environment.
  # Be sure to commit your .rbenv-version to your repository.
end

# Put any custom mkdir's in here for when `mina setup` is ran.
# For Rails apps, we'll make some of the shared paths that are shared between
# all releases.
task :setup => :environment do
  queue! %[mkdir -p "#{deploy_to}/shared/log"]
  queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/log"]

  queue! %[mkdir -p "#{deploy_to}/shared/config"]
  queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/config"]

  queue! %[touch "#{deploy_to}/shared/config/database.yml"]
  queue  %[echo "-----> Be sure to edit 'shared/config/database.yml'."]

  queue! %[touch "#{deploy_to}/shared/config/secrets.yml"]
  queue %[echo "-----> Be sure to edit 'shared/config/secrets.yml'."]

  # sidekiq needs a place to store its pid file and log file
  queue! %[mkdir -p "#{deploy_to}/shared/pids/"]
  queue! %[chmod g+rx,u+rwx "#{deploy_to}/shared/pids"]
end

desc "Deploys the current version to the server."
task :deploy => :environment do
  deploy do

    # stop accepting new workers
    invoke :'sidekiq:quiet'

    invoke :'git:clone'
    invoke :'deploy:link_shared_paths'
    invoke :'bundle:install'
    invoke :'rails:db_migrate'
    invoke :'rails:assets_precompile'

    to :launch do
      invoke :'sidekiq:restart'
      invoke :'unicorn:restart'
    end
  end
end

I absolutely love how easy it is to read that file!

Ok, let’s run that setup task to create the necessary folders and files on your Droplet. If one doesn’t get created that you need, no worries. Just SSH back in there and create the folder yourself.

$ mina setup
-----> Creating folders... done.

Set Up database.yml and secrets.yml

Let’s get back on your Droplet and set up your database.yml and secrets.yml files:

$ ssh deployer@YOUR_IP
$ nano /home/deployer/YOUR_APP/shared/config/database.yml

Tweak that file:

production:
  adapter: postgresql
  encoding: unicode
  database: APPNAME_production
  username: postgres
  password: DB_PASSWORD_SET_ABOVE
  host: localhost

…and your secrets.yml…

nano /home/deployer/YOUR_APP/shared/config/secrets.yml

production:
  secret_key_base: RUN `rake secret` TO GENERATE A KEY

Deploying

And finally, deploy!

$ mina deploy
-----> Deploying to 2012-06-12-040248
       ...
       Lots of things happening...
       ...
-----> Done.

Conclusion

That’s it! If you have any questions, leave them in the comments. I want to make this document better for the next guy, so let me know if there’s anything I can improve.

Jon McCartie

How To Delete All Your Tweets

Create an Oauth App

Download Your Data

Delete Your Tweets

Delete Your Likes

Conclusion

How Teams Grow, Thrive, and Fail

Problems

LifeCycles

Early Struggle

Fun

Whitewater

Predictable Success

Treadmill

The Big Rut

Death Rattle

Application

Where is your team?

Implementing Process

Finally…

How Not To Squash Your Team With Red Tape

Building Healthy Teams - Elevate Summit, Denver

Intro

Clarify the Why

Find and Develop Individual Strengths

Create a High-Feedback Culture

Conclusion

Rails 5.1: Default Primary Keys

A Bit of History…

But why not UUID?

Migrate

Rails Core Team FTW

Token-Based API Authentication

Migrations

The Base Controller

Signing In

User Model

API Usage

Expiration

Conclusion

Rails and Slack: Creating a Slash Command App

The Slack Webhook

Receiving the Command

Returning Data

Attachments

Extending

Conclusion

Added "not_in?" to Rails

Heroku National Park Name Generator

Letting Users Add Custom Domains On Heroku

Copy Heroku Database To Local

Modify Your Ruby App's DNS Lookup

Rails' Tagged Logging + Heroku Dyno Metadata

Removing Distractions For Your Team

Measure, Release, Measure Again

The Problem

Hypothesis

Next Steps

Delegation in Rails

Law of Demeter

Example Anti-Pattern

Delegate

Thinking Ahead

Default UUID's In Rails

UUID’s

The Problem

The Solution

Warning

SMS Verification With Rails

Database

Controller

Model

Background Job

Re-Send SMS

Conclusion

User Time Zone On Signup

The Guts

Hide The Dropdown

Non-Blocking