Sam Ritchie

New Blog!

2014-12-09T00:00:00+00:00

Date - Boulder

New Blog

Hey all,

I've moved over to a Ghost blog at samritchie.io. Check there for new posts. I've mirrored the current content over and moved all Jekyll comments. See you at the new site.

Leadville Trail 100, 2014 Edition

2014-08-20T00:00:00+00:00

20 August 2014 - Boulder

Leadville Trail 100, 2014 Edition

Last weekend, I raced the Leadville Trail 100 for the second time. Last year's race was physically brutal; I sat curled up at the 50 mile point, 11 pounds light and unable to keep down fluids, for almost two hours before rallying and banging out a strong second half for a finish of 26:15 (strava report). That race earned me the silver belt buckle awarded to all finishers under 30 hours:

Badass, right? Not as badass as the ungodly huge GOLD belt buckle you get for finishing in under 25 hours.

(Not sure whose crotch that is.) I knew that if I could figure out my nutrition issues, I could break the 25 hour barrier. Last winter, with this arbitrary goal spurring me on (I don't even have a belt that can support one of these), I registered again for the Leadville Trail 100.

PreRace

A lot's changed in my life since last year. I quit my job at Twitter, founded PaddleGuru and moved to Boulder with Jenna. Running in the mountains instead of the streets of San Francisco has given me a huge confidence boost, though I've been logging fewer road miles than last year, and Leadville has a good deal of flat… oh my god, did I screw up my preparation? I should have run more road!! Thoughts like those flared and faded all week. I guess it's impossible not to be anxious before a 100.

My parents, my brother Mike and his girlfriend Katie flew in to Boulder the Sunday before the race. I ignored the race and showed off the town. We drove up to Twin Lakes on Monday to get used to the elevation get into Leadville zone. The cabin we rented had an unbelievable view of the town of Twin Lakes, mile 40 of the course, over which towered Hope Pass. It was hard to breathe, as expected, but Jenna and I had been up to Leadville for training runs this summer and weren't surprised. Here's the view from the cabin porch:

Not a bad place to acclimatize for the #LT100. Good vibes to @sritchie! pic.twitter.com/kWiFLemJST
— jenna (@jennadawn) August 12, 2014

This year I'd be racing with three friends. Aaron Steele from Berkeley was flying out Friday to compete. I'd met Michael O'Brien after the Leadville Marathon, where he came in 4th (!!). Jason Antin is a fellow member of the Satan's Minions scrambling club in Boulder and just an absolute crusher, gentleman and mountain man. Thinking of this as a "competition" was refreshing, as absurd as it sounds to race a hundred miler. Something to focus on besides the slog.

I gave up on preparing my clothes and food early. My anxiety builds as the race gets closer, and packing up supplies the day before the gun goes off is soothing. As long as there are tasks, I don't have to think about the race. And I managed not to think about it, not to really acknowledge it, until the day before it started. I spent so much time this year telling Michael and Aaron about last year's race, about my lack of salt, about how you need to go SLOW, but it felt like a story I'd read, not suffered through. Forced to think about it, I knew that I could BEGIN the race. I knew I could get to mile 40, walk slowly over Hope Pass, then see about coming home. Of course I could do that.

Aaron Steele showed up Friday AM with my parents. He looked cool, just the consummate 100 miler. I'd paced Aaron through an amazing 100 at the Rio del Lago last year, complete with a bonk at mile 72 and a HUGE surge in for a 22:35 finish. Dude's an animal.

Aaron and I did a 30 minute shakeout on a nearby trail and I pointed out Hope Pass for the 15th time that week. I felt a goofy pride at being able to identify markers on the course, even markers as obvious as the enormous mountain pass looming above the town. My confidence faltered when a runner we saw on the trail told me that the right turn out of Twin Lakes I'd been describing vividly to my parents - "This is where you see the town as you roll downhill!" - was actually a left turn. I think I've been manufacturing memories of the course all year. Frightening.

The Night Before

Aaron went to bed at 6pm. I stayed up for another couple of hours, packing, growing jittery, snapping at Jenna and my parents as they asked routine questions. The whole thing was getting to me. Finally, bedtime.

I got up at 1am with Aaron for a pre-race breakfast of coffee, a bagel with Nutella and PB, a yogurt and a banana. I like to eat about 3 hours before the race to give my stomach time to empty. We couldn't stop giggling and woke the crew. Both of us had loopy dreams. I couldn't get back to sleep, so I laid in bed and sent out a burst of tweets about the race preparation. Backfire, as the sound of a Favorite interrupted sleep for my remaining hour.

At 2:30 it was game time. Full body smearage of Aquaphor to prevent chafeage (and I mean full body - I hit the bathroom and slid INTO the toilet); short shorts, warm shirt, hat, fancy Salomon pack, the works. Into the car, to the startline, jacket to the parents, and there we were, lined up again in the dark at 4am. Oh my fucking god.

Start to May Queen (13.5)

My heart rate at the start last year fluttered around 100 and hit 120 with the gun. This year it held closer to 80.

The gun goes and we're off, moving down the road toward May Queen, our first aid station at mile 13.5. Aaron and I started at the front and let the pros stream past us. Aaron let me pull ahead, but I was determined to start slow and ran the first few miles looking back over my shoulder, trying to keep him in sight as he snuck in and out of other runners. The headlamps behind us looked like fireflies and washed out all details of the trail. Betsy Kalmeyer, the woman I paced at this year's Hardrock, told me that in the mountains she sometimes mistakes these lights for stars.

Aaron's plan was to run a really conservative race and hit Winfield at 12 hours, then cruise back for a strong, sub-25 finish. I was trying to break 11 hours into Winfield to match last year's pacing. If I could do that and subtract my 1:40 bonk, I could break 25 and get that big gold buckle.

Two miles in I noticed Michael and our friend Patience to my right. Yes! Company! We talked about how happy we were to be going slow, how we weren't worried, and how great it was going to feel to reel in racers later in the race. Lots of bluster. It always feels like you're going too slow at the beginning. I pointed out spots to remember on the way back to give them hope during their finish sprints. A stump. Some random parking lot.

On the dirt road 4 miles in, moving toward the first huge uphill, I noticed another runner ten feet off the trail, sitting against a tree, taking a shit and pointing his headlamp directly at his ass. Burlesque, baby! Someone's always gotta start strong.

As we hit the uphill and get onto the single track around the lake the runners around us start to break out the stories. One old man tells me about his drop 10 years ago. He's been running hundreds ever since, and is finally prepared to attempt this one again. Later, on the single track, another guy tells me he's here because of a Dean Karnazes quote he'd memorized:

"If you want to run, run a mile. If you want to experience a different life, run a marathon. If you want to talk to God, run an ultra."

I'm not as fuzzy about the "experience" of running - I think those Talk to God moments are mostly because of mistakes in nutrition and pacing - but I smiled and told him he'd come to the right place. I felt bad when I had to cut him off so I could wheel off into the woods and pee by headlamp. There's always an agenda to self-talk in a 100, my "I feel great!" chatter included. Really, I'm so desperate to have someone to run with that each time I stop to pee, I run hard back up to Michael and Patience, risking ankle carnage on the rough single track.

At the boat ramp, I saw my crew, hugged Jenna and Pretzel (my dog) and tried to mentally beam the message that this day is going to go BEAUTIFULLY, then on into the night again. I hit May Queen at mile 13.5 about ten minutes slower than last year and feeling calm.

May Queen (13.5) to Outward Bound (23?)

Last year I had to wait at May Queen for ten minutes while my crew scrambled to fill my pack with food. This year, I'm ten minutes slower, but the crew was spot on. Such a great start to the day. I grabbed some snacks and a new bladder of Perpetuem from Jenna, hit the sunscreen and hustled up the road to join Michael for the hike up Sugarloaf. We started passing people right away on flat sections of trail, jumping around on rocks, keeping the heart rate low. A girl from New York City kept pace with us for a while and told us about her training runs in Central Park. Her boyfriend was pacing her, and he'd already eaten a good amount of her supplies for breakfast by the time she'd seen him at mile 13. Just a sad situation all around.

She was moving and breathing way too hard for me to believe her stated goal of "just finishing", but one thing I learned last year is that no one will be honest with you about their time goals. No need to tempt the race gods by declaring that you want to do go fast; better to stay humble and pretend you're just trying to sneak through the damned thing.

The climb up Sugarloaf went by much faster than last year. I still walked almost every uphill, Now that I'd seen the course I could relax and not wonder how much farther I had to go. My nerves about rolling the ankle I'd sprained weeks before were fading! Michael and I talked to Barefoot Alex, a stud in huaraches and dreadlocks who seemed to know everyone on the course.

Near the top of the climb, Alex said, "it's funny, my fingers are swollen up like sausages!" Holy shit. That's what happened to me last year. He confirmed that he'd been peeing a lot all morning.

"Dude, you've got to start taking salt pills." I pulled a couple out of my pack and pushed them into his huge hands like a salt evangelical. He was ME, and I had to fix him. He took the pills.

Phew, crisis averted. I cruised down Powerline, moving through familiar territory. I pulled away from Michael a little on the downhill, trying to move fast but favor my quads. I passed Jason's girlfriend Jenny on the way down and asked where Jason was - she said he was WAY ahead and hammering. So impressive. There went my chances of winning our little group race.

Outward Bound (23) to Treeline (27.5)

After a mile of quad-punishing descent, I bottomed out, crossed the creek and went up the short rise to the road that leads to Outward Bound and Treeline beyond. My parents hadn't seen Jason, so I figured he must be at least 45 minutes ahead.

Jenna walked with me and I switched to a hand bottle with Perpetuem and some snacks; Michael caught up and we moved on together, acting as pacers for each other by now. The weather was nice and cool, the conversation was good, and BOOM, there was the next aid station! We were maybe 23 miles in now, but Ultramarathon time dilation had set in and it didn't feel long at all.

"Michael, we're halfway to halfway!", I yelled. "Jokes" like that are a nice way to check in on your condition. If you're doing well, they're not funny, but maybe worth a smile. If you're heading down a bad road, the phrase "halfway to halfway" makes you want to cry and tuck into a sleeping bag.

The Perpetuem started tasting bad here. This had happened to me last year on the way up Hope Pass and signaled the start of terrible dehydration. Scared, I dumped the bottle at the aid station, filled it with water and at a PB&J. Crisis averted.

The next bit is usually a long road stretch, but the organizers had subbed in traverse across a grassy field. Cheers had gone up when they announced the change at the pre-race meeting. I was excited, but when I hit the grass I realized that the field was pocked with ankle twisting bomb craters. It was just an absolutely fucked up horrible little pasture that cows couldn't even use. I dumped water on my head and moved through it, feeling strong without the backpack, loving the cool breeze, and tried not to think about how dangerous this stretch might be at night. I hit the Treeline just ahead of Michael and we moved out to Half Pipe together.

Treeline (27.5) to Twin Lakes (39.5)

The Half Pipe aid station came up faster than I thought. So good! I was feeling the heat and eating a bunch of salt pills, measuring myself against last year's ghost. Time is hard to measure here in the trees. It's very hot, and there's nothing notable about the long road rollers and the slow, insidious uphill. This section is a great to stage a push on the way home, but not outbound. Michael and I puzzled over the riddle of the three prostitutes and the two condoms to pass the time. I dumped water on my head at every stream to battle what was looking to be a very, very hot day. Finally the downhill arrived.

I rolled downhill behind a strong looking runner in a blue tanktop. His goal had been to finish sub 20, but he'd been hit with intense diarrhea, forcing him to go slow.

"It's actually really great for ultras," he said. "The diarrhea forces you to walk up hills and save up energy for that second half. Total blessing." Whatever you say, buddy. The state of his asshole bedamned, no way was I passing a guy whose time goal was five hours faster than mine. I stayed behind him into Twin Lakes. We hit the town stretch, and the watch confirmed what I'd suspected - I was back on track with last year's schedule, only about five minutes behind!

Twin Lakes (39.5) to Winfield (50.0)

Last year I left Twin Lakes and immediately puked up the contents of my stomach. Melancholy set in on the way up Hope Pass, and I had to walk into Winfield, losing maybe an hour on my goal time. This year, my plan was to go easy and cross the Pass in great condition to run down. That alone would get me a half hour on least year; if I could get out of Winfield quickly and keep it together, I'd be on track for a sub 25 finish.

I brought my iPhone up Hope and listened to a Radio Lab about this tiny population of whooping cranes and an old man with Alzheimer's. So sad. I was close to crying when I caught diarrhea boy and tried to tell him about the podcast. He didn't have much to say about the cranes, and didn't seem affected emotionally. What a brute.

It was so hot. I dumped water on my head on the way up to keep the heart rate low and made it to the Llama field before the summit with my heart rate below 150, still calm. I pressed on through the aid station and past the bonfire. Near the top, one of the timers called out, "Great to see a smile up here on the pass!". You got it, brother.

When I topped out, I felt the urge to cry. I was so happy that I'd made it up without feeling like death or puking. Reverse self pity!

Heart jackhammering, trying to settle the rate, I moved downhill fast, busting up my quads on the incredibly steep terrain, then turned up the horrible, deceptively uphill two miles of Colorado Trail that leads into Winfield. Stay cool, get to the halfway point.

I saw Jason Antin on the way in, probably 30 minutes ahead of me. I wasn't thinking about catching him at this point. He gave me a bear hug and yelled "Sam Ritchie!". I learned later that he'd seen Jenna at the aid station and told her that I'd be there soon - he could sense my presence on the trail. Not only is Jason an athletic beast, he's also a wizard. Here's Jason at the top of Hope Pass on his second crossing:

Winfield (50.0) to Twin Lakes (59.5)

I reached Winfield 10 hours and 33 into the race. Jenna ran out smiling and walked me in past my parents. My weight was stable, maybe two pounds down from my starting weight. Much better than 11 pounds last year. I ate some oranges, refilled my water, stuffed some ice into my headband. We were out within 12 minutes. BOOM! Suddenly I had almost 40 minutes on last year's time! I wasn't 11 pounds light and on my knees in the dirt! Such a celebration.

I was feeling rough. Last year, even though puking outside the aid station tent wasn't exactly recovery, those two hours gave some minimal break to my stomach. This year I felt weak from the hot, hot sun. Jenna and I got to the Colorado trail and managed to keep up with last year's slow pace. The goal for the back 50 was to recreate what I'd done last year. No need to go faster. If I could keep moving, I'd be sub 25, earn my big arbitrary gold buckle and be free of the demons.

My stomach started to revolt on the way up Hope. My heart rate hammered up to 150 at the left turn onto the Hope Pass trail and wouldn't settle. I walked, let downhill runners fly by and tried to keep my food down. Jenna kept pushing me to eat. "How about just one Pretzel?" Even these took a few minutes to choke down. It was really hard.

The back side of Hope Pass has three phases. The woods, the boulder fields and the switchbacks. Each is about a mile long. Near the top of the woods phase we passed Jenny heading down. Jason had pushed, met her at the top of Hope Pass and proposed. "He even put the ring on this cord since he knew my fingers would be so swollen!" Everyone descending around her was smoked, favoring their quads and grimacing, but Jenny had the best smile on her face and was telling everyone she could about the engagement. She ended up finishing in 29:23. So awesome.

We met Aaron Steele about halfway up the boulder fields. Aaron looked worked. I had been worried about his nutrition situation, and it was clear that he was a little behind, but smiling as always. It felt SO good to stop and talk and let my heart rate drop down to 115. I gave him a Honey Stinger bar, told him to stay cool and dunk his head in the stream, then pushed on. Jenna was dousing me with water by now, filling up our water bottle with stream water and dumping it all over my neck and arms. The heart rate kept climbing. We hit the section of trail where I'd gotten the Strava CR weeks before… no chance of pushing anywhere close to that speed now.

And then, finally, we were at the top among waving Tibetan prayer flags.

.@palmajl will do! We'll be getting into night soon so the good vibes will be well taken! cc @sritchie pic.twitter.com/S8Sp8Wws1b
— jenna (@jennadawn) August 17, 2014

At the aid station, Jenna took selfies with the Llamas while I tried to eat noodles and drink water. I felt like puking every time we started running. God, is there no way to avoid this after 55 miles? Maybe my body could only function with a two hour break halfway. Maybe I couldn't get through a hundred miles without getting so fucked up I had to stop… the only question was where.

Cranking down the shaded trail on the north side of Hope, I focused on moving over the slippery roots and making good time as Jenna pushed behind me. Finally we were at the bottom, still in full daylight. My stomach allowed me to eat more pretzels near the bottom. I got another mental boost when I realized that we might beat last year's Hope crossing by 15 minutes.

I'd already clawed back a couple of hours on my time; if I could hang on, I might be able to push the 24 hour mark. On the other hand, aiming for too fast of a time goal could cause another blowup. Well, what the hell. I started getting excited about going faster.

Twin Lakes (59.5) to Treeline (71.5)

I changed my shoes at Twin Lakes and picked up Eric Coppock, a fellow minion who'd volunteered to pace me the final 40. There was bad pain in a couple of my toenails, but I was prepared to lose a few and didn't look closely when I changed my socks.

Water was still making me nauseous, but I loaded on on watermelon at the aid station, and DAMN was it good. Eric had a few bottles of Ensure in his pack; these had gotten me through the night last year.

Out of Twin Lakes, you climb up the Colorado trail to the Mt Elbert trailhead through groves of enormous aspens. It's a huge climb, but runners only talk about Hope Pass and Powerline, so it can come as a slap in the face if you're not expecting it. In the evening, it's one of the most beautiful spots on the course.

We hiked uphill. I described my day so far to Eric, trying to reassure myself by putting a positive spin on the tale. We'd only run once together at Mt Sanitas in Boulder. Eric was calm about the stomach issues and suggested, over and over until I agreed, that I try the Ensure… and, YES, it stayed down! There could be hope after all. After half of the Ensure my stomach had settled and I was able to start drinking gulps of water again. We started jogging the small flat breaks in the climb.

We hit the Mt Elbert mini aid station 3 miles up the hill, and the course began to flatten out. We started to crank downhill, hitting 8 or 9 minute mile pace. I needed a 15 minute mile average at this point to break 24 hours. It seemed daunting, but I knew that these long downhills would let us put a few minutes in the bank for the later climbs. Every 11 minute mile that passed was encouraging.

Running, drinking Ensure, drinking water. Trying to get a bottle an hour down. This was my life. We started passing people. The sun was still up, unlike last year, and I could run in a t-shirt and feel warm. My heart rate was down to 140 now that I was out of the heat. All I had to do now was deliver and use this excellent terrain to gain time before the sun set and the cold forced me to slow down.

To our right was the most beautiful sight I've ever seen in the mountains. I don't have a picture, and can't find one, but the mountains to the East were blood red. A storm was pouring rain in the distance, and ruby shadows flickered in the enormous column of water pouring down over the hills. I yelled to the runners ahead, "LOOK TO THE RIGHT!". Some turned and hooted, others kept marching, grimaces untouched. Hey, I tried.

At the Half Pipe aid station, I grabbed more watermelon and we moved out for the last two miles to Treeline. We took a walking break and prepped the headlamps. Eric's was this wild homemade deal with no on/off toggle that he clipped to his belt. It was incredibly bright, and seemed to give him energy. I think it was some sort of Ironman core.

Treeline (71.5) to Powerline (79.5)

It was dark when we hit Treeline with its line of cars. Treeline was quiet; we were going fast enough now that we were ahead of the 25 hour press, so runners were only coming through every couple of minutes. My parents, my beautiful parents, had COFFEE. Oh my god, so good. I clutched it and sucked it down like Gollum slurping fish guts. Jenna ran out with me, then realized after a few minutes that she hadn't brought a headlamp. Eric and I moved down the trail while Jenna stumbled back to the cars in the dark. Whoops.

The road section was hard. I wanted to slow down, but Eric was telling stories and keeping a fast pace, so to be polite I kept running and talking. Soon we hit the awful, cratered field near Outward Bound and slowed to a walk to pick our way around the rocks and bomb holes. I tried to run a little, but stuck my foot in one hole and snapped my knee backward, then did it again and almost rolled my weak right ankle. Enough. We walked into Outward Bound, fueled on watermelon and moved out toward the bottom of Powerline, where Jenna's brother Adam was waiting for his first pacing leg ever. Yesterday Adam had told me he was worried that he wasn't going to be able to keep up with me. With 80 miles in my legs, I knew he wouldn't have a problem.

Powerline (79.5) to May Queen (86.5)

I turned down the road to Powerline with Adam and Eric. Eric wanted all the training he could get for the Run Rabbit Run 100 later this month. His justification: "Well, it's going to be hard to start running again if I sit in the warm car for a couple of hours, so I'll just head over the summit with you guys." The stud factor was high.)

We power hiked every step of the uphill at a steady, 17 minute mile pace. My heart rate moved between 125 and 140. We were passing runners, jogging every short flat, downing Ensures. Powerline is the land of a 1000 false summits. Every few minutes, you think you've topped out when another green trail-marking glowstick looms high above you and you realize that no, this flat is only here to deceive you, the trail winds up and up.

At the top of Powerline (the last climb conquered!) we ran into an impromptu aid station of blazed guys and girls with glowsticks, water, Coke and a board spraypainted with the words, "YOU FUCKING ROCK!". Eric refilled our bottles and Adam and I moved on. Behind me I heard some girls whispering, "you have to slap his ass as he leaves!" right before a girl in a skeleton shirt and a glow stick necklace ran up and whacked Adam. His first pacing experience was going all right.

I didn't know how my feet were moving so fast down the rutted, rocky jeep trail. I felt disconnected from them in the wash of the headlamp. I looked down at my feet and saw them jumping over rocks and turning over at high cadence. I had no understanding of how I was making that happen.

I thought about how many steps I'd taken so far and felt a little sick. Just keep going. Finally, the flat section of Hagerman road. I ate a gel before the final, bombing single track downhill, the last significant downhill on the course. It felt endless, but Adam was so happy - "Sick!!" - that I had to smile and gutted it out. The technical downhill of this section required an IQ level that was beyond my reach. Keep drinking water, keep moving.

May Queen (86.5) to Leadville (100.0)

We hit May Queen over an hour ahead of the schedule I'd set out for my parents, and left the aid station just one minute before they arrived. Adam found a bar of service and called Jenna. "We're leaving May Queen." I could hear "What??" on the other end of the line, and then the service cut out.

At this point, my brain could understand the remaining distance. A half marathon, nothing crazy. I knew Jason was ahead of me, probably more than thirty minutes. These are dangerous thoughts with hours left, but I thought that it MIGHT be possible to catch him at the end if I pushed hard. Let's see.

The first four miles of lake is very technical in the dark. I had some hot chocolate at the aid station, which raised the IQ by a couple of points, and then we were back at it, walking the uphills, running the down, switching gears every few seconds at times. Don't think about how much time is left, just run.

We could hear the boat ramp, and then there it was! I was just a total asshole by this point. Jenna asked me, "Wow, how did you take an hour off your time?"

"By running fast, obviously… how did you guys take two hours to get May Queen?" Ouch. It's embarrassing to think about now. I was at a dark place, desperate to get the final Ensure and two gels and leave. We dropped Adam off and kept moving fast, passing a couple more runners. Eric was quiet and efficient, offering water, holding me back.

Oh, man, this was it. A 12 minute mile average MIGHT be able to get me in in under 23 hours. I knew that I could hammer the last five miles, but was the pain worth it? I decided I could at least try, and if I bonked and walked it in still get a time closer to 23 hours than 24, far faster than I'd ever expected. Okay. Keep eating. Almost game time.

We hit the rocky downhill off the lake. My nerves had calmed. I knew my final push was going to start at the bottom. I kept drinking Ensure. The calories had to be there for this part. I could see my breath and relaxed down the hill, ignoring my swollen hip flexors, ignoring the throbbing pain in my toes. We hit the dirt and started running, really running. My technique felt flawless, though it probably looked terrible. We dropped down to an 8:30 pace, and started to gain on the headlamps in front of us. Eric has his own 100 miler coming up and decided not to get too wild; after a mile of our insane push, he handed me the water bottle and a gel and sent me off to the finish. I knew the markers now. I knew I could go fast. Every minute below a 12 minute mile would bank me time if I bonked.

I was flying, slowing down on the bumpy dirt road by the train tracks, taking in a gel at the bottom of the turn to the last big uphill, hiking hard over the rocks, three miles to go, maybe a little more, don't think about how long that ACTUALLY takes - three miles is nothing! - I stopped looking at the heart rate monitor and watched my breath fog ahead of me. Headlamps flared up as pacers looked back at the noise. Everyone was wonderful, cheering for me as I passed. "Finish strong, man! Yeah!" Two miles to go, now. Don't think, don't look at the watch, just wait for the road and gun it.

Then, finally, I could see pavement. I hit the last dirt uphill and turned left onto the road, alone now, no fire in me for a final sprint but going as fast as last year. I'd been going this fast for miles, now, still down at 8:20 pace. Up the final rise, and I could hear the finish line now… and then, holy shit, ahead of me, could it be Jason? I recognized him by his huge calves. I didn't care about catching him anymore, just about finishing and fighting off the quitting bargain my brain was trying to make.

It was Jason! He saw me passing on the left and yelled, "Is that Sam? Is that Sam Ritchie? You're killing it, man! You're killing it!" So, so good to hear those words. We had maybe a half mile to go, and I could see the finish line, but I couldn't make sense of the patterns of lights. My vision started to wobble and I forced it steady, drinking water, not letting myself fade, trying to stay fast and steady uphill…. and there it was, the clock at, holy fuck, 22:38! Up the red carpet, looking for my parents and Jenna, hoping they hadn't missed the finish… and there they were. Finally, finally, I could stop. I stopped. I was sore, happy, and mentally there. And finally, finished.

Later, I found out that I'd gotten third in my age group. This is completely surreal to me. I used to be a kayaker, then a crossfitter. The running realm is intimidating. But whatever happens now, Leadville 2014 was an amazing time out in the mountains. Here's a shot Jenna took at the awards ceremony:

And, of course, the bounty:

#LT100 Bounty from this year! Glad I didn’t have to fly… pic.twitter.com/tL0SGTqZ1h
— Sam Ritchie (@sritchie) August 19, 2014

UPDATE: Here's a fun Storify with social media from the day! Thanks to my lovely crew for firing these off while I raced.

API Authentication with Liberator and Friend

2014-01-17T00:00:00+00:00

Jan 17 2014 - Boulder

API Authentication with Liberator and Friend

I've just finished rewriting a number of PaddleGuru's internal APIs using two great open-source libraries; Liberator and Friend. Liberator is a library for writing RESTful resources in Clojure. Friend is an authorization and authentication library written by the prolific Chas Emerick, Dominator, Esquire. You've certainly seen his stuff around if you've played with Clojure(Script) in any level of detail.

Authentication and authorization are both really important in RESTful APIs. These libraries are made for each other, I thought to myself. I'll just use them together and life will be wonderful. Right?

Well, not so much. Friend and Liberator both have opinionated approaches to authentication. Integrating the two was trickier than I imagined, and required a bit of code massage.

In this post, I'll give a short overview of Liberator and a not-so-short overview of Friend. (Friend's more confusing, and needs a longer treatment.) I'll discuss how each of these libraries deals with authorization and authentication, and walk through an example project I built that demonstrates how PaddleGuru uses Friend and Liberator in concert to define RESTful API endpoints with really nice, consistent authorization and authentication handling.

The code for the example project is up on GitHub.

Liberator

Liberator lets you define a RESTful resource as a graph of decision points and responses (represented as key-value pairs in Clojure). It handles content negotiation, resource caching, all of the tough stuff that you end up doing as nested function calls without a library like Liberator.

From the project site:

Liberator is a Clojure library that helps you expose your data as resources while automatically complying with all the relevant requirements of the HTTP specification (RFC-2616). Your resources will automatically gain useful HTTP features, such as caching and content negotiation. Liberator was inspired by Erlang’s Webmachine. By following the constraints and requirements in RFC-2616, liberator will enable you to create application according to a REST architecture.

Here's a simple resource that handles only HTML responses, and returns 406 Not Acceptable if the user requests some other content type:

(defresource hello-resource
  :available-media-types ["text/html"]
  :handle-ok "<html>Hello, Internet.</html>")

Because a resource is a function of a request, you can use resources with Compojure like this:

(ANY "/foo" [] hello-resource)

Check out the project page for more tutorials and documentation.

Friend

Friend provides Ring middleware that handles authentication and authorization for your app. ("Authentication" is whether or not the system knows who you are; authorization is whether or not you're allowed in to a particular resource, one the system identifies you.)

The middleware looks like this:

(ns liberator-friend.middleware.auth
  (:require [cemerick.friend :as friend]
            (cemerick.friend [workflows :as workflows]
                             [credentials :as creds])))

(defn friend-middleware
  "Returns a middleware that enables authentication via Friend."
  [handler users]
  (let [friend-m {:credential-fn (partial creds/bcrypt-credential-fn users)
                  :workflows
                  [(workflows/http-basic :realm "/")
                   (workflows/interactive-form)]}]
    (-> handler
        (friend/authenticate friend-m))))

This middleware sits over top of all of your resources and routing layer (typically handled with a library like Compojure), and provides all of the plumbing necessary for authorization and authentication.

(Okay, this is where it gets confusing. For me, anyway. There's a lot of indirection to keep track of in the API. Follow me as best you can, and supplement with Friend's extensive documentation.)

Authentication

Friend's "workflows" provide pluggable authentication for your app.

Friend considers a request to be authenticated if the incoming request's session has a certain special key:

{:session {::cemerick.friend/identity <user's identity!>}}

(That's a namespace-qualified keyword, by the way.)

Friend's middleware examines every incoming request for this key. If the key is present, Friend passes the request on, no problem. (If you're using a session store, this will prevent your app from having to run through the login workflows on every request).

If ::cemerick.friend/identity is missing from the session, Friend's middleware attempts to authenticate the session using its workflows. The middleware passes the request into each workflow in turn until one kicks out a return value, or all have returned nil. Only then will the middleware pass your request on.

Let's talk about the supported return values. Workflows can return one of three things:

nil: nil means that the workflow has no nothing to say about the supplied request. Friend will send the request to the next workflow in the list, if one exists. If no workflows are left, Friend calls your handler.
Friend Auth: This is ANY Clojure map with a type of ::cemerick.friend/auth. The default workflows try to authenticate a user using the :credential-fn you supplied to the middleware (see my above example). If :credential-fn returns a map, the default workflows interpret it as a user record, associate the ::cemerick.friend/auth type metadata, merge the identity into the request under {:session {::cemerick.friend/identity <returned auth map>}} and call your hander with the updated, authenticated request.
Anything else: Any other response is treated as a ring response, and passed back immediately. Your handler is never called.

Let's look at an example workflow to see how it handles these three cases. In my snippet above I included Friend's http-basic workflow:

(require 'cemerick.friend.workflows)
(workflows/http-basic :realm "/")

(Here's the code if you want to follow along.)

When this workflow sees a request, it first checks the request for an authorization header. If that header's missing, it returns nil, and Friend proceeds to the next workflow, or lets the request through.

If the authorization header IS present, the workflow extracts the supplied username and password and passes them in to :credential-fn.

If this check succeeds (ie, returns something non-nil), the workflow returns the required ::cemerick.friend/identity key described above. If it fails, the workflow returns a failing ring response:

{:status 400
 :body "Malformed Authorization header for HTTP Basic authentication."}

You can use these three response types to implement some pretty interesting authentication workflows.

You can do a lot in this framework. Ddellacosta's Friend OAuth2 workflow intercepts the initial OAuth request and uses the "failure" return to send out an OAuth2 redirect to the configured provider. When the provider redirects back to the app, the OAuth2 workflow again intercepts the command, does token negotation, then either succeeds or fails the response. Two intercepts! There's a lot going on there.

Authorization

Okay, phew. That covers authentication. Now we need to talk about authorization, or protecting your resources.

Resources typically handle authorization with some function of the ::friend/identity that the middleware added to the session. If the identity is missing (IE, the request isn't authenticated) or the identity doesn't have the required permissions, the resource can throw an exception with cemerick.friend/throw-unauthorized with the identity that didn't pass the check. (This might be nil, of course.) This function throws an exception with some special metadata.

Friend's middleware is wrapping the entire app, and catches exceptions with this special metadata as they bubble up. Once this happens, Friend takes responsibility for the response with one of two actions:

If Friend sees that the user is authenticated, it calls :unauthorized-handler. (You supply this option when you create the middleware. This is where you'd return some sexy, custom page, or redirect to the home page with a flash yelling "You're not authorized!"). You can include custom info in the thrown exception to make that flash all custom and sexy.
If the request is NOT authenticated (no ::friend/identity in the session), Friend calls the :unauthenticated-handler. By default, this stores the URI the request was originally trying to access in the session map and redirects the user to your login page.

Now, in the latter, unauthenticated case, Friend typically redirects to a route that's being watched by one of the workflows. Friend's supplied interactive-form workflow does this; it redirects to a URI like "/login", then intercepts POST requests to "/login" and tries to pull out credentials and authenticate.

Once you're authenticated (and this is a new thing I didn't mention above), if the session has any record of the URI you were trying to access when the app threw the unauthorized! exception, Friend will BREAK from the pattern I mentioned above and instead redirect to that stored URI. This gives the resource another chance to check your (now populated) credentials.

If you make it through, great. If the resource throws an exception again, Friend will catch it again, but this time take the first branch and call :unauthorized-handler.

I find all that throwing and catching to be extremely confusing. I'm not really sure how to clean it up, but please, please let me know if you have ideas after ingesting all of this.

Combining Friend and Liberator

Liberator has a decision point to deal with authorization and authentication: :authorized?. You provide a predicate for the :authorized? key in your resource definition, and Liberator will either call its :handle-unauthorized handler (on false) or proceed down the decision tree (on true).

After figuring out Friend and absorbing all of the intricate subtleties described above, it became clear to me that a single predicate was NOT enough for really good auth. Rather than rolling my own session management, redirect handlers, etc, I had to figure out how to use the two libraries together.

My main blocker here was that Liberator didn't allow resources to inherit key-value pairs from other resources. The resources are effectively maps, and you should be able to define a base map of decision points and then merge them together.

So I wrote a pull request that extends Liberator's resources to accept a :base key. The :base key takes a map of liberator decision points and creates your resource by merging the other kv pairs into these defaults.

This pull req allows you to define a base resource like this:

(def base-resource
  "Base for all resources.

   Due to the way liberator's resources merge, these base definitions
   define a bunch of content types, even if the resources that inherit
   from them don't. The defaults are here to provide reasonable text
   error messages, instead of returning big slugs of html."
  (let [not-found (comp rep/ring-response
                        (route/not-found "Route not found!"))
        base {"text/html" not-found}]
    {:handle-not-acceptable
     (->> {"application/json" {:success false
                               :message "No acceptable resource available"}
           "text/plain" "No acceptable resource available."}
          (with-default "text/plain")
          (media-typed base))

     :handle-not-found
     (->> {"application/json" {:success false
                               :message "Resource not found."}
           "text/plain" "Resource not found."}
          (with-default "text/plain")
          (media-typed base))}))

And then write other resources that extend the base like so:

(defresource hello-resource
  :base base-resource
  :allowed-methods [:get]
  :available-media-types ["text/plain"]
  :handle-ok "Welcome to the resource!")

This tiny resource now shares the :handle-not-acceptable and :handle-not-found behavior from the base. If I hit the resource and ask for JSON, for example, I'll get a "No acceptable resource available." message in plain-text. (There's more work here to make this perfect, but hey, it's a start.)

Check out my customer version of defresource in the post's example project. That namespace also contains base-resource and all the helper functions.

Authenticating Resources

liberator-friend that shows off my final solution: Liberator resources that delegate to Friend into the authorized? point. The code is on GitHub.

The example project defines a Friend base resource that provides a handler that Liberator calls when :authorized? returns false:

(def friend-resource
  "Base resource that will handle authentication via friend's
  mechanisms. Provide an authorization function and you'll be good to
  go."
  {:base base-resource
   :handle-unauthorized
   (media-typed {"text/html" (fn [req]
                               (unauthorized!
                                (-> req :resource :allowed?)
                                req))
                 "application/json"
                 {:success false
                  :message "Not authorized!"}
                 :default (constantly "Not authorized.")})})

friend-resource extends base-resource from above, just for fun. The unauthorized! function above is also mine; it pulls the ::friend/identity key out of the request, and also sends the function representing next step in the Liberator decision tree up to Friend's middleware. (If the user's not authenticated, this lets Friend workflows perform auth with a database, then jump BACK into Liberator's decision tree at the allowed? stage to try again. Pretty awesome.

That covers the Friend middleware integration. Now all we need to do is override :authorized? on each resource to return true or false, and everything else will just work. I wrote a few helpers that make it easy to test Friend's identity map in Liberator's authorized? function:

This resource extends the base resource, but adds in a default unauthorized handler. This is all Friend needs - if the user's unauthorized, either handle it immediately, OR, in the HTML case (assuming browsers always access via HTML), the resource throws the proper redirect.

Now all we need to do is override :authorized? on each resource to return true or false, and everything else will just work.

I wrote a helper function that defines nice authorization predicates based on Friend's concept of a role:

(defn roles
  "Returns an authorization predicate that checks if the authenticated
  user has the specified roles. (This is the usual friend behavior.)"
  [roles]
  (fn [id]
    (friend/authorized? roles id)))

This function creates a new base resource that extends friend-resource above, adding in the supplied authorization function:

(defn friend-auth
  "Returns a base resource that authenticates using the supplied
  auth-fn. Authorization failure will trigger Friend's default
  unauthorized response."
  [auth-fn] {:base friend-resource
             :authorized? auth-fn})

Those two helpers work together to create Friend-aware (Friend-ly?) base resource generators. All resources that use these bases will be protected by the Friend middleware. In the example project, this means that they'll be protected with HTTP basic authentication, but you can add more workflows to perform different auth in a way that doesn't require you to rewrite your resources.

(defn role-auth
  "Returns a base resource that authenticates users against the
  supplied set of roles."
  [role-input]
  (friend-auth (comp (roles role-input) :request)))

(def authenticated-base
  "Returns a base resource that authenticates users against the
  supplied set of roles."
  (friend-auth (comp boolean friend/identity :request)))

The first, role-auth, takes a set of roles and allows access to the resource if the authenticated user has a role that's in the set.

authenticated-base just checks that the user is authenticated (that the ::friend/identity key is present); no additional authorization comes into play.

The example project performs authentication using an in-memory "database":

(def users
  "dummy in-memory user database."
  {"root" {:username "root"
           :password (creds/hash-bcrypt "admin_password")
           :roles #{:admin}}
   "jane" {:username "jane"
           :password (creds/hash-bcrypt "user_password")
           :roles #{:user}}})

Now, let's define some resources that use these helpers. These resources all use Friend for authorization. They allow, respectively, admins, users and any authenticated user.

(require '[liberator-friend.resources :as r :refer [defresource]])

(defresource admin-resource
  :base (r/role-auth #{:admin})
  :allowed-methods [:get]
  :available-media-types ["text/plain"]
  :handle-ok "Welcome, admin!")

(defresource user-resource
  :base (r/role-auth #{:user})
  :allowed-methods [:get]
  :available-media-types ["text/plain"]
  :handle-ok "Welcome, user!")

(defresource authenticated-resource
  :base r/authenticated-base
  :allowed-methods [:get]
  :available-media-types ["text/plain"]
  :handle-ok "Come on in. You're authenticated.")

Now we can serve these out using Compojure:

(defroutes site-routes
  (GET "/" [] "Welcome to the liberator-friend demo site!")
  (GET "/admin" [] admin-resource)
  (GET "/authenticated" [] authenticated-resource)
  (GET "/user" [] user-resource))

Now let's hit the shell to test out the custom auth.

Testing with CURL

You can follow along by cloning the example code and running lein run in the project's root. The default route has no authentication requirement, and returns the string defined in the compojure routes above:

[sritchie@RitchieMacBook ~]$ curl localhost:8090
Welcome to the liberator-friend demo site!

Now let's hit the admin resource without basic authentication.

[sritchie@RitchieMacBook ~]$ curl localhost:8090/admin
Not authorized.

Because we didn't include a basic auth header, Friend's basic-auth middleware returned let the request through without adding ::friend/identity. The request hit the Liberator resource, the :authorized? check failed, and Liberator delegated to the :handle-unauthorized decision point defined in friend-resource. This decision point ONLY throws the Friend exception for "text/html" requests, since I only wanted to redirect for Browser requests. Instead we get the default "Not authorized." response defined here, decked out with the proper 401 Unauthorized response code. Thanks, Liberator.

Let's try it with bad credentials.

[sritchie@RitchieMacBook ~]$ curl -u root:wrongpass localhost:8090/admin

We get no text response, just a 401 Unauthorized. Because I included basic auth credentials and an authorization header, The basic-auth workflow in Friend's middleware DID try to authenticate. When authentication against the users failed, rather than pass the request through to my liberator :handle-unauthorized hook, Friend returned its own default response.

I think that this is the most confusing aspect of integrating Liberator and Friend. Because Friend's workflows DO sometimes return their own responses, if you're going to throw an unauthorized! exception you need to prepare for this and share the proper responses between the middleware resources and your custom workflows.

Finally, with proper credentials:

[sritchie@RitchieMacBook ~]$ curl -u root:admin_password localhost:8090/admin
Welcome, admin!

The basic-auth workflow adds ::friend/identity into the session, :authorized? checks for the :admin role and returns true, and :handle-ok returns "Welcome, admin!".

What if we supply valid credentials, authenticate properly with Friend, but try to access a route that we're not authorized to see?

[sritchie@RitchieMacBook ~]$ curl -u jane:user_password localhost:8090/user
Welcome, user!

[sritchie@RitchieMacBook ~]$ curl -u jane:user_password localhost:8090/admin
Not authorized.

Friend's basic-auth workflow lets both requests through, but :authorized? returns true in the first case, false in the second. Because Friend's middleware was happy Friend supplies no response, leaving the response to Liberator. Liberator calls :handle-ok in the first case and :handle-unauthorized in the second.

For completeness, here are the same routes with valid admin credentials:

[sritchie@RitchieMacBook ~]$ curl -u root:admin_password localhost:8090/admin
Welcome, admin!

[sritchie@RitchieMacBook ~]$ curl -u root:admin_password localhost:8090/user
Not authorized.

And proof that the /authenticated route allows any valid credentials:

[sritchie@RitchieMacBook ~]$ curl -u root:admin_password localhost:8090/authenticated
Come on in. You're authenticated.

[sritchie@RitchieMacBook ~]$ curl -u jane:user_password localhost:8090/authenticated
Come on in. You're authenticated.

Conclusions

So, there you have it. Friend and Liberator, working in glorious harmony.

As confusing as I find Friend, I think it's the best solution out there for authentication and authorization for Ring applications. Communication through exception football can be pretty confusing, but it seems like the best way to handle the redirect coordination you need if you want users to be able to "pause" a route, authorize at a different route, then come back to the original URI for another try.

Both of these libraries are worth exploring, and together they sing. After the initial learning curve, the combination has made it easy to iterate on RESTful APIs in Clojure here at PaddleGuru.

Upcoming Talks in 2013

2013-08-24T00:00:00+00:00

24 August 2013 - San Francisco

Upcoming Talks in 2013

This is the year I teach myself to become a better public speaker. I've spent the past year coding up a number of powerful Scala and Clojure projects, all the while avoiding the important and difficult work of teaching and writing about the import and use of those projects. Well, no longer.

To mind that gap I'll be giving a number of talks in 2013 on my recent work on Summingbird and Cascalog. If you're in the Bay Area, Boston, or Northern VA (my home town!), I'd love to meet you.

Introduction to Summingbird (Sept 3rd, Bay Area Storm Users)

Oscar Boykin and I will be giving an Introduction to Summingbird at the Bay Area Storm Users meetup on Tuesday, September 3rd. Summingbird is a streaming MapReduce framework we developed at Twitter that allows you to write a streaming computation once and execute it in batch-mode on Scalding, in realtime mode on Storm, or on both Scalding and Storm in a hybrid batch/realtime mode that has very attractive fault tolerance properties. Sound exciting? RSVP at the event's meetup page.

Realtime MapReduce at Twitter (Sept 22nd, CUFP)

Check this one out at the Commercial Users of Functional Programming conference, September 22nd-24th in Boston. This talk will focus more on the functional programming ideas we used in the design of Summingbird's real-time MapReduce DSL and its component libraries (Storehaus, Bijection, and Algebird). Which ideas helped our design? What hurt?

Summingbird, Scala and Storm (Sept 25th, Boston Storm Users)

Another Introduction to Summingbird talk, with a focus on how Scala's type system and style helps us write correct Storm topologies. Register at the Boston Storm Users' meetup page.

Twitter: Taking Hadoop Realtime with Summingbird (Oct 19th, PNWScala)

This talk at the Pacific Northwest Scala Conference will get into the guts of Summingbird's batch/realtime merging abstraction and the streaming MapReduce graph planner we've developed. Here's the abstract:

Twitter's Summingbird library allows developers and data scientists to build massive streaming MapReduce pipelines without worrying about the usual mess of systems issues that come with realtime systems at scale. This talk will discuss the development of Summingbird's hybrid Batch and Realtime operating mode, the power of clean, mathematical abstractions and the massive creative leverage that functional design constraints can give to a project.

The talk will also discuss some of the applications for these technologies currently being used at Twitter.

Registration is filling up fast, so sign up today!

Streaming MapReduce in Clojure (Nov 14-16, Clojure/Conj)

This talk will discuss my work on Cascalog 2 and its new "execution platform" abstraction. The recent Cascalog 2 redesign split the datalog compiler apart from the underlying Cascading functions that get called when you run a Cascalog query on a MapReduce cluster. With luck, by the time I deliver the talk, Cascalog will be able to run on Trident and Spark.

I'll be delivering this talk in my hometown of Alexandria, VA at this year's Clojure/Conj.

So You Want to Build a MapReduce DSL? (Jan 11th, Data Day Texas, Austin TX)

Still working on the meat of this one, but some initial thoughts - I'd like discuss the vast number of choices in the MapReduce DSL world. Spark, Scoobi, Summingbird, Scalding… folks are writing Hadoop DSLs in Ruby, in Mathematica, in Clojure. Where does it end? What's the right level of abstraction? This talk will discuss why so many different options exist and some pitfalls to beware of when choosing between them.

More information on Data Day Texas, including registration info, on the event page.

Leadville Trail 100

2013-08-20T00:00:00+00:00

20 August 2013 - San Francisco

The Leadville Trail 100

This past weekend I ticked off one of my athletic life goals; the Leadville Trail 100, a brutal 100 mile running race in Colorado. I signed up for Leadville this May after a team injury forced our boat out of the 2013 Texas Water Safari, a 262 mile canoe race down in Texas that is my usual ultra-length torture for the year. Less than three months is less training than recommended for a hundred miler, but what the hell. Rational thinking never led anyone to a successful hundred mile finish.

Here's how the trip unfolded.

The Buildup

I arrived in Colorado with parents the Sunday before the race to acclimate. A week living at 9,000' seemed enough. The plan was to code throughout the week, but as nerves took over, work fell by the wayside and I started devouring running books and movies. Once a Runner and Again to Carthage by John L. Parker, Eat and Run by Scott Jurek, started a Pre biography… I watched Unbreakable and In the High Country and generally spent my time staring at the mountains and trying to push out Krupicka-style beard hairs. This was not successful.

This was my parents' first time crewing an ultra, other than the Texas Water Safari. That race is interesting in that all the support crew needs to provide is water and ice. They need to get to locations along the way, but they don't necessarily need to be creative or imaginative about WHAT the racer will need along the way. We do that a lot for each other in the boat.

In the days leading up to the race I tried to teach my parents what would be required. We went for a hike up the back side of Hope Pass on Tuesday. I power-hiked, feeling great, and ran down, imagining what a stud I'd be doing the same thing during the race. I even picked out a couple of hundred yard sections that I would run (making up, what, 10 seconds?). How young and optimistic I was. After the hike we drove back to Leadville, bought a course map and visited each of the aid stations. I pretended my brother was racing and tried to conjure up images of the coming race for my parents. "Mike's coming in, he's been out for this many hours…" and so on. Sort of worked. I made the mistake of calling our way to Tabor boat ramp a "secret way", which freaked my dad out beyond belief. What other way is there? Are there other racers heading some other way? Jesus, chill out. Don't do anything different on race day.

Oscar Boykin arrived Wednesday night, ready for action. We did a run the next morning before my girlfriend Jenna showed. I was feeling good and banged out a 7 minute mile before letting Oscar go kill himself at altitude. I felt pretty low on energy that week, but okay with the altitude after a day or two. Strangely lethargic, I slacked on buying food until the day before the race, when I finally forced myself to imagine what I'd need and kicked my ass into gear for a big shopping trip. My parents bought a TON of organizational items (tupperwares! Zip locs! Trash bags!), perhaps not realizing how fast and light a crew needs to be at each of the aid stations. No worries, we'd figure that out later on the road.

Race Morning

The gun goes off for Leadville at 4am. The plan the morning of the race was to wake up at 1am, get about 800 calories in me along with a couple cups of coffee, then hit the sack for a half hour. I got up on time and chowed everything bagel with almond butter and jelly, a greek yogurt and a banana. Chugged one cup of coffee, then back in bed. Another cup of coffee in the car on the way to the race.

My resting heart rate on the way over there was 80 bpm, about 30 beats high. At the start line, it jacked up to 105. Calm down. My plan was to hold back, not go out too fast, and err on the side of SLOW. I figured if I fell back I could always make up the time later. I was shooting for a 24 hour time (foolish on the first Leadville?) And had planned out a pace chart that had me going at the average speed of a 24:32 finisher. If this felt too slow, then all would be well; I'd have energy to start beating the splits in the second half of the race and push the pace down. If too fast, no problem. I'd be in a great position to finish the race.

We pulled into town in the dark and found a spot right near the finish line. I got out, lubed my feet up with BodyGlide, put on my Minimus shoes and ambled over to the coffee shop across the street from the starting line. Some folks were doing warmups back and forth along the road, as if the race wasn't long and brutal enough to provide that service. So many faces looked haunted in the coffeeshop. I imagine that the excited faces were first timers like me, looking forward to the adventure. Those who had been through the ordeal were wiser and terrified, perhaps remembering some of the pain from past years and questioning their decision to come back to this damned town. Healthy living at 10,200 feet!

I was trained and ready to go. Oscar, Jenna and the parents would drive the course and meet up with me to provide food and water at various points along the first 50 miles. At mile 50, Andrew Stephens, my partner in the Texas Water Safari and good friend, was to show up and run with me over the mountain. Jenna wanted to be waiting at the top with the Llamas at Hopeless aid station and would take over from Stephens. From there, we'd figure it out on the go.

Start to May Queen (13.5)

I left my folks and Jenna with a kiss and moved out to the start line. The temp was about 40 degrees. I felt calm, but the heart rate was up at 105. The gun went off and I started my watch, shuffling out of the gate at 10:00/mile pace right in the middle of the pack. We all moved down the hill in the cold. Some folks started walking on the first uphill, pacing just right for the distance ahead.

The first few miles were surreal. I remembered to look back and take in the sea of headlamps moving down the hill, looked up at the stars and thought about The Long Walk by Stephen King. So many hopes and dreams out on the road. It was nice making conversation with people from different towns, starting to make friends then suddenly having folks veer into the woods to pee.

The start of the course rolls down a mile of pavement, transitioning into a long dirt corridor between trees. I focused on the average pace I was trying to maintain - 9:47 minutes per mile - and just stayed slow and shuffled. Ann Trason's advice, via Aaron Steele, was to be as efficient as possible with every step. I kept my cadence high and started to eat at the half hour point, conserving energy and fueling up.

Casual first five miles, feeling easy. The race was going by faster than I expected, there in the dark. The Peloton hiked up the long scree section at the end of the dirt road, and then it was on, shuffling around Turquoise lake to Tabor boat ramp at mile 7.5.

This is when the first warning signs of later trouble began. I had stopped to pee about 30 minutes into the race. All was clear, which was good. I'd hydrated nicely that morning. Twenty minutes later, I had to pee again. I ended up peeing five times in the first hour and a half. I was drinking fluids and thought I was just on the hydration, but I was peeing more than I drank, which just didn't make any sense. I later found out that this was a sign of low salt, and that my body would become worse and worse at retaining fluids until I started taking salt pills.

The run around the lake was really beautiful. I was in a SLOW group, walking everything, making about 13 minute miles, so I did try to pass a few people, but was generally happy with the pace and figured that if I lost even 25 minutes here, rested up and then stayed on my pace for the rest of the race, I'd still be within reach of the big buckle. Getting ten minutes up on my pacing this early in the race was going to be a lot harder than taking back ten minutes from the plan near the finish, when most runners spend huge blocks of time walking. Almost took a wrong turn, got back on track, peed again (!), and made it into Tabor where I surprised the parents, Oscar and Jenna with a high five, then back on single track until May Queen.

May Queen (13.5) to Fish Hatchery (21.5)

I pulled into May Queen only eight minutes behind schedule, at around 2:18. Doing so well! Oscar was right there to meet me and handed me my pack. It seemed like a beautiful handoff, until I asked, "Did you add the two scoops of powder?" and Oscar's eyes got wide.

"I thought you already mixed it!" he said. We were jogging along now and passed Jenna and the family.

"Are there gels in here?", I said to Jenna.

"No, we thought you added those too!"

I was running along with a pack of straight water and no food. I had skipped the aid station, anticipating a full on-the-move refill, and here I was with nothing. Jenna went back to get me some gels and Oscar took off up to the car to find the carbohydrate powder. I waited at the trail head for seven minutes, knowing how foolish it would be to leave without any food but not wanting to run the half mile back to the aid station. After twelve minutes waiting, it was time to leave. I poured the new pack's water into my existing pack, saving SOME calories from the remaining perpetuem, and begged a bar off another pace crew. Finally Jenna caught up with me and handed me a couple of gels, but in the confusion I put these in the pack that I left behind… so I was off over the first big mountain pass, two hours with a 150 calorie bar when my plan had been 300 calories per hour. It was going to be a long day.

@sritchie had a minor crewing difficulty at the last station. Hoping to make up for it at the 22 mile mark. pic.twitter.com/yxKtjJDbut
— P. Oscar Boykin (@posco) August 17, 2013

The mountain was fine. I took the steep stuff at a hike, passing people all the way up. I hike well, and the altitude isn't a big deal at that pace, so I made great time, trying to silence the urge to go fast and make up missing time from the earlier fumble. At this point I was twenty minutes behind pace, a gap that would be easy to catch up. Go slow.

The sun up was beautiful, and I looked around quite a bit, taking it all in. Here I was racing at Leadville! The first tough climb was underway. I started asking folks around me what their pace goals were and seemed to be solidly in a pack of 23-25 hour racers. Was I going out too fast? Or were they all destined to blow up? I chatted with a dude going for his 30th finish (!!) and a girl looking to finish in 23:30 - 11 hours out, 12:30 back. Aggressive pacing goals. We crested the summit and I ran down the powerline, eating up the miles and letting the heart rate drop. The feet and lungs were feeling great. I was hungry, but I was almost to the next aid station, so no problem there.

Fish Hatchery (21.5) to TreeLine (27.5)

I saw Jenna and Oscar again when I hit the road. Perhaps embarrassed about the last pacing situation, they presented me with a pack that was just CRAMMED with food. I had only two hours until I would see them again, but they'd given me dozens of items, along with fresh drink mix. I applied sunblock, gave a few handfuls of food pouches back, tried to be nice and moved on out. The clouds were out, which gave the racers a break from the heat. I was 21 miles in and feeling really good, like not much time had passed, but just a bit rough and queasy. I walked the uphills, favoring the achilles a bit, but noticing that I hadn't really felt any pain at all for quite some time. Other problems had overtaken my achilles issue.

A few miles down the road was the Fish Hatchery aid station. I ate a couple of pretzels and choked down a PB&J half, then back out to the road for a long stretch up to treeline. I was dead on the expected pace for the 24 hour finish, maybe pushing a little fast, keeping things in control. Treeline aid was great; I met up with Oscar and the folks, sat down and shook a rock out of my shoe that had been bothering me. Better to take care of these things now than suffer later. They swapped in another pack and some more food and I moved on out into unknown territory.

TreeLine (27.5) to Twin Lakes (39.5)

The next section was a long, steady uphill with a big downhill finish into Twin Lakes. I hiked up the initial climb, knowing that I could make up time on the downhill. The goal here was to keep moving and get to Twin Lakes with solid legs in preparation for the grueling trip up Hope Pass.

I started to ask more folks about their time goals. One guy told me that he was going for 25 hours, "like every damned runner within a two mile radius of us". Fair enough. I ran into a jacked crossfittish guy with his shirt off who seemed to be having trouble with his stomach and gave him some advice on how to eat. Shortly after that, I joked about my swollen fingers to a dude running behind me. "You need salt, man," he said, and how right he was. I didn't put this together with my early peeing. Again, an ominous sign of things to come.

The trail was gorgeous, with long uphills and good conversation all the way to the Mt Elbert trailhead. The elevation gain was pretty large here, but the slope was gentle and didn't feel as terrible as I had expected. By the time we hit the downhill I had taken a couple of salt pills and felt on top of my game. I pushed through the Mt Elbert station and down Twin Lakes. One guy noted that I must have huge balls to have worn Minimus shoes in the run. All my Krupicka-watching and running reading have made a convert of me. I'm not ready to run in Huaraches, but those Hoka One Ones look like a serious departure from the One True Way (sorry Aaron). I cruised downhill at the lead of our little group, MAYBE one guy passing me on the way down. We pulled in to Twin Lakes in 7:18, just three minutes behind the expected pace. Oscar was waiting for me at the top of the hill and led me down to the rest of the family.

@sritchie is through 40 miles. Heading up Hope Pass. #LT100 pic.twitter.com/1d139eRZae
— P. Oscar Boykin (@posco) August 17, 2013

Twin Lakes (39.5) to Hope Pass (44.5)

My coach and two-time Leadville champion Duncan Callahan was waiting at Twin Lakes. I sat for a couple of minutes, taking in food and going over the items in the pack with my parents. I wanted to leave at 7:30. I applied some sunscreen and took my iPhone and headphones from Jenna for the next stretch; the dreaded Hope Pass. Duncan had me eat a couple of pretzels, and I grabbed a PB&J and some boiled potatoes for the trip out. I took my two Excedrin, a salt pill, ate two pepto pills, and chugged a cup of flatted coke.

Mental check-in: at this point I'm feeling epic. It's mile 40, I'm injury-free and slowly eating away at my already optimistic pace goal. My planned pace up Hope Pass was 22 min/mile; I knew that I could hike the steep at around that pace, and the mile to the base of the climb is perfectly flat. It should be no problem to make up ten more minutes right there.

Oscar walked me out of the aid station. Just outside on the trail as I pushed the PB&J down I started feeling queasy. I told Oscar that I might have to puke, matter-of-factly, trying to stay casual. I kept my gut down, but just after he left I went to my knees on the side of the trail and surprised myself with an incredibly forceful vomiting session. I opened my mouth and out poured a stomachful of caffe latte perpetuem, almost no solids in the food, just a liter or so of liquid. Racers along the trail asked me if I was okay. One guy yelled, "Hey, at least you know you'll feel great when it's over!" One more time, emptying out the rest of my stomach, and then I was back up, feeling good. I ate another pepto tablet, put my headphones in and kept going. I wanted to start running hard and thank the racer who made the comment. I knew this wasn't smart and kept the pace back, but knocked out the flat mile to the base of hope in 11 minutes. I thought to myself, "this race isn't so hard," and started walking up the hill.

I became more dejected as we rose. The trail on this side of hope was just so steep. I came upon one strong, young racer sitting by the side of the trail, head in hands, moaning to himself. I asked if he was okay. He replied, "yeah, I'm good," not looking up. Yeah, right. I caught up to a woman who looked really strong and passed her. Ten minutes later I became dizzy and had to sit down. She caught up and offered me a gel, and I decided that passing a bad idea. Time to relax. I was hiking at a 22 min/mile pace, which would land me at the Hopeless aid station exactly on target. My heart rate was up at 168 by now. I had set my heart rate alarm at 165, and after a number of "too high!!" beeps I turned the alarm off. Why was my heart rate so high? Could it be that I was extraordinarily dehydrated and my heart had to work hard to pump my thick blood? Don't think about it, just shut off the alarm and keep hiking. This is a hard climb. You're SUPPOSED to work hard.

Half way up I found that I couldn't quit thinking about water. I knew I had vomited up everything in my stomach and was probably behind on fluid, but I had been very diligent about drinking and didn't imagine that I was dehydrated. I hadn't peed in a long time. I was fascinated with the noise of the stream that we were hiking past. A mile more and I actually saw the stream water and dunked my hat, squeezing the cool water on my neck and feeling the amazing restorative effect of cold. I wasn't out of breath, but I was dizzy from the high heart rate and made myself to stop again. So difficult.

I pushed out of the trees at the 9 hour mark and could see the tents of the aid station up ahead. I knew that I only had a half a mile to go to the top of of Hope Pass; after the summit I could cruise downhill, just as I had on my training run. I'd be at Winfield FAR ahead of schedule. If I stayed smooth I'd get to Winfield at 10:30, leaving me 14:30 to get home. Even feeling diminished, I felt like I had this in the bag. If you ever feel this way on Hope Pass, something is wrong. I had underestimated Leadville.

Hope Pass (44.5) to Winfield (50.0)

I headed to the aid station tent, chugged a cup of coke and decided that I would sit down for five minutes. I had puked up the Excedrin earlier, so I grabbed two Tylenol out of a sample dispenser. Ten minutes ahead of pace over a five mile stretch in the hardest part of the race seemed like a foolish way to move. A five minute rest would get me back on track and let me CRANK over the top of the pass.

Two minutes into this break, the nausea hit again and I headed off by a tree just in time to puke up all of the water and calories in my guts yet again. Both heaves were incredibly productive and forced liquid out of my mouth and nose. After the second heave, an aid station medic came over to me and asked how I was doing. I told her I'd let her know in a minute and heaved again, this time coming up with nothing. my stomach was empty, and the two tylenols lay half-dissolved in the grass. The medic made me drink a cup of ramen and asked me how often I had been peeing. I told her that I had peed at Twin Lakes, though thinking back I can't remember if that's actually true. Lighting up, I told her that I had puked down there after chugging a cup of coke, just as I had now.

"You don't have enough salt in your system, honey. You're trying to put too much into your stomach, and your stomach is shutting down. You need to take smaller bites." She asked me about my salt intake, then had me lick my wrist and poured a bunch of salt onto it, "just like a tequila shot." I licked it up and finished my ramen. She brought me another cup of ramen and told me to finish it by the summit. Boom, I was released! I was shivering heavily by then and pulled out my shell, silently congratulating myself on some solid thinking. I would hike slowly to the top and get warm.

As I approached the summit, down came the two guys in the lead, hauling ass down the mountain, not more than a couple of minutes apart. It was fantastic to have almost reached the top before seeing the leaders, and I was grateful for the small rest I got as I moved off of the trail to let them pass.

At the top of the pass down the other side, I started to feel better right away, as expected, and started catching up with the runners in front of me. And suddenly in front of me was a familiar face! Scott Jurek, HAMMERING up the back side of Hope on a hunt for the leaders. I recognized his pacer; Hal Koerner, two-time Western States champion, from my recent viewing of Unbreakable. I whooped and yelled, standing aside as he came past. Jurek had blue pasties over his nipples.

"Nice pace, man! And nice nipples!" I yelled.

"Thanks," breathed Scott as he passed. What a warrior.

Inspired, I cruised downhill for about a mile before I remembered that I was hosed and couldn't keep up that pace. I was starting to sweat and feel terrible inside of my shell. Racers started to pass me for the first time all day. I would step off the trail to let them go, hands on knees, just miserable and thinking about water. The taste of my perpetuem mix was so nauseating. I would take the tiniest sip possible to help me dissolve food, then struggle to keep the mix down. I got down to the treeline without many problems, but just couldn't run at all and resigned myself to walking the trail. So many incredible runners were passing me on the way up by now. Some of these studs were stopped by the side of the trail, trying to puke as their pacers congratulated us runners coming down.

I thought about the woman at the Hopeless aid station and her question about my pee schedule. When was the last time I had really peed? I decided that I needed to make it happen, and stopped at the side of the trail. I forced it and was surprised and scared to see that it was a dark, dark yellow. Oh boy. I knew that I had been vomiting fluids, but I figured that this was excess and SOME had been getting absorbed. Wrong. I was severely dehydrated, about an hour from aid. Time to get moving.

Finally, there it was, the turn onto the Colorado trail to Winfield. I was walking, but still on pace to get into Winfield at 11 hours. I knew that I would need to stop, but in my mind was thinking, "if I can just get to that station and rest for 10 minutes, I'm going to be okay." I still thought that I was on target for the Big Buckle. We crossed a stream and I dunked my hat again. Bad idea with the wind. I started shivering, but the cold quelled the pounding in my head and I felt good enough to keep walking and push onward. The desire to sit down by the side of the trail started to set in, but I knew that it wouldn't help. After a while I could SEE the aid station. My shirtless CrossFit friend caught up. His stomach had started working again, and he was pushing. Many of my old friends from the treeline->twin lakes section passed me, all trying to cheer me up. "It's San Francisco!" yelled one lady as she moved by me. Looking great, everyone. One awful mile to go.

Finally, the road. A quarter mile more on the flats and I was at the aid station, 11:05 into the race, RIGHT on track with Duncan's schedule even with the dehyrdration. Excellent. Just sit down and take ten minutes to get back on pace. Jenna and Oscar found me on the road and led me into Winfield and the checkpoint. I took a moment to smile. I was halfway through the Leadville! Whatever happened from here on out, it was exciting to have gotten through the first 50 miles. I was 17 miles into unknown territory.

Winfield (50.0)

The medical staff led me to the weigh-in, and there it was on the scale - 154.5 pounds, down from 165.5 at the start. I had lost 11 pounds of fluid. I sat and Jenna brought me some broth and my fantasy, a cup of straight-up WATER with nothing mixed into it. I drank two cups of water and some broth and started to shiver heavily. This young girl, a doctor, came over and asked me what I wanted to eat. I told her water and oranges, and she brought me these and gave me a sleeping bag to wrap around my legs. My lips were very blue. I thought that my parents might ask me if I wanted to quit, but I knew that I wouldn't. I also knew that in my current state I wouldn't be able to continue. How to resolve the paradox?

This is where the race really surprised me. I had done everything right, I thought. I was eating and drinking, pacing myself really well, shielding my eyes and skin from the sun… but here this doctor was, telling me that my stomach had shut down and that my kidneys might be failing. I was extremely low on salt and wasn't retaining water, and they wouldn't let me go until I could pee again. What the hell? Renal failure? How are you supposed to tough out a race when the machine, your body, betrays you? Was my body just not build to run ultras? After three Texas Water Safari wins, this was hard to believe. Still, I didn't know how to recover.

At this point, I must have looked seriously fucked up because Oscar wouldn't even take a video of me. Here's a Vine he took after forty minutes:

As I was suffering, like a dream, Andrew Stephens showed up, ready to RACE. He had his pack on and immediately took command of the situation. He made me change into a warm, dry shirt, put a hat on me and gave me two salt pills. He looked mildly annoyed when I jumped out my chair, ran behind the tent and puked up everything in my stomach. I sat back down inside and he started feeding me water and more salt pills, then asked the doctor if they had any Zofran. They did, and I took one. Zofran is an anti-nausea medication for chemo patients. I knew that if this didn't settle my stomach, I was hosed.

But it worked! I was able to hold down two liters of water and four more salt pills. I warmed up and began to feel better. I was swaddled in blankets at this point, an hour and ten minutes into my rest at Winfield. I couldn't pee yet, but I was drinking water with an electrolyte supplement. M&Ms tasted good, as did broth and small sips of coke. I started sipping on an Ensure as well, keeping it all down.

And then, an hour and a half into the rest, I had to pee! I went out to the porta-potty and forced out a sample. It was dark, but not AS dark. I had gained back three pounds now. I showed another doctor and he smiled. "Your kidneys aren't failing, you're just severely dehydrated. Start running!" My pulse ox was back up to 97% (from 86%). Once Stephens heard that, it was go time. I felt like Jurek now, MUCH better, and wanted to hammer. My motivation was back. I had lost sight of the 25 hour belt buckle, but I thought now that we'd be able to finish this race in a respectable time.

Winfield (50.0) to Twin Lakes (59.5)

So there it was, out of the aid station 12:45 into the race, 1:15 ahead of the cut-off time. I wanted to run, but Stephens warned me that this was a bad idea. We were already passing dozens of runners on the trail. His new rule was that every time I wanted to start running, I needed to drink. He started feeding me pieces of bagel and cream cheese, which I washed down with the water. "If you go slow, you're going to finish this thing. If you start puking, it's all over." I knew he was right and made me choice. We went slow.

At the turn to the Hope Pass climb Andrew said, "this is the most badass race of all TIME. I'm definitely coming back to do this next year." He was obsessed already, like me during my first Texas Water Safari. I was mildly pleased to hear him gasping as we pushed it up toward 11,000 feet. I tried to keep my breathing under control to seem like a badass, then gave in and huffed and puffed.

Once we hit the switchbacks, Andrew said, "Just two more to go and then we're at the summit."

"Dude," I said, "we have 1.2 miles to go. I just ran the course."

"I know that. I was just trying to trick you and make you feel better," he said. Nice. Runners around us started pitching in with stories of their pacers trying to trick them too. We all death marched up to the summit at 12,400, taking the time to look at the gorgeous scenery and enjoy the day.

I had come back from the dead, and I KNEW that I was going to be able to finish the race. Now that I was hydrated my heart rate had dropped to 140, down from 168 of the first trip over Hope. At the summit we started running downhill to the aid station. The station had run out of cups, but Andrew figured out some system and got me another cup of broth. We took five minutes, I cleaned out my shoes and made Andrew take a picture of me by the llamas so we could show Jenna what she was missing. Seven minutes later, now 1:45 ahead of the cutoff, we took off down the hill, passing people and comparing Leadville to the Texas Water Safari. I kept eating and drinking, feeling better and better with every mile.

Twin Lakes (59.5) to TreeLine (71.5)

We pulled in to Twin Lakes far enough ahead of the cutoff that I stopped thinking about it, and Oscar switched in on the pacing duties for Andrew. After the morning pacing situation, I was overly worried about having a pacer other than Andrew, and was a little mean and intense toward Oscar. I tore into him right away at the Twin Lakes aid station with rapid fire questioning. "What do you have in your pack? Where's your headlamp? Where are the extra batteries?" Just chewing him out like he was a kid. I'm sure he wanted to punch me, and would have if he didn't have his own killer background in long-distance running and an understanding of the strange mental states that one can experience during these races. My big concern was that I would hit the wall and need my pacer to take complete control. Only later did I realize the incongruity of yelling, "Are you going to tell me what to do!!?".

It was just getting dark. I changed my shoes, hit the bathroom (thank god, organs functioning again!) and Oscar and I took off up the hill. It was time to hike again using the Stephens method, keeping the heart rate under control and eating as much as I could without getting sick.

At this point I was feeling great again. I though I understood what had gone wrong, and knew that if I could keep eating and drinking and stay ahead on fluids and salt that I would be able to finish this race. I hadn't known how important salt was, but I damned well understood it now, and felt that the potential problems of the race were all under my control. I wasn't affected by the altitude, and had excellent supplies and aid stations the whole way home. I was going to finish this race.

Excellent pacing by Oscar through the whole section. We came to the Mt Elbert aid station, fluids only. I rolled through while Oscar filled up a bottle then caught up to me, cruising along in his Merrell trail gloves. We were hiking the uphills and running every flat and downhill, passing people like crazy and feeling great about the beautiful, warm night.

In no time, we were through the long stretch and walking the last mile or two into the Half Pipe aid station. I drank a delicious cup of hot chocolate and some broth, and we carried potato chips and other items out along the road. As before, we spent very little time in the aid station. A couple of miles later we hit the TreeLine, mile 71.5. The parents, Jenna and Stephens were waiting for me with an Andrew Stephens special: the Leadville pizza. Salami and cheddar cheese on top of a bagel. (Where had the cheddar come from?)

TreeLine (71.5) to PowerLine (79.5)

As happens in the Safari, at this point I had the feeling that the race was almost finished. This was ridiculous, as the farthest I had run before the day in question was around 33 miles, only a short 5k farther than the distance we had yet to go. Oscar and I started talking about code, running for a while, walking for a while and the miles melted away. A woman along the road heard us talking about Scala and Hadoop and called us a "couple of trail philosophers!" I was surprised that we were so upbeat, given the day's events. The night was incredible, stars just blanketing the sky above. So beautiful. In a few miles, we'd reach the base of powerline and Jenna would pick up crewing duties. Another cup of hot chocolate in the Fish Hatchery aid station, a water top-off and we were off.

Just as I was pouring my unwanted hot chocolate out on the road, I heard Oscar say, "Can I get a sip of that?" But it was too late. Only a drop remained, but it seemed enough to cheer him up and power him through the remaining mile. We cruised the final downhill to Jenna, all decked out in pacing attire and a fresh headlamp. My spirits were getting higher and higher.

PowerLine (79.5) to May Queen (86.5)

Not much to report for this section, other than fantastic pacing from Jenna. I was sticking to the Stephens formula now, drinking, looking at my watch and eating every twenty minutes, trying for 250 calories per hour and keeping my heart rate under 145. I was happy to be out and healthy around Jenna, but I took time to prepare her for the monster that would reveal itself if I did crash, and how she'd have to force me to eat and drink as I was crying and puking at her feet. Luckily the nightmare didn't come to pass. We passed a bunch of people and ran the entire downhill, all the way through the two miles of single track, over the bridges and out to the road. An older couple was waiting at the exit of the single track.

"Go runners, you look great!" they yelled. Then, "Hey, is that Sam?" It was my parents, huddled in the cold with a table and a little tray of various treats. Jenna and I ran up, grabbed a few items of food and a packet of Chamois Butter out of the box and kept on cruising. We were both feeling great and Jenna decided to crew me for five more miles. 15 miles of pacing is no joke, but she maintained her stoke the entire time, hauling water and food for me, fetching and mixing up various treats and medicinal concoctions as I requested them.

The May Queen aid station was just intense. The runners looked like ravaged mine workers, slumped, dejected, sipping away at cold broth and lukewarm cups of coffee. I sat down to relube my feet while Jenna ran to grab extra food. We greeted the runners who were present (including the old Italian man we'd met in town before the race, the one running the Spartathlon in a couple of weeks!) then got up and hauled ass out of the gate. We weren't going to break any records, but I still wanted to come into the finish solidly under 27 hours. I had rested at the 50 mile point for 1:40. My goal was to finish 1:40 slower than my intended target time of 24:35. We left MayQueen with this in mind, solidly on track, sipping Ensure and pushing for the boat ramp.

May Queen (86.5) to Leadville (100.0)

I started to feel quite tired along this stretch and lost my discipline, running the uphills on the single track around the lake. I wanted to reach town before sunrise. Jenna saw what was happening and forced me to slow down and eat. I knew that Stephens would have been proud. We still had 11 miles to go, probably 2.5 hours on fatigued legs. We were passing racers and pacers, and there was no need to do anything special. Now that it was night, just as it had been on the way out, I remembered the course well and applied myself to the task at hand. Eat, drink, and shuffle along the miles.

At 7.8 miles to go we hit Tabor boat ramp and there was Oscar, packed up and right on schedule. He switched in for Jenna and we kept on at the same pace, passing runners, talking less than before as I dealt with the exhaustion. My ankles were hurting quite a bit. Get to the downhill scree, then it's smooth sailing to the finish, I thought.

Finally, we were away from the lake. We crossed a road, following a line of glow sticks over to a steep, sandy, rock-covered slope, about a quarter mile long. At the bottom, finally, blessedly… flat road, and the five mile mark.

Now, at this point the distance to go became comprehensible. I knew that five miles back home was about the distance from the golden gate bridge to my apartment. I'd run this route often in training, no problem. I knew what pace I could hold. We could see the barest glow of sunlight over the town, dimming the stars out above the mountains. I forced the pace down to an 11 minute mile, fast enough that Oscar could stop pretending to jog and actually move his feet a little. Gotta give the old man a workout. The last five miles wasn't marked and no one was out to cheer for us, so it was a quiet push. I ate a peanut butter cracker every ten minutes and tried to keep the pace.

Right turn at the rail road tracks. Down the road, by the trailers, left turn up the rocky slope, passing runners… and there it was, 3 miles to go, the long dirt road lined with trees where a long day earlier I'd had conversations with new friends, all happy, all full of hope. The road sloped gently uphill and we started to push faster and faster, only 2.5 miles to go now, 1.5 miles to the road. A while later we could see it, and we started to run faster and faster. All of the pain was gone from my legs. I couldn't feel my ankles, I couldn't feel my lungs, and my heart rate stayed low, unaffected by the altitude. I clicked the pace higher as we hit the final incline to the pavement, checking my watch. We'd dropped our average from 11:00 to 9:30 per mile.

"How fast are we going?" asked Oscar, pushing uphill next to me.

"Must be eight minutes," I said. I ran harder and joked, "What's wrong, too fast for you?" Oscar was running a marathon in two weeks, and had already jammed 26 miles. He said, "It is, man. Go for it, I'll watch you run," and slowed down, letting me pull ahead.

This was it. The final mile. I was feeling fantastic and pushed faster and faster, past racers, up over the hill by the middle school to the final stretch… and there it was, the finish line, with 26:12 on the watch and just a short distance to go. I was going to make it! I was down at a 6:30 now and hit the downhill flying, feeling a little ridiculous moving past limping runners trying to drag themselves in. Who would race for such an arbitrary time goal? 26:15?

Up the hill with Jenna, arms pumping, down the red carpet and across the line, finally, finally finished in 26:15:12. I hugged the race director, who hung my finisher's medal around my neck and sent me over to to medical. I stepped on the scale and weighed in at 163.5, 9 pounds up from my diminished state at mile 50.

I walked over to my parents, Jenna and Oscar, feeling fantastic and joking about taking another lap around the building. I felt so good! It was eerie, walking around not even tired. I sat down for a minute to take my shoes off, and when I tried to get up…. boom, it hit me. I suddenly could barely stand. My body locked up and I started shivering again. But it was all over. We'd done it, and I was happy.

Cascalog Testing 2.0

2012-01-22T00:00:00+00:00

22 Jan 2012 - San Francisco

Cascalog Testing 2.0

A few months ago I announced Midje-Cascalog, my layer of Midje testing macros over the Cascalog MapReduce DSL. These allow you to write tests for your Cascalog jobs in a style that mimics Cascalog's own query execution syntax. In this post I discuss midje-cascalog's 0.4.0 release, which brings tighter Midje integration and a number of new ways to write tests. I'll start with a refresher on the old syntax before debuting the new. If you're eager, add the following to your project.clj:

[midje-cascalog "0.4.0"]

Midje-Cascalog Refresher

Take the following Cascalog query:

(use 'cascalog.api)

(let [src [["word"]]]
  (?<- (stdout)
       [?out-word]
       (src ?word)
       (str ?word " up!" :> ?out-word)))

Executing this code at the repl prints a single tuple with the string word up! to standard out.

How would you go about testing that this is true? With midje-cascalog, you would swap out the ?<- form for its testing equivalent: fact?<-. Here's the same Cascalog test alongside a typical Midje test:

(let [src [["word"]]]
  (fact?<- [["word up!"]]
           [?out-word]
           (src ?word)
           (str ?word " up!" :> ?out-word)))

(fact "+ should add two numbers."
  (+ 2 2) => 4)

I find that fact?<- and fact?- macros can be a bit confusing when you start mixing Cascalog and Midje tests, as they break the Midje pattern of <thing-to-test> => <expected-thing>. The syntax updates fix all of this with a set of checker functions that mimic Midje's excellent set of collection checkers.

The "produces" checker

Midje-cascalog 0.4.0 introduces the produces function, mirroring Midje's just. Let's define a source of tuples and a query to test.

(use 'cascalog.api)
(require '[cascalog.ops :as c])

(def src
  [[1 2] [1 3]
   [3 4] [3 6]
   [5 2] [5 9]])

;; adds the values in each input tuple, sorts the output and returns
;; 2-tuples of the first number and the sum. [1 2] becomes [1 3], for
;; example.
(def query
  (<- [?x ?sum]
      (src ?x ?y)
      (:sort ?x)
      (c/sum ?y :> ?sum)))

You can think of a query as a set of tuples waiting to be generated (through query execution). With Midje, you test sets using the just checker:

(facts
  [1 2 3] => (just [1 2 3])    ;; true
  [1 2 3] => (just [1 2 3 4])) ;; false

The cascalog analog to just is the produces checker. produces works like just, but against queries instead of bare collections. Executing the following test shows that the query produces the expected set of pairs, in any order:

(facts
  query => (produces [[3 10] [1 5] [5 11]])  ;; true
  query => (produces [[1 5] [3 10] [5 11]])) ;; true

You can read this test as saying "query, when executed, produces [3 10], [1 5] and [5 11]. You can also check that a query doesn't produce a set of tuples by swapping out =not=> for =>:

(fact
  query =not=> (produces [["string!" 11] [1 5] [5 11]])) ;; true

Using the :in-order keyword after the expected tuple sequence forces the test to respect ordering:

(facts
  query =not=> (produces [[3 10] [5 11] [1 5]] :in-order) ;; true
  query => (produces [[1 5] [3 10] [5 11]] :in-order))    ;; true

(:in-order is really only helpful in cases where output is sorted, like our query above.)

produces-some

The produces-some checker tests that a query's output contains a subset of tuples:

(fact
  query => (produces-some [[5 11] [1 5]])) ;; true

Note that the behaviour of produces-some is similar to the behavior of Midje's contains collection checker.

As with produces, you can use the :in-order keyword to force produces-some to respect ordering. Gaps between tuples are okay.

(facts
  query =not=> (produces-some [[5 11] [1 5]] :in-order) ;; true
  query => (produces-some [[1 5] [5 11]] :in-order))    ;; true

Adding the :no-gaps keyword introduces the constraint that tuples must also be contiguous:

(facts
  query =not=> (produces-some [[1 5] [5 11]] :in-order :no-gaps) ;; true
  query => (produces-some [[1 5] [3 10]] :in-order :no-gaps))    ;; true

produces-prefix and produces-suffix

produce-prefix mimics the has-prefix collection checker by checking that some set of tuples is produced at the beginning of the query's output. produces-prefix implicitly assumes that tuples will be produced in order with no gaps:

(facts
  query => (produces-prefix [[1 5]])         ;; true
  query => (produces-prefix [[1 5] [3 10]])) ;; true

Similarly, produce-suffix mimics the has-suffix collection checker by checking that the supplied set of tuples is produced at the tail end of a query:

(facts
  query => (produces-suffix [[5 11]])) ;; true

log-level keywords

In addition to the keyword options supported above, every one of these checkers supports on optional logging-level keyword. For example, the following two facts are equivalent, but the second one produces :info level logging when it runs:

(facts
  query => (produces-suffix [[5 11]])        ;; true
  query => (produces-suffix [[5 11]] :info)) ;; true

Log level keywords can be useful when debugging tests, as errors will often only appear in the logging output. Currently supported keywords are :off (the default), :fatal, :warn, :info and :debug. The log level needs to be the first keyword argument if you supply multiple.

wrap-checker

The real power of the 0.4.0 update is the way in which the previous query checkers were defined. Each of the above checkers mimics the behavior of one of Midje's built-in collection checkers with slightly different keyword arguments. This makes sense if you think of a query as a collection of tuples waiting to be produced (by query execution). The above checkers will get you quite a ways, but what if you want to test a query against some other Midje collection checker?

The answer is wrap-checker. wrap-checker is a higher-order function that accepts a midje collection checker and wraps it up, turning it into a Cascalog query checker. I'll demonstrate the power of this function by wrapping Midje's has checker.

has is a powerful way to run functions across every value in some sequence:

(fact
  [1 3 5 7 9] => (has every? odd?) ;; true
  [1 3 5 6] => (has some even?))   ;; true

If you try to use has against a query it will fail, as it expects to be tested against a sequence, not an unexecuted query. Here's how to get around this:

(defn odd-tuple? [tuple]
  (odd? (first tuple)))

(defn even-tuple? [tuple]
  (even? (first tuple)))

(def has-tuples
  (wrap-checker has))

(def new-query
  (let [src [[1] [3] [5]]]
    (<- [?x] (src ?x))))

(fact
  new-query     => (has-tuples every? odd-tuple?) ;; true
  new-query =not=> (has-tuples some even-tuple?)) ;; true

has-tuples will support log-level keywords like any of the predefined query collection checkers.

A few more examples:

(defn id-query [src]
  (<- [?x] (src ?x)))

(let [one-of-tuples (wrap-checker one-of)
      two-of-tuples (wrap-checker two-of)
      src [[1] [3] [4]]]
  (facts
    src            => (two-of odd-tuple?)           ;; true
    src            => (one-of even-tuple?)          ;; true
    (id-query src) => (two-of-tuples odd-tuple?)    ;; true
    (id-query src) => (one-of-tuples even-tuple?))) ;; true

Backwards Compatibility

All of the collection checkers discussed above can be used with the fact?<- and fact?- macros:

(fact?<- (produces-some [[1 5] [5 11]] :in-order)
         [?x ?sum]
         (src ?x ?y)
         (:sort ?x)
         (c/sum ?y :> ?sum)) ;; true

fact?<- and fact?- are also compatible with all of Midje's unwrapped collection checkers, as discussed here.

Conclusion

Midje is an astonishingly good testing framework; I'm continually surprised by how well its idioms and conventions satisfy Cascalog's needs. In my next post here I'll go over some of the more subtle details of the wrap-checker function. For the curious, here's the code.

If you'd like more information or additional features, please add your thoughts to the midje-cascalog github issues page, or let me know in the comments below (or on twitter! I'm @sritchie09.)

Introducing Cascalog-Contrib

2011-11-15T00:00:00+00:00

Nov 15 2011 - Washington DC

Cascalog-Contrib

I've had the pleasure of working with Cascalog for about ten months now, and have seen the community produce some fantastic work. A number of businesses are using Cascalog in production; I use Cascalog at Twitter every day to write MapReduce queries for the new Twitter Web Analytics product.

One thing Cascalog doesn't yet have is a community repository for generic queries and operations. To fill this gap we've created cascalog-contrib.

Cascalog-contrib will be home to any higher-level abstractions over Cascalog that the community is willing to submit. If you have an idea for a module, file a pull request on GitHub or bring it up on the mailing list for discussion.

The first cascalog-contrib modules are now live on clojars. To include them in your leiningen or cake project, add any of the following to project.clj:

[cascalog-checkpoint "0.1.1"]
[cascalog-incanter "0.1.0"]
[cascalog-math "0.1.0"]

Contrib currently has three modules: cascalog.math, cascalog.incanter and cascalog.checkpoint. math and incanter are still fairly sparse, but checkpoint is quite interesting and battle-tested at Twitter. Read on if you're interested in the details of the checkpoint module; otherwise, I'll see you on the mailing list!

cascalog.contrib.checkpoint

The workflow macro in the checkpoint module allows you to break complicated workflows out into small, checkpointed steps. If one of these steps causes a job to fail and you restart the job, the workflow macro will skip every step up to the previous point of failure. Fault-tolerant MapReduce topologies ftw!

Let's look at the workflow macro in action. The following function takes an input-path to some existing Twitter data and an output-path, and executes a tweet-processing workflow with five steps:

(defn -main
  [input-path output-path]
  (workflow ["/tmp/example-checkpoint"]
            step-1     ([:tmp-dirs [staging-path]]
                          (transfer-tweets input-path staging-path))

            step-2     ([:deps :last :tmp-dirs user-path]
                          (harvest-users staging-path user-path))

            step-3a    ([:deps step-2 :tmp-dirs [cluster-path friend-path]]
                          (cluster-users user-path cluster-path)
                          (count-friends user-path friend-path))

            step-3b    ([:deps step-2 :tmp-dirs age-path]
                          (examine-ages user-path age-path))

            final-step ([:deps :all]
                          (big-analysis cluster-path
                                        friend-path
                                        age-path
                                        output-path))))

Let's look at this one piece at a time. The first argument to workflow is a vector with some path that the workflow can use to stage temporary files.

(workflow ["/tmp/example-checkpoint"] ...)

It doesn't matter what path you choose; just make sure that Hadoop has access and can write data to the folder.

Following this vector, workflow expects pairs of the form

step-name ([:deps <optional-deps, defaults to :last>]
             :tmp-dirs [<optional, creates temp-dirs if supplied>]
             ...<body, same as inside let>...)

Steps can identify other steps as dependencies by referencing their step-names with the :deps keyword argument.

The first step creates a temporary directory by supplying the symbol staging-path to the :tmp-dirs keyword argument. It then transfers tweets from the input directory into this staging directory, where they will remain available for future steps to consume.

step-1 ([:tmp-dirs [staging-path]]
          (transfer-tweets input-path staging-path))

Step 2 marks :last as a dependency. :last is the default, and marks the step as dependent only on the step directly above. A step will not execute until all of its dependencies have completed successfully.

step-2 uses staging-path defined in step-1 and creates a new temp directory (user-path) for its results.

If step-2 fails for any reason and you restart the workflow, the workflow macro will skip step-1, destroy any temporary directories created in the previous run of step-2, and start step-2 afresh.

step-2 ([:deps :last :tmp-dirs user-path]
          (harvest-users staging-path user-path))

The next two steps, step-3a and step-3b, each mark step-2 as a dependency. Once step-2 completes, step-3a and step-3b will run in parallel.

step-3a ([:deps step-2 :tmp-dirs [cluster-path friend-path]]
           (cluster-users user-path cluster-path)
           (count-friends user-path friend-path))

step-3b ([:deps step-2 :tmp-dirs age-path]
           (examine-ages user-path age-path))

The final step marks its dependencies as :all. This signifies that the step must wait for every step defined above it to complete before running. Again, if final-step fails and the workflow restarts, all previous successful steps will be skipped.

final-step ([:deps :all]
              (big-analysis cluster-path
                            friend-path
                            age-path
                            output-path))

In Conclusion

I'm quite excited about the Cascalog-contrib project and hope you all make heavy use of it as its functionality grows. In the short-term, I'm planning on hooking Cascalog in to Incanter's amazing visualization suite through the cascalog.incanter module.

Testing Cascalog with Midje

2011-09-30T00:00:00+00:00

30 Sept 2011 - San Francisco

Testing Cascalog with Midje

I've been working on a Cascalog testing suite these past few weeks, an extension to Brian Marick's Midje, that eases much of the pain of testing MapReduce workflows. I think a lot of the dull work we see in the Hadoop community is a direct result of fear. Without proper tests, Hadoop developers can't help but be scared of making changes to production code. When creativity might bring down a workflow, it's easiest to get it working once and leave it alone.

The antidote to all of this fear is a functional testing suite. As I discussed in Getting Creative with MapReduce, Hadoop workflows are difficult to test at all; testing application logic in isolation of data storage is impossible.

Cascalog is free of this weakness. midje-cascalog allows you to test Cascalog queries as pure functions, both in isolation and as components of more complicated workflows. the resulting tests are truly beautiful.

I'll start by introducing midje-cascalog's testing operators, then move on to a Cascalog implementation of Word Count, tests included. You can find all source code from this post on github.

Testing Operators

In this section, I'll discuss midje-cascalog's testing operators: fact?- and fact?<-. (The syntax mirrors ?- and ?<-, Cascalog's query execution operators.) These operators provide the abstractions necessary for testing complex Cascalog workflows. Add them to your namespace by including (:use midje.cascalog) in the namespace header.

fact?-

Let's begin by defining a function to test:

(defn mk-inc-query [src]
  (<- [?a ?b]
      (src ?a)
      (inc ?a :> ?b)))

mk-inc-query accepts a source of 1-tuples and returns a query that generates 2-tuples. To test that mk-inc-query actually does this, you need to:

supply mk-inc-query with tuples and
check that it produces an expected set of result tuples.

Each of the following forms uses the fact?- operator to state a distinct "fact" about our query. fact?- expects a sequence of result tuples followed by the query tasked with producing them.

These two facts about mk-inc-query are true, and pass:

;; The query returned by (mk-inc-query [[1]]),
;; when executed,
;; returns a single tuple: [1 2]
(fact?- [[1 2]]
        (mk-inc-query [[1]])) ;; fact is true!

;; The query returned by (mk-inc-query [[1] [10]]),
;; When executed,
;; returns two tuples: [10 11] and [1 2]
(fact?- [[10 11]
         [1 2]]
        (mk-inc-query [[1] [10]])) ;; fact is true!

This fact is false, and fails:

;; The query returned by (mk-inc-query [[1]]),
;; when executed,
;; returns a single tuple: ["fail!" 10].
(fact?- [["fail!" 10]]
        (mk-inc-query [[1]])) ;; fact is FALSE!

fact?- can take multiples pairs of result-tuples and queries:

;; Same as two true facts above.
(fact?- [[1 2]]
        (mk-inc-query [[1]])

        [[10 11] [1 2]]
        (mk-inc-query [[1] [10]])) ;; both facts are true!

Strings are ignored wherever they appear, so feel free to pepper your facts with comments.

(fact?- "These results:"
        [[1 2]]

        "Are produced by this query:"
        (mk-inc-query [[1]])) ;; true

Note that facts don't have to be top level forms. It's perfectly acceptable to wrap facts in let, if it makes the test clearer:

(let [src     [[1]]
      results [[1 2]]]
  (fact?- results (mk-inc-query src))) ;; true

Custom Log Levels

Cascalog pipes quite a bit of logging to stdout. Facts suppress this logging by default, only showing entries with a FATAL log level.

If you want to see more information on fact execution, you customize the log level by placing a keyword at the beginning of your fact:

(fact?- :info
        [[1 2]] (mk-inc-query [[1]])) ;; true

As of version 0.2.1, midje-cascalog supports the following log-level keywords, and defaults to :fatal:

:off
:fatal
:warn
:info
:debug

fact?<-

The fact?<- operator allows you to define a test a query within the same form. The following two facts are equivalent:

(let [src [[1]]]
  (fact?- [[1 2]]
          (<- [?a ?b]
              (src ?a)
              (inc ?a :> ?b)))) ;; true

(let [src [[1]]]
  (fact?<- [[1 2]]
           [?a ?b]
           (src ?a)
           (inc ?a :> ?b))) ;; true

Where fact?- is useful for testing full queries and workflows, I find fact?<- useful mostly for testing how def*op functions behave inside of queries.

future-fact?- and future-fact?<-

If you want to stub out an unfinished test and prevent it from throwing errors, you can use future-fact?-, like so:

(future-fact?- "unwritten-query will convert input integer tuples to
               strings."
               [["one"] ["two"]]
               (unwritten-query [[1] [2]]))

(let [src [[1] [2]]]
  (future-fact?<- "num->string is unwritten."
                  [["one"] ["two"]]
                  [?string]
                  (src ?num)
                  (num->string ?string)))

future-fact?- and future-fact?<- prevent their forms from being evaluated.

If you include a string at the beginning of a stubbed fact, it shows up in Midje's test report looking like this:

WORK TO DO: unwritten-query will convert input integer tuples to strings.
WORK TO DO: num->string is unwritten.

The fact?- and fact?<- operators provide the tools necessary to test complex MapReduce workflows as pure functions. Let's expand on these concepts by creating a small project with Cascalog code we'd like to test.

Example Project

Dependencies

To add midje-cascalog support to your own project, add these entries to to the :dev-dependencies vector within project.clj:

[lein-midje "1.0.3"]
[midje-cascalog "0.2.1"]

And add (:use [midje sweet cascalog]) to the namespace declaration of each of your testing namespaces.

Implementing Word Count

Let's begin with an implementation of word count, the typical "Hello World!" of MapReduce. A word counting application must be able to read in any number of textfiles and generate tuples of the form [word, count] for each distinct word across all files.

The following code accomplishes this nicely. (Bear with me! a detailed discussion follows the code block.)

(ns cascalog.testing-demo.core
  (:use cascalog.api)
  (:require [cascalog.ops :as c])
  (:gen-class))

(defmapcatop split
  "Accepts a sentence 1-tuple, splits that sentence on whitespace, and
  emits a single 1-tuple for each word."
  [^String sentence]
  (seq (.split sentence "\\s+")))

(defn wc-query
  "Returns a subquery that generates counts for every word in
    the text-files located at `text-path`."
  [text-path]
  (let [src (hfs-textline text-path)]
    (<- [?word ?count]
        (src ?textline)
        (split ?textline :> ?word)
        (c/count ?count))))

(defn -main
  "Accepts the following arguments:

   - text-path (path to a textfile, or directory with textfiles)
   - results-path (location of textfile containing results)

     And prints lines of the form \"word count\" to a textfile at
     results-path. Each distinct word in the textfiles at text-path
     gets a count."
  [text-path results-path]
  (?- (hfs-textline results-path)
      (wc-query text-path)))

The -main function is the entry point to the word counting program. -main passes text-path on to wc-query, and writes all tuples generated by the returned query to a text file at results-path.

All of our program's application logic occurs in the query returned by wc-query; this is the most important function to test. Let's discuss how wc-query works:

wc-query is a function that returns a subquery.
The function calls hfs-textline internally to generate a source of ?sentence tuples.
These sentences are passed into split, a Cascalog function that creates words from sentences, like this:

(let [sentence [["two words"]]
      words    [["two"] ["words"]]]
  (fact?<- "split converts a sentence into words."
           words
           [?word]
           (sentence ?sentence)
           (split ?sentence :> ?word)))

Each word gets a count via the cascalog.ops/count function
The subquery returns each [?word ?count] pair.

This logic looks right, but the only way to tell is to write a series of facts and see if they're true.

Testing Wordcount

Let's put our tests in ./test/cascalog/testing_demo/core_test.clj (mirroring the core.clj, with _test tacked on):

(ns cascalog.testing-demo.core-test
  (:use cascalog.testing-demo.core
        cascalog.api
        [midje sweet cascalog])
  (:require [cascalog.ops :as c]))

Here's an initial try at a test of wc-query using fact?-:

;; /path/to/textfile points to a textfile with a single line:
;; "another another word"
(fact?- "wc-query should count words from all lines of text at
        /path/to/textfile."
        [["word" 1] ["another" 2]]
        (wc-query "/path/to/textfile")) ;; FALSE!

This fact fails. Here are a few of its problems:

The fact depends on the way tuples are stored; it depends on an outside textfile located at a hard-coded path. If the textfile disappears, the fact will fail whether or not the logic of wc-query is correct.
The fact depends on the correctness of hfs-textline. if hfs-textline fails, our fact fails.

Testing wc-query in isolation is difficult! How can one test the logic of wc-query- without regard to how lines of text are stored?

Mocking with Midje

The solution lies in Midje's ability to mock out a function's return values. Midje can hijack hfs-textline and force it to return anything you choose inside the body of a fact.

provided

Using Midje's provided form, the above fact passes:

(fact?- "wc-query should count words from all input sentences."
        [["word" 1] ["another" 2]]
        (wc-query :path)
        (provided
          (hfs-textline :path) => [["another another word"]])) ;; true

This fact states

when wc-query is called with :path,
it will produce two tuples: ["word" 1] and ["another" 2],
provided (hfs-textline :path) produces a single tuple: ["another another word"].

Here's another true fact about wc-query that uses multiple input sentences:

(def short-sentences
  [["this is a sentence sentence"]
   ["sentence with this is repeated"]])

(def short-wordcounts
  [["sentence" 3]
   ["repeated" 1]
   ["is" 2]
   ["a" 1]
   ["this" 2]
   ["with" 1]])

;; when =wc-query= is called with =:text-path=
;; it will produce =short-sentences=,
;; provided =(hfs-textline :text-path)= produces =short-wordcounts=.
(fact?- short-wordcounts (wc-query :text-path)
        (provided
          (hfs-textline :text-path) => short-sentences)) ;; true

A provided form only applies to the result-query pair directly above. The first fact is false, while the second fact is true:

(let [sentence [["two words"]]
      results  [["two" 1] ["words" 1]]]
  (fact?- "provided form won't apply here!"
          results (wc-query :path) ;; false

          "provided applies here."
          results (wc-query :path) ;; true
          (provided
            (hfs-textline :path) => sentence)))

Mocking Arguments

In the above facts, I used keywords (:path) as mocking arguments. Any form that evaluates to itself can be used as a mocking argument. In vanilla Clojure, this includes strings, numbers and keywords. Midje adds any symbol surrounded by dots (..path.., .path., etc.) to this mix.

These facts about wc-query from above are all true, and identical:

(fact?- "Mocking with keywords,"
        [["one" 1]] (wc-query :path)
        (provided (hfs-textline :path) => [["one"]])

        "strings,"
        [["one" 1]] (wc-query "path")
        (provided (hfs-textline "path") => [["one"]])

        "numbers,"
        [["one" 1]] (wc-query 100)
        (provided (hfs-textline 100) => [["one"]])

        "and Midje dotted symbols."
        [["one" 1]] (wc-query ..path..)
        (provided (hfs-textline ..path..) => [["one"]]))

against-background

As discussed, the provided form only applies to the result-query pair directly above. This limitation can make for repetitive facts, when each fact depends on a mocked result:

(defn text->words [path]
  (let [src (hfs-textline path)]
    (<- [?word]
        (src ?sentence)
        (split ?sentence :> ?word)
        (:distinct false))))

(let [sentence [["two two"]]]
  (fact?- "text->words cuts text into words."
          [["two"] ["two"]] (text->words :path)
          (provided
            (hfs-textline :path) => sentence)

          "wc-query converts a sentence into words."
          [["two" 2]] (wc-query :path)
          (provided
            (hfs-textline :path) => sentence)))

Midje allows facts to share mocked functions with against-background. An against-background form placed anywhere inside the body of fact?- will apply to all facts inside the form:

(let [sentence [["two two"]]]
  (fact?- "text->words cuts text into words."
          [["two"] ["two"]] (text->words :path)

          "wc-query converts a sentence into words."
          [["two" 2]]
          (wc-query :path)

          "wc-query fact with difference inputs."
          [["what" 1] ["a" 1] ["world!" 1]]
          (wc-query :path)
          (provided
            (hfs-textline :path) => [["what a world!"]])

          (against-background
            (hfs-textline :path) => sentence)))

Note that the third of the three above facts used its own provided form. When the two forms are mixed, provided takes precedence, shadowing against-background if need be (as above).

Collection Checkers

For the next set of facts, let's introduce a larger set of input sentences:

(def longer-sentences
  [["Call me Ishmael. Some years ago -- never mind how long"]
   ["precisely -- having little or no money in my purse, and"]
   ["nothing particular to interest me on shore, I thought I"]
   ["would sail about a little and see the watery part of the world."]])

One issue with the above facts is that they use very small input sentences. wc-query will produce a rather large sequence of <word, count> pairs for a moderate number of input sentences. Facts like this are overwhelming:

(fact?- [["Ishmael." 1]
         ["Some" 1]
         ["a" 1]
         ["about" 1]
         ["ago" 1]
         ;; and on and on...
         ]
        (wc-query :path)
        (provided (hfs-textline :path) => longer-sentences))

To solve this, Midje provides a number of collection checkers that provide you with finer control over how queries are compared with result sequences.

just

just is the default checker for fact?- and fact?<-; bare vectors of tuples resolve to (just result-vec :in-any-order). The following three facts are equivalent:

(let [src   [[1] [2]]
      query (<- [?a ?b]
                (src ?a)
                (inc ?a :> ?b))]
  (fact?- "Just form, fully qualified."
          (just [[2 3] [1 2]] :in-any-order) query ;;true

          "Wrapping tuples in a set is identical to including
           the :in-any-order modifier."
          (just #{[2 3] [1 2]}) query ;; true

          "midje-cascalog lets us drop these wrappers."
          [[2 3] [1 2]] query)) ;; true

Each of these facts checks that its subquery returns [2 3] [1 2] exclusively, in any order. Any missing or extra tuples in the result vector will cause a failure.

Note that dropping the :in-any-order modifier (or the set wrapper) will cause facts to fail if ordering doesn't match. This makes sense sometimes when checking against top-n queries, as noted in the discussion below on has-prefix.

contains

The contains form allows facts to check against a subset of query tuples. By default, contains requires result tuples to be contiguous and ordered: [1 2] within [3 4 1 2 1], for example.

These restrictions are quite limiting for most Cascalog queries. The following two facts avoid both restrictions:

(fact?- (contains #{["sail" 1] ["Ishmael." 1]} :gaps-ok)
        (wc-query :path) ;; true

        (contains [["sail" 1] ["Ishmael." 1]] :gaps-ok :in-any-order)
        (wc-query :path) ;; true

        (against-background
          (hfs-textline :path) => longer-sentences))

The above facts test that both ["sail" 1] and ["Ishmael." 1] appear somewhere in the results, in any order.

Wrapping the result tuples in a set (vs. a vector), or adding the :in-any-order keyword, relaxes the ordering restriction.
The :gaps-ok keyword relaxes the restriction that tuples must contiguous.

has-prefix

has-prefix checks that the supplied tuple sequence appears at the beginning of the query's results. has-prefix only makes sense with queries that return sorted tuples.

The following fact states that ["--" 2], ["I" 2] and ["and" 2], in order, are the three most common words across all words in longer-sentences:

(fact?- (has-prefix [["--" 2] ["I" 2] ["and" 2]])
        (-> (wc-query :path)
            (c/first-n 10 :sort ["?count"] :reverse true))
        (provided
          (hfs-textline :path) => longer-sentences)) ;; true

has-suffix

has-suffix checks that the supplied tuple sequence appears at the end of the query's results.

The following fact states that ["world." 1], ["would" 1] and ["years" 2], in order, are the last three words (by alphabetical order) across all words in longer-sentences:

(fact?- (has-suffix [["world." 1] ["would" 1] ["years" 1]])
        (-> (wc-query :text-path)
            (c/first-n 100 :sort ["?word"]))
        (provided
          (hfs-textline :text-path) => longer-sentences)) ;; true

As with has-prefix, facts making use of has-suffix only make sense when specifically testing tuple ordering.

Tabular

In certain cases, you might like to test a single query against a wide range of inputs and outputs. This quickly grows repetitive:

(fact?- [["mock" 1] ["it" 1] ["out!" 1]]
        (wc-query :path)
        (provided
          (hfs-textline :path) => [["mock it out!"]]) ;;true

        [["two" 3]]
        (wc-query :path)
        (provided
          (hfs-textline :path) => [["two two two"]]) ;;true

        [["M.M" 1] ["nathan" 1]]
        (wc-query :path)
        (provided
          (hfs-textline :path) => [["nathan M.M"]])) ;; true

Gah! against-background doesn't work here, since these facts mock against different sentences each time.

Midje's tabular form provides an elegant way to collapse this repetition:

(tabular
 (fact?- "Tabular generates lots of facts, one for each set of
         substitutions in the table below."
         ?results
         (wc-query :path)
         (provided
           (hfs-textline :path) => [[?sentence]]))
 ?sentence       ?results
 "mock it out!"  [["mock" 1] ["it" 1] ["out!" 1]]
 "two two two"   [["two" 3]]
 "nathan M.M"    [["M.M" 1] ["nathan" 1]]) ;; 3 true facts

(This one's a little involved, but the results are really beautiful.)

tabular accepts three types of arguments:

a single fact?- or fact?<- templating form
a number of "templating variables" that start with ? (?sentence and ?results, in the above fact)
any number of rows of substitutions (the above fact has three)

and generates a separate fact for every substitution row. It does this by substituting each value into the templating form in place of the header variable at the top of column.

The first fact generated by the above tabular fact looks like this:

(tabular
 ;; Tabular takes this templating form:
 (fact?- "Tabular generates lots of facts, one for each set of
           substitutions in the table below."
         ?results
         (wc-query :path)
         (provided
           (hfs-textline :path) => [[?sentence]]))

 ;; and substitutes these variables:
 ?sentence       ?results
 "mock it out!"  [["mock" 1] ["it" 1] ["out!" 1]]) ;; true

;; to produce this fact:
(fact?- [["mock" 1] ["it" 1] ["out!" 1]]
        (wc-query :path)
        (provided
          (hfs-textline :path) => [["mock it out!"]])) ;; true

Any variable prefixed by ? that appears inside both the fact template AND the header variables row is earmarked for substitution. This means that cascalog dynamic variables are totally safe, and play well with tabular.

Running Tests

lein-midje

Once you write facts within a project, you can use lein-midje to run them all and generate a summary like this:

Checking function: (midje.sweet/just [["Ishmael." 1] ["Some" 1] ["a" 1] ["about" 1] ["ago" 1]] :in-any-order)
The checker said this about the reason:
    Expected five elements. There were thirty-nine.
FAILURE: 6 facts were not confirmed. (But 37 were.)

If you're using the leiningen build manager, follow these steps:

Add [lein-midje "1.0.7"] to the :dev-dependencies entry in your project.clj
Run lein midje at the command line in your project's root directory.

This command runs all facts and tests in the project and prints a summary of all results to stdout.

If you're using Cake, follow the steps on the Midje wiki for installing and running cake midje.

Interaction with clojure.test

If you currently write deftest style tests using clojure.test, check out Midje's tips on integration. The two modes work very well together. lein midje and cake midje will evaluate all deftest forms inside of a project and include the results in its report.

In Conclusion

I believe that midje-cascalog is the most advanced MapReduce testing suite available today. The primitives discussed here make testing Cascalog queries a joy; the confidence that comes from fully tested components is a prerequisitive for creative work at large scale.

Please let me know what you think of the project! I'm happy to extend midje-cascalog in any way that helps the cause. Have fun testing!

Getting Creative with MapReduce

2011-09-29T00:00:00+00:00

29 Sept 2011 - San Francisco

Getting Creative with MapReduce

One problem with many existing MapReduce abstraction layers is the utter difficulty of testing queries and workflows. End-to-end tests are maddening to craft in vanilla Hadoop and frustrating at best in Pig and Hive. The difficulty of testing MapReduce workflows makes it scary to change code, and destroys your desire to be creative. A proper testing suite is an absolute prerequisite to doing creative work in big data.

In this blog post, I aim to show how most of the difficulty of writing and testing MapReduce queries stems from the fact that Hadoop confounds application logic with decisions about data storage. These problems are the result of poorly implemented abstractions over the primitives of MapReduce, not problems with the core MapReduce algorithms.

The Cascalog abstraction layer fixes this issue by separating logic from data, allowing you to play creatively at massive scale. (If you're not familiar with Cascalog, know that it's by far the most expressive MapReduce DSL in existence. We make heavy use of Cascalog here at Twitter. I recommend getting started with this introduction, then moving on to the wiki.)

Cascalog and its testing suite, midje-cascalog, allow us to test application logic in isolation; the resulting tests are truly beautiful.

Current Approaches to MR Testing

Say you've implemented wordcount in MapReduce, and are looking to write unit tests against your workflow. The Hadoop consulting giant Cloudera defines current best practices in this article:

The current state of the art often involves writing a set of tests that each create a JobConf object, which is configured to use a mapper and reducer, and then set to use the LocalJobRunner (via JobConf.set(”mapred.job.tracker”, “local”)). A MapReduce job will then run in a single thread, reading its input from test files stored on the local filesystem and writing its output to another local directory.

This process provides a solid mechanism for end-to-end testing, but has several drawbacks. Developing new tests requires adding test inputs to files that are stored alongside one’s program. Validating correct output also requires filesystem access and parsing of the emitted data files. This involves writing a great deal of test harness code, which itself may contain subtle bugs. Finally, this process is slow. Each test requires several seconds to run. Users often find themselves aggregating several unrelated inputs into a single test (violating a unit testing principle of isolating unrelated tests) or performing less exhaustive testing due to the high barriers to test authorship.

This is insane. Written this way, unit tests meant to target application logic are dwarfed by the maintenance they impose.

Most real world MapReduce applications involve chained sequences of many queries; these flows are a nightmare to coordinate and test using the above methods. Perhaps for this reason, Cloudera recommends testing mappers and reducers separately:

your map() and reduce() calls should still be tested individually, as the composition of separate classes may cause unintended bugs to surface.

Again, this is totally batshit crazy. MapReduce queries are functions that operate on datasets. Imagine writing a Clojure function and testing its lines one at a time, in isolation; the approach makes no sense.

A well designed testing suite will examine each MapReduce query as the function that it is. Queries should be tested against a range of inputs, both in isolation and as components of more complicated workflows. It shouldn't matter how one's data is stored; query logic is independent of data storage, and should be tested as such.

I'd like to offer a more sane approach to MapReduce testing. In the following example, I'll write MapReduce query with Cascalog and test it using midje-cascalog, a thin wrapper I wrote over the Midje testing DSL.

Functional MapReduce Testing

Let's say you want to test a Cascalog workflow that examines your user datastore and returns the user with the greatest number of followers. Your workflow's top level query will generate a single tuple containing that user's name and follower-count. Here's the code:

(defn max-followers-query [datastore-path]
  (let [src (name-vars (complex-subquery datastore-path)
                       ["?user" "?follower-count"])]
    (cascalog.ops/first-n src 1 :sort ["?follower-count"] :reverse true)))

max-followers-query is a function that returns a Cascalog subquery. It works like this:

The function accepts a path, (datastore-path) and passes it into a function called complex-subquery.
complex-subquery returns a subquery that generates 2-tuples; this subquery is passed into name-vars.
- name-vars binds this subquery to src after naming its output variables ?user and ?follower-count.
first-n returns a subquery that
- sorts tuples from src in reverse order by follower count, and
- returns a single 2-tuple with the name and follower-count of our most popular user.

At a high level, the subquery returned by max-followers-query is responsible for a single piece of application logic:

extracting the tuple with max ?follower-count from the tuples returned by (complex-subquery datastore-path).

A correct test of max-followers-query will test this piece of logic in isolation.

Cloudera's "State of the Art"

If you were to follow Cloudera's advice on how to test this query, you would have to:

Dig through the code of complex-subquery, and figure out what data it depends on to produce its tuples;
Set up temporary directories and SequenceFiles with appropriate starting tuples;
Execute the subquery returned by max-followers-query into a temporary SequenceFile;
Read tuples out of this SequenceFile and test them against expected values;
Clean up all temporary paths and directories.

This series of steps adds tremendous friction to the testing process, and obscures the purpose of the test. Moreover, the results of any test of max-followers-query depend on any number of subqueries invoked the call to complex-subquery.

If your tests are this difficult to write, you're not going to write very many tests. You need tests to be creative; without tests, you won't change production queries in fear of introducing some bug you don't understand.

Fact-Based Testing

Midje circumvents all of this complexity by mocking out the result of (complex-subquery datastore-path) and forcing it to return a specific Clojure sequence of [?user ?follower-count] tuples.

The following form is a Midje test, or "fact":

(fact?- "Query should return a single tuple containing
        [most-popular-user, follower-count]."
        [["richhickey" 2961]]
        (max-followers-query :path)
        (provided
          (complex-subquery :path) => [["sritchie09" 180]
                                       ["richhickey" 2961]]))

Facts make statements about queries. The fact passes if these statements are true and fails otherwise. The above fact states that

when max-followers-query is called with the argument :path,
it will produce [ [ richhickey" 2961] ],
provided (complex-subquery :path) produces [["sritchie09" 180] ["richhickey" 2961]].

Fact-based testing separates application logic from the way data is stored. By mocking out complex-subquery, our fact tests max-followers-query in isolation and proves it correct for all expected inputs.

This approach is not just better than the "state of the art" of MapReduce testing as defined by Cloudera; it completely obliterates the old way of thinking, and makes it possible to build very complex workflows with a minimum of uncertainty.

Fact-based tests are the building blocks of rock-solid production workflows.

If you're interested in how to construct fact-based tests with midje-cascalog, I go into great detail on functional MapReduce testing idioms and methods in Testing Cascalog with Midje.

Cascalog 1.8.1 Released

2011-09-26T00:00:00+00:00

26 Sept 2011 - San Francisco

Cascalog 1.8.1 Released

Nathan Marz and I are releasing Cascalog 1.8.1 today! We've added a few interesting features, and I thought I'd provide a bit more detail here for anyone interested.

Cross Join

cascalog.api now includes support for cross-joins; just add (cross-join) to your query as its own predicate.

Think of a cross-join as a "tuple comprehension", or cartesian product, with similar results to clojure.core/for; it's not very efficient, as it forces all tuples through a single reducer (and causes a massive blowup in the number of tuples!). Here's an example:

(let [a-src [[1] [2]]
      b-src [[3] [4]]]
  (?<- (stdout)
       [?a ?b]
       (a-src ?a)
       (b-src ?b)
       (cross-join)))

This results in the following on stdout:

RESULTS
-----------------------
1       3
1       4
2       3
2       4
-----------------------

If you're interested, here's the implementation:

(def cross-join
  (<- [:>] (identity 1 :> _)))

This is the only predicate macro I know of that can get away with no input OR output vars.

defmain

When running a cascalog query on a cluster, usual practice is to include (:gen-class) in the namespace form, and write a -main method that gets AOT-compiled and called by Hadoop. This can be a little annoying, if you have a bunch of small queries you want to run.

defmain lets you skip the :gen-class form and write something like:

;; inside myproject.jobs...  
(defmain FirstQuery [in-path out-path]
  (?- (hfs-textline out-path)
      (some-query in-path)))

(defmain SecondQuery [out-path]
  (?<- (hfs-textline out-path)
       [?a ?b] ...))

Each defmain will compile to a class with the supplied name, prefixed by the namespace. (myproject.jobs.FirstQuery and myproject.jobs.SecondQuery, in this example.)

(As always, make sure to add the :aot [myproject.jobs], kv-pair to project.clj, including each namespace containing a defmain or (:gen-class) in the vector. If you want to call some defmain function directly, (defmain Query ...) can be called from the REPL with (Query-main ...).) I recommend keeping your defmain functions skinny, and testing the components it calls.

with-serializations

Damn you, serializations. This one JobConf entry, "io.serializations", has caused me much pain. We've added with-serializations, which makes the supplied Hadoop serializations available to all queries enclosed within the form. These forms nest, and play well with the existing with-job-conf. For example:

;; You can specify serializations in string form...
(with-serializations ["org.apache.hadoop.io.serializer.JavaSerialization"]
  (<- [?a] ...))

;; ... or directly, with the class.
(import 'org.apache.hadoop.io.serializer.JavaSerialization)

(with-serializations [JavaSerialization]
  (<- [?a] ...))

;; Serializations nest and unique against each other!
(with-job-conf {"io.serializations" "my.ns.SomeSerialization"}
  (with-serializations [JavaSerialization OtherSerialization]
    (with-serializations ["my.ns.SomeSerialization" ThirdSerialization]
      (<- [?a] ...))))

cascalog.ops/first-n

first-n can now handle straight-up vectors, lists, and cascading taps, in addition to queries.

Say we've previously run a wordcount job that output [?word ?count] 2-tuples to a sequencefile, and we want to pull the top 100 words by count. Here's how we do that with first-n:

(use 'cascalog.api)
(require '[cascalog.ops :as c])

(defn wordcount-tap [path]
  (-> (hfs-seqfile path)
      (name-vars ["?word" "?count"])))

(defn top-100 [file-path]
  (c/first-n (wordcount-tap path)
             100
             :sort ["?count"]
             :reverse true))

(defmain Top100 [tuple-path results-path]
  (?- (hfs-textline results-path)
      (top-100 tuple-path)))

first-n with vectors and lists is mostly interesting for testing purposes.

Other Bugfixes

Just a few bugfixes to note:

Fixed a bug preventing cascalog-taps in (:trap (some-tap ...)) option predicates.
Fixed bug preventing keywords as static arguments (as demonstrated here).
Failed cascading flows now always throw errors. (This is a workaround to a Cascading bug that allows some flows to fail silently.)

Haskell in Emacs

2011-09-25T00:00:00+00:00

Sept 25 2011 - San Francisco

Haskell in Emacs

I spent some time today getting my emacs config set up to learn Haskell, and ran into a few issues; I figured I'd go ahead and document the process here for everyone's enjoyment. We're going to install and configure Haskell mode, then add a few extensions that'll make learning Haskell fun and easy!

I'm currently running haskell-mode for emacs, with the hs-lint plugin, Haskell support for FlyMake (which provides on-the-fly syntax checking from the Haskell compiler), and code autocompletion. The steps covered by this tutorial are:

Installing Haskell
Configuring Haskell-Mode for Emacs
Installing Haskell-Mode Extensions (Flymake support, hs-lint and autocompletion)

Installing Haskell

Before any of this Emacs jazz, we have to get Haskell, of course. The easiest way to do is to download the Haskell Platform, a "Batteries Included" package of the Glasgow Haskell Compiler.

Emacs Haskell Mode

The Haskell Mode for Emacs page at the Haskell wiki is a nice place to start your explorations, and describes how to install haskell-mode. I found it easier to use ELPA, the Emacs Lisp Package Archive (install instructions here). If you're using the Emacs starter kit, you've already got ELPA.

Once elpa's all set, install haskell-mode with the following:

run M-x package-list-packages in Emacs
tap i by haskell-mode
hit x to the start the install.

Configuring emacs.el

Before we get into linting or any other customizations, Add the following to your emacs config (~/.emacs, or ~/.emacs.d/init.el, if you're using the Starter kit):

(add-hook 'haskell-mode-hook 'turn-on-haskell-doc-mode)

;; hslint on the command line only likes this indentation mode;
;; alternatives commented out below.
(add-hook 'haskell-mode-hook 'turn-on-haskell-indentation)
;;(add-hook 'haskell-mode-hook 'turn-on-haskell-indent)
;;(add-hook 'haskell-mode-hook 'turn-on-haskell-simple-indent)

;; Ignore compiled Haskell files in filename completions
(add-to-list 'completion-ignored-extensions ".hi")

At this point, you should be able to start using Haskell in emacs. Let's write our first function.

Create a file called test.hs.
type C-c C-z to get to the Haskell REPL, supplied by inf-haskell mode
Now, add the following to test.hs:

module Examples where

square :: Integral a => a -> a
square x = x * x

Then type C-c C-l in that buffer to load the file's contents into the REPL. C-c C-z over to the repl again and try it out:

*Main> square 10
100
*Main> square 34
1156

Nice.

Flymake

Next, we're going to add support for Flymake, the emacs syntax-checker. This is a piece of Emacs functionality that'll run our Haskell files through the compiler every few seconds and associate any warnings and errors with some specific line in our Haskell file.

While we're at it, we'll configure hs-lint, which we can run periodically on our Haskell files for a list of hints as to how to structure code better. For example, running hs-lint on a Haskell file containing times2 x = (*) 2 x yields a buffer with:

hlint /Users/sritchie/test.hs
/Users/sritchie/test.hs:1:1: Error: Eta reduce
Found:
  times2 x = (*) 2 x
Why not:
  times2 = (*) 2

 1 suggestion

Definitely helpful in the learning process.

Flymake depends on an external perl script, hslint, to pass the contents of the current buffer into the Haskell compiler.

Download hslint from this gist and place it somewhere on your path.
run chmod a+x hslint to give the file executable privileges
Add the following to .emacs:

(defun flymake-haskell-init ()
  "When flymake triggers, generates a tempfile containing the
  contents of the current buffer, runs `hslint` on it, and
  deletes file. Put this file path (and run `chmod a+x hslint`)
  to enable hslint: https://gist.github.com/1241073"
  (let* ((temp-file   (flymake-init-create-temp-buffer-copy
                       'flymake-create-temp-inplace))
         (local-file  (file-relative-name
                       temp-file
                       (file-name-directory buffer-file-name))))
    (list "hslint" (list local-file))))

(defun flymake-haskell-enable ()
  "Enables flymake-mode for haskell, and sets <C-c d> as command
  to show current error."
  (when (and buffer-file-name
             (file-writable-p
              (file-name-directory buffer-file-name))
             (file-writable-p buffer-file-name))
    (local-set-key (kbd "C-c d") 'flymake-display-err-menu-for-current-line)
    (flymake-mode t)))

;; Forces flymake to underline bad lines, instead of fully
;; highlighting them; remove this if you prefer full highlighting.
(custom-set-faces
 '(flymake-errline ((((class color)) (:underline "red"))))
 '(flymake-warnline ((((class color)) (:underline "yellow")))))

Haskell Extensions

Auto Complete Mode

Now, let's add autocompletion. Autocomplete mode is awesome; it provides IDE-like word tab completion of words based on info in open buffers, and some knowledge of the modes of the emacs buffer you're currently working in. In Haskell, we'll get autocompletion of every function we define, plus help with core language constructs. Head over to Auto Complete Mode to download the package, and install with the following:

Download and unpack Autocomplete mode
Open emacs, and run M-x load-file
Point the minibuffer to <autocomplete-root>/etc/install.el
Follow the remaining AC install instructions.

That should get you all set for the next step.

Linting!

Download hs-lint.el and haskell-ac.el and place each file inside of ~/.emacs.d. (hs-lint is our linter, of course, and haskell-ac.el provides autocomplete mode with some knowledge of a few core Haskell constructs.
Add the following to your .emacs file:

(require 'hs-lint)    ;; https://gist.github.com/1241059
(require 'haskell-ac) ;; https://gist.github.com/1241063

(defun my-haskell-mode-hook ()
  "hs-lint binding, plus autocompletion and paredit."
  (local-set-key "\C-cl" 'hs-lint)
  (setq ac-sources
        (append '(ac-source-yasnippet
                  ac-source-abbrev
                  ac-source-words-in-buffer
                  my/ac-source-haskell)
                ac-sources))
  (dolist (x '(haskell literate-haskell))
    (add-hook
     (intern (concat (symbol-name x)
                     "-mode-hook"))
     'turn-on-paredit)))

(eval-after-load 'haskell-mode
  '(progn
     (require 'flymake)
     (push '("\\.l?hs\\'" flymake-haskell-init) flymake-allowed-file-name-masks)
     (add-hook 'haskell-mode-hook 'flymake-haskell-enable)
     (add-hook 'haskell-mode-hook 'my-haskell-mode-hook)))

The above code binds C-c l to hs-lint inside of Haskell buffers, and configures a large number of Haskell keywords for autocompletion. Go and test it out inside of test.hs; you should find that square autocompletes when you begin typing.

Now go ahead and add the following line to test.hs:

face :: Int -> Bool

Upon save, or within seconds, you should see an angry red underline. Move the cursor over that line and type C-c d, and you'll see a tooltip with the following text:

1: The type signature for `face' lacks an accompanying binding.

Adding this will clear things up:

face x = 5 < x

Finishing Up

I hope this helped those of you looking to get started exploring Haskell! Please let me know in the comments if anything could be clearer; I'll be posting more down the road, and all requests are welcome.

Simple Hadoop Clusters

2011-05-31T00:00:00+00:00

31 May 2011 - Washington, DC

Introducing Pallet-Hadoop

I'm excited to announce Pallet-Hadoop, a configuration library written in Clojure for Apache's Hadoop.

In the tutorial, we're going to see how to create a three node Hadoop cluster on EC2, and run a word count on MapReduce. We'll be following along with Pallet-Hadoop example project for the introduction; for a more in-depth discussion of the design of pallet-hadoop, see the project wiki.

Background

Hadoop is an Apache java framework that allows for distributed processing of enormous datasets across large clusters. It combines a computation engine based on MapReduce with HDFS, a distributed filesystem based on the Google File System.

Abstraction layers such as Cascading (for Java) and Cascalog (for Clojure) make writing MapReduce queries quite nice. Indeed, running hadoop locally with cascalog couldn't be easier.

Unfortunately, graduating one's MapReduce jobs to the cluster level isn't so easy. Amazon's Elastic MapReduce is a great option for getting up and running fast; but what to do if you want to configure your own cluster?

After surveying existing tools, I decided to write my own layer over Pallet, a wonderful cloud provisioning library written in Clojure. Pallet runs on top of jclouds, which allows pallet to define its operations independent of any one cloud provider. Switching between clouds involves a change of login credentials, nothing more.

Setting Up

Before you get your first cluster running, you'll need to create an AWS account. Once you've done this, navigate to your account page and follow the "Security Credentials" link. Under "Access Credentials", you should see a tab called "Access Keys". Note down your Access Key ID and Secret Access Key for future reference.

I'm going to assume that you have some basic knowledge of clojure, and know how to get a project running using leiningen or cake. Go ahead and clone the example project to follow along:

$ git clone git://github.com/pallet/pallet-hadoop-example.git
$ cd pallet-hadoop-example

Open up ./src/pallet-hadoop-example/core.clj with your favorite text editor. example-cluster contains a data description of a full hadoop cluster with:

One master node functioning as jobtracker and namenode
Two slave nodes ((slave-group 2)), each acting as datanode and tasktracker.

Start a repl:

$ lein repl
 user=> (use 'pallet-hadoop-example.core) (bootstrap)

Compute Service

Pallet abstracts away details about specific cloud providers through the idea of a "compute service". The combination of our cluster definition and our compute service will be enough to get our cluster running. We define a compute service at our REPL like so:

user=> (def ec2-service
         (compute-service "aws-ec2"
                          :identity "ec2-access-key-id" ;; Swap in your access key ID
                          :credential "ec2-secret-access-key")) ;; Swap in your secret key
#'pallet-hadoop-example.core/ec2-service

Alternatively, if you want to keep these out of your code base, save the following to ~/.pallet/config.clj:

(defpallet
  :services {:aws {:provider "aws-ec2"
                   :identity "your-ec2-access-key-id"
                   :credential "your-ec2-secret-access-key"}})

and define ec2-service with:

user=> (def ec2-service (compute-service-from-config-file :aws))
#'user/ec2-service

Booting the Cluster

Now that we have our compute service and our cluster defined, booting the cluster is as simple as the following:

user=> (create-cluster example-cluster ec2-service)

The logs you see flying by are Pallet's SSH communications with the nodes in the cluster. On node startup, Pallet uses your local SSH key to gain passwordless access to each node, and coordinates all configuration using streams of SSH commands.

Once create-cluster returns, we're done! We now have a fully configured, multi-node Hadoop cluster at our disposal.

Running Word Count

To test our new cluster, we're going log in and run a word counting MapReduce job on a number of books from Project Gutenberg.

Point your browser to the EC2 Console, log in, and click "Instances" on the left.

You should see three nodes running; click on the node whose security group contains "jobtracker", and scroll the lower pane down to retrieve the public DNS address for the node. It'll look something like ec2-50-17-103-174.compute-1.amazonaws.com. I'll refer to this address as jobtracker.com.

Point your browser to jobtracker.com:50030, and you'll see the JobTracker web console. (Keep this open, as it will allow us to watch our MapReduce job in action.).=jobtracker.com:50070= points to the NameNode console, with information about HDFS.

Next, we'll SSH into the jobtracker, and operate as the hadoop user. Head to your terminal and run the following commands:

$ ssh jobtracker.com # insert actual address, enter yes to continue connecting
$ sudo su - hadoop

Copy Data to HDFS

At this point, we're ready to begin following along with Michael Noll's excellent Hadoop configuration tutorial. (I'll cover some of the same ground for clarity.)

Our first step will be to collect a bunch of text to process. We start by downloading the following seven books to a temp directory:

Running the following commands at the remote shell should do the trick.

$ mkdir /tmp/books
$ cd /tmp/books
$ curl -O http://www.gutenberg.org/cache/epub/20417/pg20417.txt
$ curl -O http://www.gutenberg.org/cache/epub/5000/pg5000.txt
$ curl -O http://www.gutenberg.org/cache/epub/4300/pg4300.txt
$ curl -O http://www.gutenberg.org/cache/epub/132/pg132.txt
$ curl -O http://www.gutenberg.org/cache/epub/1661/pg1661.txt
$ curl -O http://www.gutenberg.org/cache/epub/972/pg972.txt
$ curl -O http://www.gutenberg.org/cache/epub/19699/pg19699.txt

Next, navigate to the Hadoop directory:

$ cd /usr/local/hadoop-0.20.2/

And copy the books over to the distributed filesystem:

/usr/local/hadoop-0.20.2$ hadoop dfs -copyFromLocal /tmp/books books
/usr/local/hadoop-0.20.2$ hadoop dfs -ls
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2011-06-01 06:12:21 /user/hadoop/books
/usr/local/hadoop-0.20.2$

Running MapReduce

We're ready to run the MapReduce job. wordcount takes an input path within HDFS, processes all items within, and saves the output to HDFS – to books-output, in this case. Run this command:

/usr/local/hadoop-0.20.2$ hadoop jar hadoop-examples-0.20.2-cdh3u0.jar wordcount books/ books-output/

And you should see something very similar to this:

11/06/01 06:14:30 INFO input.FileInputFormat: Total input paths to process : 7
11/06/01 06:14:30 INFO mapred.JobClient: Running job: job_201106010554_0002
11/06/01 06:14:31 INFO mapred.JobClient:  map 0% reduce 0%
11/06/01 06:14:44 INFO mapred.JobClient:  map 57% reduce 0%
11/06/01 06:14:45 INFO mapred.JobClient:  map 71% reduce 0%
11/06/01 06:14:46 INFO mapred.JobClient:  map 85% reduce 0%
11/06/01 06:14:48 INFO mapred.JobClient:  map 100% reduce 0%
11/06/01 06:14:57 INFO mapred.JobClient:  map 100% reduce 33%
11/06/01 06:15:00 INFO mapred.JobClient:  map 100% reduce 66%
11/06/01 06:15:01 INFO mapred.JobClient:  map 100% reduce 100%
11/06/01 06:15:02 INFO mapred.JobClient: Job complete: job_201106010554_0002
11/06/01 06:15:02 INFO mapred.JobClient: Counters: 22
11/06/01 06:15:02 INFO mapred.JobClient:   Job Counters
11/06/01 06:15:02 INFO mapred.JobClient:     Launched reduce tasks=3
11/06/01 06:15:02 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=74992
11/06/01 06:15:02 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
11/06/01 06:15:02 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
11/06/01 06:15:02 INFO mapred.JobClient:     Launched map tasks=7
11/06/01 06:15:02 INFO mapred.JobClient:     Data-local map tasks=7
11/06/01 06:15:02 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=46600
11/06/01 06:15:02 INFO mapred.JobClient:   FileSystemCounters
11/06/01 06:15:02 INFO mapred.JobClient:     FILE_BYTES_READ=1610042
11/06/01 06:15:02 INFO mapred.JobClient:     HDFS_BYTES_READ=6557336
11/06/01 06:15:02 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=2753014
11/06/01 06:15:02 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=1334919
11/06/01 06:15:02 INFO mapred.JobClient:   Map-Reduce Framework
11/06/01 06:15:02 INFO mapred.JobClient:     Reduce input groups=121791
11/06/01 06:15:02 INFO mapred.JobClient:     Combine output records=183601
11/06/01 06:15:02 INFO mapred.JobClient:     Map input records=127602
11/06/01 06:15:02 INFO mapred.JobClient:     Reduce shuffle bytes=958780
11/06/01 06:15:02 INFO mapred.JobClient:     Reduce output records=121791
11/06/01 06:15:02 INFO mapred.JobClient:     Spilled Records=473035
11/06/01 06:15:02 INFO mapred.JobClient:     Map output bytes=10812590
11/06/01 06:15:02 INFO mapred.JobClient:     Combine input records=1111905
11/06/01 06:15:02 INFO mapred.JobClient:     Map output records=1111905
11/06/01 06:15:02 INFO mapred.JobClient:     SPLIT_RAW_BYTES=931
11/06/01 06:15:02 INFO mapred.JobClient:     Reduce input records=183601
/usr/local/hadoop-0.20.2$

Retrieving Output

Now that the MapReduce job has completed successfully, all that remains is to extract the results from HDFS and take a look.

$ mkdir /tmp/books-output
$ hadoop dfs -getmerge books-output /tmp/books-output
$ head /tmp/books-output/books-output

You should see something very close to:

"'Ah!'  2
"'Ample.'       1
"'At    1
"'But,  1
"'But,' 1
"'Come! 1
"'December      1
"'For   1
"'Hampshire.    1
"'Have  1

Success!

Killing the Cluster

When we're finished, we can kill our cluster with this command, back at the REPL:

user=> (destroy-cluster example-cluster ec2-service)

Next Installment

That's it for now! Next, we'll talk about how to test hadoop clusters using pallet-hadoop with vmfest to create a virtual machine cluster identical to your production cluster on the cloud.