Excited to announce that today is my first day as VP of Eng at @agrilyst. Looking forward to building with this awesome product and team!

— Aaron Quint (@aq) March 14, 2016

I actually love consulting and the work was going well. Finding clients was hard but I was lucky to have some stable gigs that I enjoyed. I liked being able to spread my expertise around multiple teams and get to learn and absorb a lot at once. I didn't like the ratio of work to meta-work. The amount of time I spent getting, keeping, and prospecting clients far outweighed the amount of time I got to spend working with them. I'm aware this is often the nature of the work, but it was just starting to get to me.

At the same time, I started to feel this yearning I hadn't felt in a couple years - the strong desire to build. I wanted to be a part of a small team again: have a big impact on a product, celebrate all the shipping, work with peers and mentor younger devs. I reached out to a bunch of friends and Charlie O'Donnel introduced me to the co-founders of Agrilyst.

I was immediately interested in Agrilyst because it checked a large number of boxes for the team I wanted to join:

• An industry that I am interested in, though knew little about: Indoor Agriculture and Hydroponic Farming.
• A product that could have an impact on the physical world and had a reach beyond the internet: Growers use the platform to manage and optimize their operations, leading to more cost and energy efficient crops.
• A plan to generate revenue: The product is a SaaS that growers pay directly for.
• A good team: Allison Kopf (CEO) and Jason Camp (CTO) managed to build and grow the beta version by themselves and even win TechCrunch Disrupt last year.

After talking to the team, it wasn't a hard decision. It's been a big transition for me but I'm loving it so far. I've had to go heads down and write code for the past couple weeks (something I haven't done for an extended period in a long time) and it's been fun. We've had to do some technical planning and we're proud to be choosing a boring stack. It's allowed us to get to a point where we're almost ready to release the real v1 of the product in the next few weeks. Any time that hasn't been spent writing code has been learning about the industry and visiting farms. I can tell you that indoor farming is an industry with amazing engineering feats and real innovation. It's incredible to see first hand.

When joining I was aware that my role wouldn't stay the same for very long (it never does, but especially not is such a small team). The heads-down coding is hopefully going to shift into leading a team in the very near future. We're looking for a couple of strong people to join the team and help us continue to build and improve this awesome product. I'm looking to build a unified product team that encompasses design and development so that we can work closely together to ship beautiful and functional features. If you'd be interested in working with me or hearing more, contact me.

It's still early, but it's been a while since I've been this excited about work.

I've been back working with some clients on optimizing the performance of their web applications and I've been revisiting this graphical way to explain how I think about this work. I first talked about this at RubyConf 2014:

Starting to treat your application as a square

I like to think about applications as they sit in "server-land" as squares first, with user requests traveling through the layers top to bottom. This isn't unusual - we usually talk about these different parts of our stack as "layers". On top of this, I add another aspect - each request by a client (user) to the backend application traces a vertical line and that vertical line maps to a single "path" through this application. In Ruby on Rails (or pretty much any http server), this path maps to an action. That action might map to a long call tree that calls other action-like items but for the most part that path is repeatable, even if it crosses many services and looks more like a desire path than a paved one. The purpose is to collect and trace these common paths as grouped actions¹.

When I think about requests and layers this way, we can talk about the different type of work we do to optimize performance as vertical or horizontal. Vertical performance work optimizes paths by looking at key requests and trying to fix their problems. Horizontal performance work addresses full layer or services by improving, replacing, or upgrading the functional processes.

Vertical Performance: Optimizing Requests

With the proper monitoring (through logging or a third-party service), we should be able to get a good overview of the different paths that users are traveling through our application. In an ideal set up, we should get a clear sense of the most problematic actions. These aren't always the ones with the highest response time, rather theres usually a simple function to be applied which looks something like:

95th Percentile Response Time 
x # of requests 
--------------------------------
 Made up number to sort by

If we sort that number by highest first, we get a decent sense of our "hitlist" - the requests that take up the most time in our backend that we should attack first.

When we peer into one of these vertical slices what we actually want to see first is a collection of user requests to this path and the timings across the different layers as percentiles. This gives us a good sense of what part of our stack these requests are spending time in (DB? Cache? Just code? External services?).

With these relative scales, the work becomes pounding on this slice until its vertically smaller and takes less time. I'll leave the details of this for another post, but the general idea is to create a harness to test the slowest version of this request in our development environment. From there we can use all the tools at our disposal to try to track down the individual problem aspects of this path and make them faster. Often this is removing parts of requests by deferring them into the background, putting them behind a cache, or just straight deleting code.

A job well done here will show an improvement across the site-wide response time. Thats not always the case, but an improvement of even just a few milliseconds to a path that is the #1 request by an order of magnitude can make an outsized difference.

One of my clients had a post-page load POST request that recorded metrics about the page for user-facing analytics. This request happened on almost every page and was easily the #1 request (in terms of numbers of requests). For simplicity sake lets say it was called 10K times an hour. Its 95th percentile response time was relatively fast compared to the rest of the application, 100ms. The "user dashboard" page for comparison was requested about 1K per hour and had a 95th response time of 500ms. Doing the simple math, the hitlist time for the dashboard is 500,000 (500 * 1000), while the metrics request was 1,000,000 (100 * 10000). Reducing the response time of the metrics request even by 10% creates a significant reduction in the overall time our server spends handling requests. In this case we realized there were some easy but impactful things to do. We actually didn't need to record metrics for every request (reducing the hourly count). We also pushed some of the calculations the action was doing into the background, cutting the response time in half.

Horizontal Performance: Optimizing Infrastructure

Horizontal work focuses on entire layers instead of individual requests. The work here is often infrastructure tasks instead of code. Different layers have different optimizations, but they mostly fall into similar categories. Upgrades to the processing power (hardware) can sometimes be the thing that solves the biggest performance problems. This often comes down to questions of money - bigger boxes require more dollars. If you can have some open conversations with your team (or yourself if it's a team of one) about the tradeoffs of spending time on piecemeal performance work vs buying new hardware, then occasionally you can just throw money at the problem and be done with it (for now at least).

This isn't always the case. Sometimes its not about the size of the box but its configuration, or even the way the layer is being used. An easy example would be a relational database with a poorly designed schema or missing indexes. The work here isn't fixing an individual request, but improving the performance of this layer horizontally across the square.

I've come across this a number of times: the database or caching layer was poorly configured and was queuing trying to handle too many simultaneous connections. Implementing connection pooling across the board improves the overall throughput and you see a noticeable drop in your overall metrics.

You don't have to only choose one

The goal here is not to pick a single angle and only focus on that, its to identify two distinct ways to think about web application performance. With the awareness of these different angles we can pick and choose which are the most pressing to work on and create rubrics for identifying the highest value work.

I'm interested to hear how other people think about and describe these different types of work. How do you approach the performance work in your app? Do you take specific repeated steps? Do you divide these different areas between different teams?

I’ve been thinking a lot about the concept of self-awareness at home and as part of team. A month or so ago, I spoke at the last JSCONF about the wild n’ crazy journey the past five years has been. In the talk, I sketched out a lot of the great things I learned from reading about management and humans, specifically the highlighting book Emotional Intelligence 2.0. It seems I’m on a mission to get a lot of people to read it. It might appear corny and have a lot of drawn out content, but the basic ideas have stuck with me.

Emotional Intelligence (EQ) breaks down into four quadrants, the starting point being self-awareness. Self-awareness (or the abesence of) is particularly interesting to me as I’ve always been someone who “wears their emotions on their sleeves”. This can be a really positive thing when I’m excited and passionate about something, but also can drag the people around me down into a burning inferno when I’m not. Being aware of the lack of self-awareness in myself and in the people around me put me on the path to improving.

The book shows a bunch of case studies of all of the different aspects of EQ and you start to get a sense of the things that you or someone you know does that fit into these categories. Instead of trying to paraphrase, here are some REAL WORLD scenarios from the past couple of years.

Steam coming out my ears

“It’s a pretty straightforward plan”.

I was trying to explain to my boss that we had to delay a bunch of new feature work and spend some time fixing some underlying problems instead. They seemed to be listening.

“We have to just buckle down and fix X and Y no matter what or we’re not going to get anything done”

“OK, makes sense”, they said.

Forty-five minutes later, we were sitting in the conference room with the team that was about to get their plans changed. I introduced the general concepts, “We’re not going to do exactly what we thought, but we think this is a good plan” and then passed the baton to my boss. Instead of outlining the plan we just agreed on, they started outlining a completely opposite plan; one that completely negated the good faith I had carved out with the team and set them on a completely different course.

The whole team looked at me, stunned. Only much later did I realize that I had steam coming out of my ears. I was controlled enough to not break into fisticuffs in the moment, but not self-aware enough to realize that everyone in the room knew exactly what I was thinking.

A couple years later, confronted with a similar situation, but a new ability to tap into “what I’m actually feeling” I’m able to open my mouth and express my feelings in a constructive way. The end result isn’t perfect, but its 100000x better than the raw and clouded emotion that was making me want to flip tables.

The point isn’t to not be angry – it’s about understanding the frustration and not letting it hurt the people around you. This is something I’m still working on every day (especially as the father of of a two year old) and I have a million other examples of ways I’ve hurt people, relationships, or work and remained unaware until it was too late.

We’re doing great

A couple of months ago I was approached by a developer who was part of a growing medium sized team. He had seen a talk I gave about teams and growth (and the failures there of) and was seeing some of the same problems I had faced. Specifically, there was a lot of churn and a lack of clarity around goals and how he, as a member of the team, could effectively contribute. He was excited about having me come in and give some advice or mentorship to the team, so we ended up sitting down with himself and the CTO. Immediately, I sensed something was off, he started drilling me as if I was applying for a developer job. When he asked “Well, what are you looking for?” I explained what my current work is, doing high level consulting for teams. I gently let them know that I heard that there was potential for me to bring my experience to bear in helping them grow, structure, and plan. The response was longer but can be summed up as “We’re doing great.” Instead of listing the problems that they had been having (as I had heard them from a team mate) I heard a rattling off of accomplishments of scale, etc.

Personally, I wasn’t angry, though I knew I could help if given the chance. The clear lack of self-awareness as a leader translated into a lack of organizational awareness that could open them up for critical feedback and improvement.

How it’s effecting me today

These two cases seem unrelated but really stem from the same inability to reflect on our own emotions and behavior. The latter scenario has especially been on my mind lately, as I work to get my consulting business off the ground. I’m lucky to have found and worked with some great clients so far, but I know I’m not doing as much as I could or reaching the companies that really need help. I’m sure part of it is marketing and “getting the word out”, but I’ve come to realize that my market is smaller than I previously thought.

If you think of it like a Venn diagram, you can see the problem. With the original and naive opinion that “There are a lot of companies that could use my help!” the market is huge. The problem is the second case above was not a single occurrence. The shortage of awareness in leadership is a giant blind spot that prevents me from getting in the door. I’d be a fool if I thought the only reason I don’t get hired is because the other people aren’t self-aware enough; Of course there are other reasons. However, the difference between the leaders of the teams I have gotten the chance to work with and those that are “doing great” is stark in terms of the awareness of their problems and the ability to get better by seeking help.

I haven’t found a solution for this yet. Maybe if I just keep writing and pushing these books, some leaders will awake into self-awareness (picturing this like Neo waking up outside of the Matrix) and the middle of that diagram will grow.

We're all the same Special Snowflakes

The first thing you experience being thrown into a number of different functioning teams, is that they all believe their problems are unique and they all have very different ways of solving them. Taking an outsider's perspective, you start to realize that while, yes, the large number of individuals and choices that led each organization to where they are today are unique, every modern engineering team suffers from a lot of the same symptoms. We can call these patterns or anti-patterns or commonalities, but the fact remains that when you have the opportunity to put these situations side by side, you see the same threads.

This shouldn't really be a surprise to anyone - the confounding part is that many people want to believe that they have it the worst (or the best) and refuse to look outside for help or even just compare notes with other people in similar positions.

I'm barely talking about technical problems here. We all blog and talk about the challenges of this language or framework and how we moved from x.js to y.go and from mono-services to micro-liths. We hear a lot less about all the problems of our personal infrastructure and how we plan (or don't) to refactor them. This is certainly in part because we have this strong belief that these problems are ours and ours alone and how could anyone benefit from conversation around it? I'll keep saying this until the day I quit tech for good, but your biggest problems are always going to be people problems and the only way you'll improve is by talking and listening to other people (historical or present).

As a consultant I'm able to gather expertise because I'm able to see, communicate, and act on these similarities between teams. If every team was a special snowflake and singular, then the work I do would be almost impossible.

Everyone is swimming in Technical Debt

A primary reason I've been called in is to help a team navigate and come up with a plan for addressing their technical debt. There's a general assumption amongst teams that technical debt is a product of "moving fast". While I don't disagree that this often exacerbates the situation, the buildup of "legacy", broken, and problematic code is just a fact of working on a long running project. If you have a reasonably successful product with a team thats grown and changed over time, there will be areas or whole repositories in your code base that people are afraid to touch, that emit a horrible odor, that "we don't talk about". I'm not saying this to complain - I'm saying this because every team has to deal with this. When you're tip-toeing around a codebase and cursing the state of things, it might be helpful to remember that despite the claims of other teams on the internet of their glistening and healthy code bases, they too have a bunch of dirty corners where they sweep things that they just don't talk about. Either you're swimming in technical debt or you're just too early or too unsuccessful to have accumulated it.

How teams manage this and continue to grow and even thrive is what I'm interested in. The challenge is not only around actually planning and implementing individual fixes, but addressing these in a functional and long term way. I've said it before but big rewrites are rarely the solution, so working in the scope of incremental changes, though harder to plan up front, is usually the only achievable solution. I've seen this work. It's not an overnight change, but with dedication and motivation, I've seen codebases shift.

Technical Debt is easier to solve than distrustful workplaces

The thing about technical debt, though, is often its rate of growth or stagnation is tied or exacerbated by other dysfunctions. It's one thing to have to put refactoring aside to just maintain the status quo as a user-base grows. It's an entirely other thing when wounds are taken for throw-away features, endless scope-creeps or clever developer tangents. These are all side-products of teams that are not on the same page about goals or teams where the balance of power and ownership is off-kilter.

In a vacuum, tackling mountains of bad code is challenging, but not impossible. You can see a solution and work your way to it. In the real harsh environment of Planet Startup, if you can't effectively communicate the need or the severity of the work that needs to be done, it will never happen. The test then is not the actual work of writing code, but the work of navigating the politics and demands of a growing team. Reminder: this isn't impossible, but it might not be the challenge that you set out to solve.

Sometimes all you need is a spark

In technical operations we often talk about "the probe effect", in which the tools you use to measure (performance, load, etc) often have a visible effect on the system they're measuring. I've noticed a similar effect when I'm working with a team - the act of taking the time to talk about and start to address some of the known issues makes a difference in-and-of-itself. I'd like to believe that my presence, guidance, and positive energy has something to do with it, too, but I can honestly say that a large part of making progress is just about making the space for progress. Working with an outside perspective that's only there for a limited time forces this and ideally lights a little fire under everyone to really start moving on some of the goals they want to achieve.

In all of these organizations, I've observed a lot of very talented people who are wholly capable of getting massive things done on their own being restricted by lack of time and clear direction. This is not a problem with an individual - it's an organizational problem, one that is common and takes a lot of work to actually solve. There are ways to get big wins with minimal effort, though: giving these ambitious individuals clear and achievable goals and the ownership and help to reach them goes a long way.

As a consultant I'm given a bunch of special powers of internal navigation and leeway that I wouldn't get as a full time employee. That has advantages for me, but it also speaks to how much more is possible when individuals are given trust and leverage to make change. I'm not suggesting you treat every employee as a consultant, but sometimes trying to look past the institutional weight of change can allow for the spark that people need to do greater things.

If you're interested in working with me or just want to talk more about this, you can email me at aaron@quirkey.com or find more information about the work I do.

Cross-posted at Medium

• Astronaut
• Hip Hop Producer
• Restaurant Owner
• Disney Imagineer
• A Dad

I keep coming back to this feeling that I don't know what I want to be. I'm probably deluding myself that I haven't grown up already. At least I'm a dad.

For the past 6 months, since I left my full time job, I haven't been doing nothing. In fact, I've been doing more different things than I have in a long time. I've been doing high level consulting with a number of different and awesome startups. I helped run a conference. I'm authoring a book about pizza that's successfully been funded on Kickstarter.

But what am I supposed to be?

For 6 years I had blinders on and I worked with a mission. Whether or not the mission was a real or meaningful one was irrelevant. I was aimed at a target and I was moving fast.

Now I'm missing that feeling - I've been recreating it with these individual and short term projects. Each feels good, but does it add up to a whole? Is it OK to be many things?

Area man seeks target.

I do know that I love working with people and building awesome things, so more of that please. Maybe what I want to be is just that - a Collaborator. And that's OK.

At the end of October I spent the weekend at the Ashokan Center, about 30 minutes from my house in Kingston, running a conference for the first time. To say it was a fun time would be the understatement of the year. It’s taken more than a few weeks to process it all.

What we set out to do

We knew immediately that Catskills Conf [Note: Kale wanted to call it CatskillZ] couldn’t be like other tech conferences – we would not only have talks and showcase awesome people and their stories, but we would have to try to integrate the unique combination of themes that we feel the Hudson Valley is about. For us this meant: Art, Music, Food, Nature, and Community. Kale immediately zeroed in on the Ashokan Center as the venue and plans quickly started to form around it. We roped in the other organizers of the Hudson Valley Tech Meetup – Dan Stone and Dan and Sabrina Schutzsmith – and we were off.

I leaned heavily on some friends who had run conferences that I was inspired by, specifically Paul Campell and Mikeal Rogers. They had a ton of infinitely useful advice and a lot of warnings about how crazy running a conference is (especially the first year). Newsflash: They were right!

We aimed to create a program that was packed to the brim with inspiration and fun. I was convinced that if we collected as many brilliant and diverse people that we could into a wooded area and fed and entertained them, it would be an amazing time.

What we accomplished

We had 15 talks that awed and inspired, with topics ranging from startups, to teaching and community, to the areas where art and technology meet. We had everyone engaged in the Saturday activities, including Foraging, a Wearables workshop, Cider-Making, Blacksmithing, Ukulele, Book Binding, and Flying Drones! We had endless flowing drinks from ImmuneSchein, Keegan Ales, and Awestruck Ciders. We had a live hog butchery demo from Fleisher’s. We had live music from Mike & Ruthy, Monogold, and Adrien Reju. We had Jonathan Mann write us a song for each day. We had bonfires. We had square-dancing. We had Canadians camping in the rain. We had a mother-f’ing Birds of Prey show.

Most of all, I think we all had fun. I went into the weekend, worried that I was going to be so overwhelmed with the organizing and running around that I wouldn’t be able to enjoy myself. While I did do a lot of running around (My health tracker counted 12 miles for the weekend) I was still able to catch up with a ton of the attendees and speakers, spend some time listening to the great music, and hanging by the fire.

What didn’t work so well

Registration was a total mess. A huge group of attendees arrived at once on a bus from NYC and we were not fully prepared to handle them and get them all to their rooms. We had some great volunteers, but we did not have a lot of time to train and set them on tasks.

The rooms were mostly shared bunk style, and one thing we didn’t really think about was what that meant for different sleeping styles. The unfortunate biggest piece of negative feedback from our post-conference survey was that people snored. Loudly! This made a lot of people sleepless throughout the weekend.

Another big point of negative feedback was that we packed the schedule too tight. Having only 48 hours and our boundless excitement, we invited everyone that we admired and that would take us seriously to participate, and that turned into a pretty non-stop weekend of activities.

Finally, we really set out to make this as inclusive an event as we possibly could. We set up Diversity and Student scholarships and looked to raise money to bring people to the event. We wanted to give as many people as possible a chance to attend. We had 27 people apply for the scholarships and every one would have been an awesome addition to the weekend. Unfortunately, despite a lot of talks with a lot of companies and individuals, no one stepped up to sponsor. This was a bummer, but we felt that we had made some promises, so instead of just closing up shop, we took money out of our budget for 11 scholarships. The worst part about this was the last minute nature of telling some of the applicants that we didn’t have the funds. In terms of speakers, I was happy that we were able to feature a gender-diverse group of talented folk, but I know we’ll be able to do better next year.

Next Year

And that’s the point, really – there will be a next year. The week after the event we got together as organizers and started the discussion of how we can make next year even better. We don’t have perfect solutions for all the problems, but we do think that we can approach each with gusto. Working on Catskills Conf was one of the best things I’ve done in a long time. I’m proud that we were able to create a unique experience that felt familial and unpretentious from the get go. Over 3 days, we all became just a bunch of friends, having fun in the woods.

Catskills Conf 2016 will be held at the Ashokan Center Oct 21-23, 2016. If you’re interested in being a part of next years event, sign up for our mailing list or email us at hello at catskillsconf dot com.

If you want to read some more perspectives on Catskills Conf: Amanda Kivet, Martin Babinec, Mike Zamansky, Watershed Post. And definitely check out Daniel Fitzgerald’s blog post and photography project “More Space”.

There’s been a lot of talk this week about Work-Life balance, pushing employees too far, and the like in the wake of the NYTimes expose about practices at Amazon. Whether you’re on the defense or the offense of this specific reveal, the bottom line is that overwork is a problem in the collective realm of our industry and one, like diversity, that’s worth talking about.

The fact is that I had starting writing this article about my own experiences with this well before the hubbub this week. I gave a talk at QConNY about my wins and losses when scaling a team earlier this summer.

Leading up to the talk and since then, my thinking has continued to become more clear. As always in the world of “wet-ware”, this is my personal experience and opinions and don’t necessarily apply to everyone. In the same breath, I’ve learned that opening up and being honest has more positive then negative effects; doing this as a community allows for growth through conversation.

Burnout is real

I’m having a hard time writing code.

Even after more than a year, I still look at code, at code that I wrote, with impatience and confusion and doubt.

This is a strange feeling for me. I would categorize myself as an extremely productive and fast programmer. At the peak of my productivity I was working on multiple projects in multiple languages with multiple teams in a single day.

But now, I’m blocked.

I wrote a little bit about the why’s of where I am now (and also talked about it at QCon, but the truth is: burnout is real. I was pushing ahead, digging a whole in the ground, and when I was so exhausted I had trouble getting up every morning, I tried to make a change.

Part of me believed that I would step away and all of a sudden be filled with ideas and energy again but for whatever reason, that hasn’t happened. Burnout for me is actually just a face of depression and the problem with either is that it’s often hard to pull out of the ditch – forcing something and failing at it often leads to more disappointment which just leads to more digging.

Do what you love is BS

I had this vision as a High-Schooler who was in to music and music production that I was going to graduate and open a recording studio. I was rallying towards the peak of my personal confidence (that period where you really do believe you know everything) and I had been telling myself and others that this is what I loved to do, and therefore I should have the power to do what I love and be successful at it.

After 4 years at a liberal arts college, though, I was facing graduation and was a bit disillusioned. I still retained some aspects of that earlier dream, but it didn’t seem as glamorous anymore, and I had started to love other things. I got back into programming after a hiatus and I did really enjoy it. Part of me had this feeling like, well, I might not love this, but I can feed myself doing it, so … That changed over time, I did love programming. With the discovery of open source and new tools I got more and more entrenched and more filled with this pride – “I’m getting to do what I love! People are paying me for this!”

Things change. I started to get involved in different aspects of this world: people, management, operations. There were constantly new things to learn and I was still excited and keeping up as best I could. There were little feels here and there, however: realizing I lived in the Bay Area and barely knew my neighborhood; constant stomach distress from eating crappy take out; only getting 5 days off after the birth of my first born. It wasn’t the fear of missing out – it was the realization that I liked other things and those other things were never in my schedule.

“Do what you love” is built on two giant fallacies: That you can only do and love one thing and that one thing has to be all that you do.

Hindsight being what it is, I’ve always loved and been inspired by many different things, and earlier in my life I was able to really embrace that and make it my mission to do all the things that I loved. At some point I lost sight of that, and now I’ve been struggling for the past year to try to throw myself into as many things as possible and try to find that balance.

Metrics for happiness

In doing a lot of thinking about this, I’ve tried to use a trick that I learned from Emotional Intelligence which is to force yourself to ask “why?” and “what’s beneath the emotion that I’m experiencing?” until you get to some dark-ish place that is more accurate or representative of a root cause.

After doing this for way longer then is probably healthy I came to the conclusion that I wanted to be successful and happy and I felt like I was unable or in the wrong situation to do that. When I broke this down more, it became a familiar narrative:

1) Work Really Hard
2) Make A Lot of Money
3) Have the time and space to do the things I want
4) Be Happy

The strange thing that it took me seemingly forever to realize is that there was this underlying assumption that to get to the real goals of #3 and #4 doing #1 and #2 were the only way to get there. Ingrained in me was the belief that success was defined by the path and not the result.

Changing my approach is a process, but one that I’m actively involved in. I’ve been thinking about how to use my skills and time to be happy instead of how to make money. This shift is hard because the pedagogy of success would assume that by stepping down in my role, making less money, etc, would mean that I’ve failed. I’m still wrestling with this because the belief is so woven in to our industry and society.

The hardest part about trying to be happy is the feeling that happy means happy all the time. While thats a pretty lofty goal, its a damaging one, as it makes us sink lower than necessary in moments of feeling sad or lost. Learning from my experience in Ops and Infrastructure, my new metric is aiming for 90th percentile Happiness. If every week, 90% of the time I’m happy and feeling content, I’m going in the right direction. This is a moving average function and it can smooth out those rough days when nothing seems to be going right.

Moderation and Balance

For myself, it’s key that there’s no ‘going cold turkey’. I’ve had fantasies of running away and just opening a sandwich shop. That’s not to say that I won’t do that someday, but personally, I’m at my most present and my most successful when theres a lot of different things going on that can form some sort of balance between work I have to do, work I like to do, and not work at all. This might be 40 hours of work, but it’s becoming clear that that’s really unlikely.

Moderation in most things is a step to finding the balance. Food journalist, Michael Pollan, famously summed up his approach to healthy and sustainable eating with the quote “Eat Food, Not Too Much, Mostly Plants”. I’m trying to adopt a similar approach to work:

“Do Work, Not Too Much, Mostly Fun”.

Whatever is next

I’m currently working on a bunch of different projects but most are not directly related to writing code. I’m working with some teams around the country to help direct their large application development efforts to be efficient and performant. I’m working on writing a book about Pizza with my best friend..

I’ve definitely had the feeling recently though, that I’m wanting to dive into something bigger. In searching for the next thing, I’m explicitly putting balance at the fore-front. Whether I end up joining another company or starting my own, 4 day work-weeks are the new “very busy”.

Cross-Posted at Medium

I’ve been aggressively pushing people to use StackProf for over a year now, specifically through talks and a bit of short writing here and there. This past weekend I got to attend GoRuCo 2015 and saw my friend, neighbor, and general badass Eileen Uchitelle deliver a talk about using StackProf and other tools to diagnose performance problems in Rails core. She did a great job describing the tools and how to use them, but it made me realize that maybe the reason StackProf hasn’t really caught on in the Ruby community is that theres not a lot (read: any) good documentation about how to read or understand the information you get from it. So here I am, aiming to fix that.

Where does StackProf come from?

StackProf was created by Aman Gupta (@tmm1) at GitHub with the goal of creating a low-impact sampling profiler for Ruby. This was inspired by Google’s gperftools which was made popular again recently by Go’s implementation.

This is all possible because Ruby 2.1 shipped with an internal API (rb_profile_frames) that allows StackProf to grab stack samples with minimal CPU and memory overhead.

What is it good for?

StackProf is a sampling profiler with very low overhead which makes it ideal for diagnosing performance problems in production environments. That doesn’t mean its only useful for production, but because its only capturing samples every n milliseconds, its more useful on long running processes then quick bits of code. It doesn’t capture every frame or every call in your code (that’s the core tradeoff: lower resolution for lower impact) so there are some cases where your code is made up of many different quick calls which might escape its clutches.

What is a stack?

Stepping back a bit, when I first presented StackProf to some colleagues they didn’t really grasp how to understand the output because they didn’t understand the concept of a stack in this context. When we talk about “the stack” in ruby we’re talking about the method call stack. Most developers experience this most frequently as error backtraces when we hit an exception. A backtrace is a textual representation of the call stack (in reverse) from where the error took place. When we call a method that calls another method that calls another method, each of those calls in put onto a “stack” and then when the lower most method finishes, it is “popped off” the stack.

StackProf takes samples of the running code as individual “frames” which represent the stack at that given moment. Each frame gives us an ordered list of all the methods and lines of code that are on the stack, from the top most method (in a Rails app, probably your server/Unicorn) to the code on the bottom (the code currently running, like an ActiveRecord call).

Here we can see a simple representation of the stack as time proceeds (left to right) and we descend and ascend the stack (up and down). When we take samples over time and then collect them into a single report, we get a simple way of representing what methods were on the stack the most and therefore taking up the most time (or CPU time) in the period that we were taking samples. 

How do I read the general output?

I created a simple gem called stackprof-remote that provides an easy way to collect dump’s from production systems. Maybe deceivingly, it also includes a simple interactive wrapper (based on pry) for navigating stackprof .dump files called stackprof-cli. Once you have a dump you can get into it via:

$ stackprof-cli last-run.dump
stackprof>

Get the top 5 methods:

stackprof> top 5
==================================
  Mode: cpu(1000)
  Samples: 5045 (3.28% miss rate)
  GC: 355 (7.04%)
==================================
     TOTAL    (pct)     SAMPLES    (pct)     FRAME
       736  (14.6%)         707  (14.0%)     ActiveSupport::LogSubscriber#start
       379   (7.5%)         379   (7.5%)     block in ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#execute
      5248 (104.0%)         168   (3.3%)     Benchmark#realtime
       282   (5.6%)         117   (2.3%)     ActiveSupport::LogSubscriber#finish
        88   (1.7%)          88   (1.7%)     block (2 levels) in Sass::Importers::Filesystem#find_real_file

At the top of the output is some general info about the dump. We were running StackProf in CPU mode sampling every 1000 frames (the default). We took 5045 samples and 355 of those were during a GC pause and we can extrapolate that 7% of the time was spent doing GC.

Below the general info is a list of methods sorted by the samples column. The first two columns show the number and % of samples that this method was on the stack. The second two columns show the number and % of samples that this method was at the _top_ of the stack. By sorting by the samples column, we're showing the methods that our process was spending the most time in across all the samples. If we sorted by total (possible via total N in the CLI) we’re showing what methods were generally on the stack the most. This is useful too, but often just shows methods that are at the root of the stack (like server and middleware code that wrap everything else).

You can also dive into a specific method and see what the callers and callees (up/down the stack) were in the samples that were captured using method METHOD.

What’s a flame graph?

Flame graphs are a visualization of the same stack sample data popularized by Brendan Gregg. Flame graphs are awesome because they let us have an overview of an entire dump/captured session in a visual way that make it easier to pinpoint hot-points. Flame graphs display time left to right and the stack vertically. The colors in the StackProf Flame graphs are used to group different sources of code (i.e. different gems, controllers, models, etc) and are sorted with darkest red being the code that had the most TOTAL samples. In the example above, which is a profile of a Rails controller action you can see that ActionPack is at the top which makes sense because we’re spending most of our time dispatching/running a controller action. We can see that the stack moves up and down and there are clear sections where different parts of the action are happening. The lowest hanging fruit for optimization here are things that are wide and close to the top. Those are methods that are themselves taking up a lot of time.

Where to next?

If you set up stackprof-remote on your production app, you can make it easy to collect dumps from you’re running prod servers. You can do this at high traffic times, or just during a normal day, and get a good sense of where your app is spending time across actions/requests. This can be immensely useful as it can identify areas of code that are not in a single action, but common between actions that are taking up an unexpected amount of time or CPU. It can be useful locally, too, as Eileen showed in her optimization of the Rails test suite.

If you're interested in working with me on performance and scaling as your team grows, I'm available for hire.

The slide was inspired by Brendan Gregg‘s essential work on Linux Performance. I’ve been worshiping obsessing over following Brendan’s work for a while now, and his System’s Performance Book is frequently quoted at work. The central theme of my talk was trying to apply the U.S.E. method to Ruby/Rails application performance. This idea ended up bearing more fruit then I could present in a short (25 min) talk, so the diagram above provides a overarching view of the work.

There are a few key takeaways here:

There are many tools, and that’s OK

It’s important to try to think about tools in the framework of the Unix philosophy of small programs that do their job well and are modular and compose-able. As application programmers, we’re often looking for the one true ring of performance monitoring and debugging, or worse, trying to use a tool we know, meant for a specific use-case, as general-purpose. So it’s great that there are a proliferation of tools, we just need to know what they’re good for and how to wield them.

It’s also worth noting that there are WAY more tools then I outlined in the talk/diagram such as schneems/derailed_benchmarks.

Development is a different world from production

This seems obvious from an infrastructure point of view, but is also true from a tools perspective. There are a ton of tools that provide little value against production because of the probe effect or non-useful data. Know which tools you can use in which environment.

Saturation is often overlooked

I was glad that @indirect brought this up in his talk, too (hopefully slides/video will be available soon) – we often spend a lot of time looking at Utilization (averages/response time) and Errors in our applications, but rarely look at Saturation (how much extra work our application has to do, but doesn’t have capacity to handle). There are some easy ways to measure this for certain stacks (nginx, unicorn) but if you’re stack doesn’t provide this easily, that should be a concern.

Ruby has some gaps, we should be aiming to fill them

Specifically, there really aren’t any good tools for debugging Ruby Memory usage in production applications. (I’ve heard rbkit has made some progress recently, though haven’t heard reports from the real world). There are finally some decent APIs in Ruby itself, I’m hoping all it takes is some smart people to steal some get inspired by the awesome tools in other mature languages.

If you or your team wants to learn more about these tools, I’m now available for training and consulting.

What I’ve begun to realize is that at the core of my frustrations with the obsessions with the latest framework and the constant need to burn down the past is that people tend to overlook the underlying choices they’re making when choosing technology.

All of my opinions on this can probably be summarized as: Awareness of the tradeoff’s you’re making is the most important part of your work.

Tradeoff-awareness goes both ways: It’s as crucial when you’re advocating as when you’re nay-saying. It means knowing that when you pick an untested or young technology, it might have problems at a later date. The key is accepting the risk in exchange for the “new-tech smell” and being aware that you made that choice[2]. Having faults is not a critical bug - there is not a single thing in the universe that doesn’t have some sort of issue[3]. Having perspective and knowledge of these limits allows you to address them and move on.

I love Ruby and I’m also fully aware that choosing Ruby for a project trades my familiarity and the ease of use with it’s performance and resource usage at a certain scale. I also love Go and I’m aware that I’m trading performance and concurrency for some of the speed, familiarity, and [over-]abundance of libraries and tools. Having an in-mind index of as many tools as possible and their strengths and weaknesses helps me pick the right tool for the current job.

It’s an idea that mainly spurs from my reading in cookbooks, but I’m finding it’s applicable to all aspects of my life and not just cooking. In the food world, this aspirational idea comes in many forms and phrases. In Daniel Humm’s kitchen at Eleven Madison Park there’s a large sign that reads “Make it Nice”. When you read their book you realize that this represents one of the core philosophies: the striving towards something great through making each individual detail “nice”.

I mentioned the inspiring and often mind boggling Fäviken Book in The “Perfection” episode of Beats, Rye & Types. Chef Magnus Nilsson sums this up pretty eloquently:

“The questions you must ask yourself about every detail are these: does this make the end result better, and do I have the time or capacity to do it at the moment? […] The important idea is not always to do things without compromise, but with thought-through attention and decisions that lead, little by little, towards creating a better end result.”

I wouldn’t in a million years compare myself to these extremely talented, driven, and respected chefs. Rather, I’m starting from what I consider zero, and slowly improving.

For the past couple of months I’ve been cooking a lot and I’m always driven to take on new recipes that are increasingly more challenging. Even in the most basic recipes, though, I started to see my lack of consistency - I started to slip and make stupid mistakes. It took a while to realize (and through reading more and more) that this wasn’t from lack of skill or bad recipes, this was just me not paying attention. It was me making small split doeskins to take a shortcut here or there (not rinsing this, not using the right pot) and that having the exact opposite effect of what the chefs described above - it made the overall result worse.

Most of these things, I realized were completely subconscious or at least not something I was recognizing in the moment. It has some of the same results as being lazy, but this is slightly different - this is about self-awareness and being present enough to do the “nice” thing.

Now I’ve gone from being generally aware of the situation, to being aware in the moment. It’s taken a huge amount of effort to put my own momentum behind living in a “No Shortcuts” world. I’ll be cooking or writing and I’ll have to catch myself, “you’re rushing, you’re skipping steps”. I’ve found myself working on a project and about to consider it done when I ask myself , “Is this good? Did I do all I could to make it better?”.

Nothing I do is anywhere near perfect, but every time I try a little harder and try to really think through the steps of something, I find myself becoming more present. Maybe more importantly, I’m getting better at the things I care about - I can make a really good pizza[2].

In and of itself, it can get you some really good information about what’s going on inside of a live ruby process. Its real power, though, comes apparent when combined with the power of other tools. This is because rbtrace’s eval functionality allows you to run anything inside of your running ruby process. Now this should all come with a serious Uncle Ben Warning (i.e. “With Great Power Comes Great Responsibility”) - you need to make sure you have proper security around your ruby processes. Enabling rbtrace is giving access to run anything that the ruby process can run.

Assuming you understand the risks, rbtrace can enable you to get amazing insight, like being able to toggle and dump stackprof information remotely with stackprof-remote. Here we start thinking of rbtrace not as a performance tool itself, but a messaging protocol and system for setting and retrieving live objects in our ruby processes. This works because (as of 0.4.3) the main functionality of rbtrace was extracted into a useable ruby API which is accessible outside of the CLI. For example:

You can see the full magic here.

In this way you can trigger events in your process or just read or set values, but dynamically and without having to provide an explicit API. Again, this power could easily be abused not only by hackers but by developers who want to build the worlds most fragile RPC system (don’t do it!).

What if we used this power for good?

About a year ago now, Kat and I first saw this house. We walked in to the entryway and everything smelled old. Like dust and mold and like air that hadn’t moved in a very long time. It was a freezing day, but it was sunny and the house felt surprisingly warm. “Stone houses stay warm because of the thick walls”, the realtor told us. It was like going back in time, the furniture (early american), the wallpaper (horse and carriage). I turned to Kat and could see it in her eyes immediately, though, this was going to be our challenge - this was going to be our home.

The House

To understand the history of the house, you have to understand at least a little bit about the history of Kingston, NY (all of which I’ve learned in the past year). Kingston was originally a pre-independence Dutch settlement called Wiltwijck, and in 1658 Peter Stuyvesant (then the Director General of New Netherland) forced the loose collection of settlers in the Esopus valley to gather into a walled town surrounded by a wooden fence (or Stockade). Our house (or at least part of it) was constructed not long after, inside what is now know as the Stockade district, by the Van Keuren family. The wall is gone, but the same streets in this map from 1695 are still present.

The original house was probably a mostly wooden structure which was then replaced by a stone structure around 1680. In October 1777, after the Battle of Saratoga, a troop of British soldiers marched into Kingston and burned many of the houses including ours. Most of the stone structure remained intact and in fact, the support beams in the basement that support the first floor are charred black and remain from that period.

The house remained in the same family until 1923 when the Carl family purchased the home and worked with prominent local architect, Myron Teller to “restore” it. “Restore” is in quotes because Teller had a very specific style and agenda and while he kept much of the original character of the home, he added a number of features (more windows, doors, walls, covered porch, etc) that definitely weren’t part of the original Dutch design.

After the Carl's finished the renovations in the mid 1930’s, other then a new stove and maybe a new toilet in the 1960’s the house really was not touched until we bought it.

Our History

When we started looking up here a year ago, we had to get really used to answering the question of why we would want to move here from Brooklyn. I lived in Brooklyn most of my life (with some hiatus’s in Massachusetts and California) and will forever be a Brooklynite no matter where I go, though the lure of something better was just too strong. Kat and I spent a bunch of time coming up to the Hudson Valley and Catskills throughout our relationship. We even got engaged up here (almost 7 years ago if I can believe it).

No, we don't didn’t really have many friends up here. We just loved the natural beauty of the area, the relaxed nature of the towns, and the artistic and bohemian vibe that Kingston especially had.

Along Came Magnus

The real reason we left the city and came up here is a little more complicated. For the past 5 years I was the CTO of a fast growing product company that went from 5 people to more than 100. It was constantly challenging and I really gave it my all. That meant many 80+ hour weeks, traveling every month, and generally working every waking hour I had. I loved working with my team, solving big problems and watching us succeed.

A lot of good friends had warned me that having a child changes you, but I didn’t really understand until it happened to me (I guess you can’t, really). So along came Magnus and even though every ounce of me wanted to just spend time with him I had to make a choice daily and stay late and push through and basically see him for a couple minutes each morning and on weekends.

It wasn’t a single moment, or a snap, but rather a dull throb growing louder. I was burnt out. The feeling is more than being tired, it’s a numbness and a sadness. I was angry at everything and frustrated with myself for wasting any moment of my day. Most of all, the motivation and passion I had, not just for my work, but for anything seemed to have disappeared. This was the hardest thing, because I’ve basically defined my entire self by my ability to be excited and passionate.

So in Jan 2014, overwhelmed by my underwhelmedness, I pushed myself to make a change. By the summer, I had helped Paperless Post find a new CTO and Kat and I had found this house, moved all our stuff and were living out of boxes in a single room while a team of contractors worked away.

The renovation consumed us. Kat did an amazing job basically single handedly redesigning the entire house (more pics). We really didn’t touch any of the historical details, but it’s amazing what a difference a new kitchen, new bathrooms and a couple coats of paint will do.

I continued to work full time for Paperless in my new role as Chief Scientist. Instead of only managing people I was working on writing code again, and managed to help ship a really big piece of infrastructure that helped us get through the 2014 holiday season without a hitch. I wasn’t any less exhausted, though, even though I was much more removed from the day to day operations.

The Spring

After what seemed like months of being buried in snow (this city kid had to learn how to shovel a driveway [many many many times]), the snow is finally all gone and things are changing all over again.

I felt I needed to step back even further and start to explore some new challenges. I’m still Chief Scientist at Paperless Post, but I’m working part time. In the rest of the time I’m dedicated to other things: Spending time with Magnus and Kat (Fridays have become DADA days), getting better as a cook (and a baker), and starting to take on some new projects as a consultant.

I’m looking forward to putting a lot of the skills I’ve honed over the past couple of years to use with new projects and people. I put together a little site to describe the type of work I’m looking for. I’m still happy to be a part of Paperless but also excited to test my expertise on new challenges.

For the longest time, I was sure that I had a very specific track ahead of me, that each job was a step in an upward career. Life is weird and awesome, though, and happiness and success are not only on a single path.

Ruby performance has always been a topic of interest, but as Paperless Post has grown its taken on a new level of importance for myself and the team. We've consistently been living on the edge of Ruby and have been running Ruby 2 since it was just released. As some friends and colleagues finally received commit privileges to Ruby core, I became even more involved, following each change set and testing out the latest features.

It's been extremely exciting to see Ruby come around and not just focus on features for developers but finally add a number of features for operators as well. I really believe this is just the beginning and within the next year Ruby will be much easier to introspect and operate in production. We're finally getting the type of tooling that other communities have had for years.

I'd love to talk to you if you're interested in Ruby performance and working on and using some of these tools.

Heres a full video of the talk as well as slides and links to relevant projects:

GORUCO 2014 - Aaron Quint - The Future of Ruby Performance Tooling from Gotham Ruby Conference on Vimeo.

• Stackprof
• Stackprof-remote
• Objspace_viewer
• Rbtrace

Last week I attended the úll conference in Killkenny, Ireland. Besides being an excuse for us to take Magnus on his first trip to Ireland, the 3 days of the conference let me recharge my inspiration battery a bit and also have a bunch of great conversations with a lot of really smart people. Isn't that what conferences are for? On the second day there was a call to give lightning talks and I volunteered to talk about something thats been very much on my mind lately: Dough. I picked up baking (more specifically Pizza and pasta making) again recently, and the trial by fire has had me thinking a lot. I'm not going to post the slides because they're really just pictures, but what follows is the transcript.

This is a call to action: start making dough.

DOUGH

not CASH

not D'OH

DOUGH

Listen, when the electricity gets and we're all holed up in a post internet globe of apocalyptic struggle, when the zombies come and you have to hold fast, well, your paleo gluten free shit is probably going to have to go out the window with all your obscure knowledge of dynamic programming languages and Simpson's halloween specials and you're going to have to learn how to feed yourself.

You're going to have to learn how to make dough.

But its not just survival, theres an amazing magic there. If you think theres craft and wizardry in making web applications, turning 3 simple ingredients into a loaf of bread. Dough is the original abstract framework. Its one of the original crafts. People have been turning ground wheat and water into bread since 8000BCE. If you think it takes 1K hours to master something, try 10K years.

Look, I didn't come to this naturally. I was pushing myself through learning how to cook everything. Following my obsessions: charcuterie, pickles, cheeses, bbq. Brining, Curing, Grilling, Braising my way through it all. But I resisted dough. Oh, I tried. And failed. All the other things came so easily to me, but bread, pasta, pizza, I kept fucking up. And spectacularly at that, flipping an uncooked pizza on to the floor, getting dough cemented into all the tiles of our kitchen.

There was a reason though. Dough IS a craft. It's something that comes with practice, with patience and study, with attention. Once you give into it though it can be an incredible experience.

What is Dough then?

Flour + Water.

The ratio of flour to water is called hydration. This is one of the primary factors that determines what kind of dough you're making. It also determines how much the dough is going to stick to you and everything else.

Flour + Water + Yeast + Heat = Bread

Yeast, which at its most basic is a living organism that turns sugar into gas, is the magical ingredient that makes flour and water turn from a flat mess into an airy wonderful castle of bread.

And yes, you can buy yeast at a store, but yeast is everywhere around us. Its in the flour itself, you just have to coax it out with my favorite ingredient: Time. Combine an equal quantity of water and flour, 200g each.

An Aside- Yes, I said grams. If you want to make bread, get a scale. Sure you might not have one when you're making bread in the wasteland - but for now, its necessary. If you care about the fact that in iOS6 the navigation dots were a couple pixels off center, but you don't weigh you're ingredients ... You're doing it wrong.

OK. So you mix half water, half flour. You let it sit for four days. It starts to get funky. You've created a monster. A living farm of yeast bacteria. So now you have to feed it. Every day you throw out 90% of it, and replace it with 50/50 water and flour. You do this every day. This is your starter.

Then, you take your farm, and you mix it with more flour and water.

750g Water 700g White Flour 300g Wheat or Bread flour 200g of Starter

You mix it by hand, you touch it, you feel it. You love it. This is your life, your craft. Fucking get into it.

This is where the failure comes in, where its so easy to fuck up, yet so easy not to as well. All it takes is attention and practice.

You form it, you let it rise. You bake it.

You've done it. You have taken some of the most simple ingredients on earth, captured them, and turned them into something 1000 times more than its parts. That is craft at its most primal.

Making dough has taught me how to fail. Not gracefully at first - there was cursing and throwing things across the kitchen. But these simple ingredients can become an art, and if you learn how to fail, you will eventually learn how to succeed.

So please, learn to make some dough.

If you're really interested in making dough, these are the people and books that taught me:

• Tartine Bread
• King Arthur Flour Bread
• Crust and Crumb
• Frankies (Pasta)
• A16 (Pizza)