There are countless more examples like this, but I was generally interested in reading a more detailed discussion about the topic. And to be honest I was secretly hoping for the book to tell me how to resolve those dilemmas, “to hand me a calculator” - as the author calls it - that gives me a result whether I’m allowed to enjoy some art or not. But obviously that’s not how it’s going. The author discusses fairly in depth the problematic lives and behaviours of a number of authors and explores the spectrum of “monstrosity” and the question when the threshold is crossed. And one of the most interesting perspectives for me was this notion of how viewing of art is influenced by the artist as well as the audience:

Consuming a piece of art is two biographies meeting: the biography of the artist that might disrupt the viewing of the art; the biography of the audience member that might shape the viewing of the art. - p. 80

And also the discussion of what consuming art says about oneself (or any other person for that matter). Especially given the very limited influence one has as a consumer on how things go with the author of the art:

The way you consume art doesn’t make you a bad person, or a good one. You’ll have to find some other way to accomplish that. - p. 242

Obviously this topic is extremely nuanced and cases of artists being or turning out to be “monsters” vary wildly. And while the book didn’t give me the hand holding guide I was initially hoping for, it definitely gave me more to think about on this topic.

The book touches on some topics from the aforementioned paper, but expands on it a lot. A big focus point is the modeling of safety culture after “authoritarian high modernism” similar to the discussion in James C. Scott’s book “Seeing Like a State:

Authoritarian high modernism believes that every aspect of our lives and work can be improved with rational planning, with better techniques and more science. - p. 35

From there the modeling of all safety culture to follow a standard, be controlled centrally and synoptically legible (i.e. loose all nuance and detail to become “understandable” on the highest level of control) leads to an eroding of safety in favour of compliance and authority.

A suggested path more towards the localized understanding and handling of safety is to take some pages out of the book of anarchism (very distinct from anarchy which the book goes into a lot of detail about). In anarchism as an idea the fostering of communities over centralized control and imposed standards and the organization of communities on a voluntary, cooperative, horizontal basis. Which reminded me a lot - unsurprisingly - of themes in David Woods’ paper “The Strategic Agility Gap”.

Overall a very interesting book and definitely recommended if you work at all with an organizational safety setup.

1. The effort of unsubscribing and re-subscribing to existing newsletters
2. Sometimes newsletters aren’t easily separated from the address I use for generally using a service (like a shop I buy from but also get newsletters)
3. My newsletters are linked to an email address that is not mine

So I wanted an easier way to benefit from the service with less hassle and also make it easier to decide to stop reading newsletters through the service without doing another unsubscribe and re-subscribe cycle. Luckily for me I still run my own IMAP server that runs procmail for filtering my emails. I mostly use it to filter out some spam and move messages from mailing lists I am subscribed to into their own folders.

In order to take advantage from the setup for getting my newsletters into feedbin more easily I first created a new email address in Feedbin as a kind of catch all for newsletters. Next up I configured a procmail rule for each email address I get newsletters from with the following form:

:0
* ^From.*news@coolwebpage.de
* ^List-Unsubscribe.*
{
  :0 c
  ! cool.address.333@feedb.in
  :0:
  $HOME/Mails/newsletters/
}

This matches any email sent from the configured address that has a List-Unsubscribe header as well. While this isn’t a perfect fit it’s a requirement for newsletters being treated properly by major mail providers and not marked as spam. Having both rules in there also makes sure that emails that aren’t newsletters from that same address (say purchase receipts from a shop) don’t get matched. Then there is a rule body that first forwards the mail to Feedbin to turn into a feed item. And then also files it into a newsletter folder. That second part is mostly a backup for me so I have the messages handy still if something gets lost somewhere.

So now if I notice a newsletter in my inbox, I can just add the sender address to a list and my configuration management system on my IMAP server generates a new procmail rule to filter it and send it to Feedbin from now on. And while procmail definitely doesn’t have the easiest syntax to understand and get going with, I really enjoy this workflow it enables for me. And many mail providers also allow this sort of things with their rule system.

Happy feed reading!

I think that’s an absolutely awful thought and extremely cruel way to talk and think about other humans. So many years ago (I don’t remember when exactly but probably somewhere around 2014) I started to instead use the phrase “vacation factor” at work. With the definition of “what’s the risk of information, knowledge, or skills being inaccessible when a team member suddenly decides to go on vacation”. As a European, vacation to me means it’s impossible to contact them because they don’t have their laptop with them, don’t check slack or email, and otherwise don’t answer work calls. As it should be. So the person needs to be assumed completely inaccessible.

This way of thinking about it sounds a lot nicer to me. And it - in addition - normalizes taking frequent vacations as it also should be. It also helps team members think of the fact that coworkers go on vacation and plan for it. Hopefully making the “guilt of taking vacation” that definitely exists for some people also grow smaller.

I’m sure I’m not the first person to think of this. But I like the phrase and think it should be used more often.

Over time and with a couple of life events, work changes, and moving countries happening my practice also changed. I had long periods over the last years where I didn’t meditate at all because it would slip my mind or not fit in the day. And while I’ve been meditating again almost daily for the last year or so I also changed my practice away from the guided meditations and towards exclusively using what the app called “Timed Meditation” that is just planning rain sounds (or any other sound from a collection) and stops after a configured amount of time. Basically Calm had been reduced to a fancy timer with rain sounds and apple health integration to record mindfulness minutes for me. Which is fine, I think it was doing a decent job, setting aside the fact that recording mindfulness minutes for open ended meditations (those that just keep on going until you hit stop yourself) just broke at some point and support didn’t seem to care much about it. The 40 Euros a year seemed a bit unnecessary to me but also not enough to worry about it.

But then I randomly stumbled upon the app in the App Store. And since I first downloaded the app all these years ago, Apple added the so called “App Privacy” section that details what kind of data an app is gathering and using as per their privacy policy. And that section for Calm presented me with this:

And it seemed like a fairly “un-zen” amount of data collection for a fancy meditation timer with rain sounds to me. I had already been in a mindset for a while where the overeager data collection of a lot of modern software was annoying me. And I didn’t want to think about that while meditating at all. So I very quickly decided to deinstall the app and cancel the subscription.

But since I don’t want to stop meditating and having a timer like that has been really helpful. I wanted a replacement. But I also didn’t want to have to deal with researching a new app and then maybe have them also change privacy policies at some point. I was also curious how much the “fancy timer with rain sounds and HealthKit integration” quip was actually true for me. And so I thought if that is true I should be able to build an equivalent workflow in the Shortcuts app. So I searched for an album with rain sounds on Apple Music and started clicking together a Shortcuts workflow. And to my surprise with a bit of tinkering and about 15 minutes I managed to finagle something together.

The workflow basically does these steps:

1. Ask for how many minutes I want to meditate
2. Set a timer for the amount
3. Start playing the rain sounds album
4. Enable the “Meditation focus” that turns off notifications
5. Then waits the number of minutes
6. After time is up it stops the music
7. Disables the meditation focus
8. And then logs the amount of time as “Mindfulness Minutes” in HealthKit

The screenshots below show what it looks like in the shortcuts app.

And I then tried out the workflow and it really works for me. Granted this doesn’t really have any of the advanced options, there’s no guided meditation, no fancy landscape pictures it shows, there is no open ended meditation option. But for my habit of meditating it does exactly what I need without gathering a whole bunch of data about me that I haven’t configured to be recorded.

It’s a bit bittersweet because I’ve been using Calm for so long. But for me the trade-off of giving all this data to another company wasn’t worth it, especially since I already pay for the product. And this Shortcuts workflow feels like a very nice and adaptable alternative.

All in all I throughly enjoyed this book and can highly recommend it.

Overall I got a lot of interesting inspiration from the book though. And even though I was already familiar with the ideas of emergent behaviour, non-sequential accident models, and learning from complex socio-technical systems, there was still a lot of angles that spurred my interest to go deeper on. And the bibliography of the book to me was a rich source of things I want to read next.

Key takeaways

The paper discusses the apparent paradox of “plan and conform” (aka centralised control) and “plan and revise” (aka guided adaptability) perspectives in safety management. As the paper discusses, this paradox is only apparent as both are needed in conjunction to navigate an ever faster changing world full of brittleness.

Detailed discussion

The paper starts of with an introduction to failure as a result of brittle systems:

“Failure is due to brittle systems, not erratic components, subsystems, or human beings.” (Woods, 2023, p. 2)

Which sets the theme for failure (and success for that matter) being an emergent property of a system that can’t be explained by looking at the individual components alone. This is a very common view on failure in complex systems (see e.g. “Engineering a safer world”) and very important as well later on when discussing the limits of the “plan and conform” perspective.

Brittleness as a core emergent property of systems is subsequently defined as:

“Descriptively, brittleness is how rapidly a system’s performance declines when it nears and reaches its boundary.” (Woods, 2023, p. 3)

Following from the definition of brittleness the central challenge of operating highly complex systems with emergent behaviour is then stated as:

“Because competence envelopes are bounded, a core question for all systems is—how does the system perform when events push it near or beyond the edge of its envelope?” (Woods, 2023, p. 3)

And this is very interesting I think because it’s not just the core question for all systems from an engineering and design standpoint. But also the core question for safety management. With the general task of safety management being to keep the system within its operating boundaries (setting aside the problem of knowing what those boundaries are) it is also core to safety management.

Next up the apparent paradox “Command-Adapt” is described in a contrasting description of upper-echelons view of the work and units-of-action view of the work (essentially the blunt-end/sharp-end contrasted view on work). And with that view of the work also comes a view on incidents and failures. For the blunt end:

“Incidents and failures generally are diagnosed as failures of operational personnel to work-to-rule/role/plan which then leads to new pressures to conform. This is the systems architecture that underlies an emphasis on rule compliance in safety management.” (Woods, 2023, p. 4)

And for the sharp end as the other side of the apparent paradox:

“The central theme of the guided adaptability perspective is “plan and revise”—being poised to adapt. This perspective recognizes that disrupting events will challenge plans-in-progress, requiring adaptations, reprioritization, and reconfiguration in order to meet key goals given the effects of disturbances and changes.” (Woods, 2023, p. 4)

And with these two perspectives (“follow the plan as it describes the safe boundaries of system operations” vs “things aren’t gonna go as planned and we need to change things around”) defined, the apparent paradox that you have to choose one or the other for how safety management is done is outlined. And then immediately uncovered as a wrong contrasting of approaches:

“Empirical studies, experience, and science all reveal that the paradox is only apparent: “good” systems embedded in this universe need to plan and revise—to do both. And the necessity of both is evident in the need to manage the risk of brittleness while coping with the side effects of growth and change” (Woods, 2023, p. 5)

And to me this is really the core of the paper. That this strict splitting up of two perspectives on safety management that we often see (plan-and-follow vs adapt-and-improvise, blunt-end vs sharp-end, Safety I vs. Safety II, etc) is largely the wrong question. Because we need to understand that both have their place and there needs to be an understanding of the trade-offs of when one is being employed rather than the other. And the problems arise when this isn’t recognised:

“The paradox dissolves, in part, when one realizes guided adaptability depends in part on plans. The difficulty arises when organizations over-rely on plans [7]. Over-reliance undermines adaptive capacity when beyond-plan challenges arise. Beyond-plan challenges occur regularly for complex systems. The catch is: pressure to comply focuses only on the first and degrades the second.” (Woods, 2023, p. 5)

In order to underline the fact that plans eventually fall short 2 classical assumptions about plans are discussed:

1. Plans can completely specify actions
2. Rationalisations about why findings about shortcomings of plans only apply to other areas and not one’s own

From the belief that the first assumption is true, it is usually derived why work-to-rule should be a guiding principle of safety management:

“If plans can fully specify actions, or nearly so, then work-to-rule/role/ plan is sufficient for productive and safe systems.” (Woods, 2023, p. 5)

This is the very common and alluring perspective that work is much more algorithmic rather than heuristic. This assumption that it’s possible to fully specify work also underlies for example the frequent over-eager and over-optimistic assumption of how much work can be (easily) automated (e.g. by a script or so-called “AI”).

Related to that assumption (and illusion) of control then rationalisations are produced of why one is in a special case and the ample findings of short comings of plans don’t apply.

“The usual response from organizations to these classic findings is simple: my world is stable and not like space operations, military operations, and emergency or critical care medicine. In my world variability can be blocked or suppressed, minimizing the need for adaptation since work-to-plan/role/rule will reliably produce desired outcomes.” (Woods, 2023, p. 7)

This rationalisation according to the paper is based on several erroneous assumptions:

• Surprises occur rarely
• It’s easy to know when a plan needs to be modified
• It’s quick to put modified plans in action
• Interdependencies are easy to limit and be analysed and modelled a-priori
• Effects of surprise can be easily compartmentalised and contained away from interdependencies

Some of these might be true for a moment but aren’t overall true throughout operation and especially lifecycle (design, growth, adaptations) of a system. And while these assumptions have been shown to be true and rediscovered over and over through research as well as experience they still serve as a kind of feedback loop to statement 1 and the call for compliance.

In order then to reconcile the two apparent-paradoxical perspectives is a reconceptualisation of plans through the lens of adaptability through 4 parts:

1. Plans are resources for action
2. Plans are necessary to recognise anomalies
3. Plans (and Automata) are competent but brittle
4. People (with the right help) provide the extra adaptive capacity to mitigate brittleness

This is to recognise that plans are useful as a starting off point and a resource to grab from when action is needed, but not as a strict specification. And one very important point is that improvisation and adaptability at the sharp-end requires the pre-requisite step of detecting anomalies and deviations. This is a lot harder without a baseline of “normal” which plans can provide. In terms of automation it needs to be recognised that automation (which includes various forms of AI in their respective hype cycles as well) is competent but brittle. And that there is a persistent believe that it just needs a new push of the technology to overcome this.

“Studies looking at joint systems of people and AI or operators and advanced automation revealed the fundamental brittleness of automata regardless of the underlying technology [13].” (Woods, 2023, p. 8)

And getting to the people part (which is the source of adaptive capacity) it’s very important to recognise that challenges and near-misses happen much more often than expected. And that created control systems like automation are subject to the same pressures as the systems they are supposed to control. And thus the same risk for brittleness:

“All systems are developed and operate with finite resources and live in a changing environment. As a result, plans, procedures, automation, agents, and roles are inherently limited and unable to completely cover the complexity of activities, events, and demands.” (Woods, 2023, p. 9)

And so the paper concludes on a way to perform work and safety management which is dubbed “Plan and revise: Guided Adaptability”. Which still means that there should be plans for the work to be done that are intended to be followed until it doesn’t make sense any more. The complement then is to learn from how they don’t make sense anymore and include that in the revision of plans for the future based on the best source there is for adaptations: humans.

“The irony is you can only monitor how well plans fit the world by understanding how people have to adapt to fill the gaps and holes that inevitably arise as variability in the world exceeds the capability of plans and the competencies built into any system [12].” (Woods, 2023, p. 11)

And to make it clear that adaptation itself is also subject to adaptation and not some perfect state of behaviour, the following quote towards the end of the paper is extremely apt in my opinion:

“You will have to establish the continuous feedback/learning loop in order to adapt how you adapt.” (Woods, 2023, p. 13)

Personal thoughts

I really liked the paper and the way it made the trade-offs of both perspectives on safety management very clear. It’s way too often the case in my opinion that a silver bullet solution is sought and once something is assumed as such, all the rest gets discarded. When in reality trade-offs and “best of both worlds” approaches to real world problems are much more likely to yield much better results. The different levels of views on the work in terms of high level planning and low level implementation also reminded me a lot of the waterfall vs agile and scheduled release vs continuous deployment discussions that are happening in technology where often one is seen as the superior approach over the other. But in reality even agile processes need longer term planning to fit into the bigger picture. And even continuous deployment means you are able to deploy whenever not that you have to. And sometimes planning a deploy to match the larger circumstances (outside of another teams test, maybe not on a Friday 😬, or even it can wait till the next morning) makes much more sense rather than deploying something as soon as you got the code review approved. As the famous mature engineering proverb goes

It’s trade-offs all the way down

Notes

Abstract

“The central theme of the centralized control perspective is “plan and conform”. The central theme of the guided adaptability perspective is “plan and revise”—” (Woods, 2023, p. 2)

“The paradox dissolves, in part, when one realizes guided adaptability is a capability that builds on plans. The difficulty arises when organizations over-rely on plans. Over-reliance undermines adaptive capacity when beyond-plan challenges arise. Beyond-plan challenges occur regularly for complex systems.” (Woods, 2023, p. 2)

8.1 Introduction: Failure is due to Brittle Systems

“Failure is due to brittle systems, not erratic components, subsystems, or human beings.” (Woods, 2023, p. 2)

“Descriptively, brittleness is how rapidly a system’s performance declines when it nears and reaches its boundary.” (Woods, 2023, p. 3)

• Is a system with tight boundaries also considered brittle?

“Because competence envelopes are bounded, a core question for all systems is—how does the system perform when events push it near or beyond the edge of its envelope?” (Woods, 2023, p. 3)

“With the right forms of adaptive capacity, systems have capabilities to anticipate bottlenecks ahead, to synchronize activities across roles and layers for mutual assistance as stress grows, and possess the readiness-to-respond to reconfigure and reprioritize activities to fit the challenges [5].” (Woods, 2023, p. 3)

8.2 The Command-Adapt Paradox

“Incidents and failures generally are diagnosed as failures of operational personnel to work-to-rule/role/plan which then leads to new pressures to conform. This is the systems architecture that underlies an emphasis on rule compliance in safety management.” (Woods, 2023, p. 4)

“The concern is how to keep pace with changing situations to mitigate the risk of brittle collapse.” (Woods, 2023, p. 4)

“From this perspective, safety staff support sharp end roles by putting in place organizational features that allow mutual assistance, or reciprocity, as situations deteriorate in the face of challenges [7].” (Woods, 2023, p. 4)

“The central theme of the guided adaptability perspective is “plan and revise”—being poised to adapt. This perspective recognizes that disrupting events will challenge plans-in-progress, requiring adaptations, reprioritization, and reconfiguration in order to meet key goals given the effects of disturbances and changes.” (Woods, 2023, p. 4)

“Empirical studies, experience, and science all reveal that the paradox is only apparent: “good” systems embedded in this universe need to plan and revise—to do both. And the necessity of both is evident in the need to manage the risk of brittleness while coping with the side effects of growth and change” (Woods, 2023, p. 5)

• This is I think a very crucial part calling out the fact that trade offs and not ultimates are the way to go

“The paradox dissolves, in part, when one realizes guided adaptability depends in part on plans. The difficulty arises when organizations over-rely on plans [7]. Over-reliance undermines adaptive capacity when beyond-plan challenges arise. Beyond-plan challenges occur regularly for complex systems. The catch is: pressure to comply focuses only on the first and degrades the second.” (Woods, 2023, p. 5)

8.3. Classic Findings on the Limits of Plans, Procedures, Automata

8.3.1. Can Plans Completely Specify Actions?

“If plans can fully specify actions, or nearly so, then work-to-rule/role/ plan is sufficient for productive and safe systems.” (Woods, 2023, p. 5)

• (Woods, 2023, p. 5) This is the automation fallacy

“Keeping pace with events invokes skills, forms of cognition, and coordinated activity over multiple roles that cannot be specified in procedures.” (Woods, 2023, p. 5)

• (Woods, 2023, p. 5) Algorithmic vs. Heuristic

“(a) plans will miss the potential for bottlenecks, overload, and oversubscription of key assets and contingency backups (this is the risk of saturation) and (b) plans will always tend to lag change in the real world. And modifying plans will lag the changes already underway” (Woods, 2023, p. 6)

“Hidden interdependencies are a potent source of saturation and lag as problems in one area push saturation to others, diagnostic work has to track effects at a distance from the originating disruption, and an expanding set of roles and players have to coordinate and synchronize their activities, often across organizational boundaries, to resolve losses of valued services [10, 29, 31]” (Woods, 2023, p. 6)

8.3.2. Rationalizations

“When an incident occurs, the limits of some components have to be part of the story (a) given the trade-offs that were necessary since resources are limited and goals conflict and (b) given that the system and its environment continue to change.” (Woods, 2023, p. 7)

“Believes the effects of surprise can be compartmentalized, whereas actually, surprises compound and spread over the extensive interdependencies in all modern systems.” (Woods, 2023, p. 7)

“In the aftermath of incidents and breakdowns, the assumptions lead to increased pressure for compliance rather than learning the importance of guided adaptability” (Woods, 2023, p. 7)

8.4. Reconceptualization

1. Plans are Resources for Action

“The finding that plans only function as resources for action—not specifications is generally traced to [11].” (Woods, 2023, p. 8)

“This is highlighted in definitions of skill: the ability to adapt behavior in changing circumstances to pursue goals despite trade-offs” (Woods, 2023, p. 8)

2. Plans are Necessary to Recognize Anomalies

“To see events and changes as unexpected requires a strong appreciation of what is typical, standard, or even “normally” abnormal” (Woods, 2023, p. 8)

“Seeing what doesn’t fit your model of what has been going on, or what should be going on, or what usually happens is a form of insight” (Woods, 2023, p. 8)

3. Plans (and Automata) are Competent but Brittle

“Studies looking at joint systems of people and AI or operators and advanced automation revealed the fundamental brittleness of automata regardless of the underlying technology [13].” (Woods, 2023, p. 8)

“the problem identified was the way the new capability was deployed produced competent but brittle systems.” (Woods, 2023, p. 8)

“Risk of brittleness is universal.” (Woods, 2023, p. 9)

4. People (with the right help) provide the extra adaptive capacity to mitigate brittleness

“(a) challenges occurred much more often than stakeholders realized, and (b) people in some roles were the critical source for resilient performance despite the stresses, risks, uncertainties, threat of overload, and bottlenecks” (Woods, 2023, p. 9)

“All systems are developed and operate with finite resources and live in a changing environment. As a result, plans, procedures, automation, agents, and roles are inherently limited and unable to completely cover the complexity of activities, events, and demands.” (Woods, 2023, p. 9)

“Without this capability for extensibility, brittle collapse would occur much more often than it is observed [6].” (Woods, 2023, p. 10)

“Adaptation is not about always changing the plan, model, or previous approaches but about the potential to modify plans to continue to fit changing situations.” (Woods, 2023, p. 10)

8.4.1. Plan and Revise: Guided Adaptability

“The new science shows that this assumption is guaranteed to be wrong in the future, regardless of how well the plan has guided performance in the past. The timing on this guarantee is linked to the pace of change within and around the organization and how those changes expand the tangle of interdependencies it exists within.” (Woods, 2023, p. 10)

“The irony is you can only monitor how well plans fit the world by understanding how people have to adapt to fill the gaps and holes that inevitably arise as variability in the world exceeds the capability of plans and the competencies built into any system [12].” (Woods, 2023, p. 11)

“Monitoring how people adapt to make the system work does not constitute approval that these adaptations are the “best” given the trade-offs faced in different situations. What is “best” is itself a dynamic judgment that can and should change as challenges vary— reprioritization rebalances goals in the trade space to fit the situation.” (Woods, 2023, p. 11)

“Driving gap-bridging adaptations underground also makes it harder to recognize how plans do not fit the changing patterns of variability in the world.” (Woods, 2023, p. 11)

“Recognizing what adaptations are going on allows one to see the resources—physical, cognitive, collaborative, and others—that people draw on to produce resilient performances in the face of challenges small and large.” (Woods, 2023, p. 11)

“(a) about challenges that recur in general even though the specifics vary in individual events and (b) about the ways people work and coordinate to handle challenges.” (Woods, 2023, p. 12)

“If safety is about “repair after something goes wrong”, no organization can keep up with the pace of change, growth, and scale of modern systems and activities [32].” (Woods, 2023, p. 12)

“However effective your organization has become, however you have developed and deployed new capabilities to grow, whatever your record of past improvement in reliability/ productivity/efficiency, and whatever the promises of new capabilities to-be-deployed, the world, in the near future, will produce challenges that go beyond the competencies embodied and require adaptive capacity to stretch.” (Woods, 2023, p. 12)

“You will have to establish the continuous feedback/learning loop in order to adapt how you adapt.” (Woods, 2023, p. 13)

Key Takeaways

This paper outlines how past successes drive the creation of bigger and ever more complex systems. And the risk of not being able to adapt fast enough to changing environments and systems. The gap between the adaptation possible and the adaptation required by an organisation is called “The Strategic Agility Gap” and a fundamental requirement to be able to bridge that gap is the ability to react to unplanned failure called “SNAFU catching”.

Detailed discussion

The paper approaches the topic of how organisations deal with the challenge of keeping up with an ever accelerating pace of change in environments and systems that get more and more complex. While the focus here is mostly threats and challenges in the term of surprising failures. It’s also mentioned that slow adaptability also has implications for the ability to seize opportunities. Organisations of course aren’t generally not adapting at all. They are always navigating within their range of agility to adapt to new situations. However the problem arises when this “strategic agility” as it is called in the paper isn’t enough to account for the surrounding changes (e.g. systems and environment) and provide adaptability for new challenges. The difference between the existing level of agility and the required agility to match the surrounding pace of change is what is called “the Strategic Agility Gap”:

“It is a mismatch in velocities of change and velocities of adaptation (Fig. 1)” (Woods, 2020, p. 96)

This is also outlined nicely in this diagram in the paper:

In order to illustrate this gap the paper discusses two cases: The 2012 Knight Capital Collapse during which an organisation wasn’t prepared to adapt to their changed environment and an incident related to “runaway” automation. And the case of large transport firms coping with the impacts and fallout of Hurricane Sandy in the same year.

In the case of the Knight Capital collapse a list of 5 risks are outlined that contributed to the organization’s inability to deal with the incident:

“First, small problems can interact and cascade quickly and surprisingly given the tangle of dependencies across layers inside and outside the organization. Second, as effects cascade and uncertainties grow, multiple roles struggle to understand anomalies, diagnose underlying drivers, identify compensatory actions. Third, difficulties arise getting authorization from appropriate roles to make non-routine, risky, and resource costly actions, while uncertainty remains. Fourth, all of the above take effort, time, and require coordination across roles. […] Fifth, when critical replanning decisions require serial communication vertically through the levels of the organization, responses are unable to keep pace with events.” (Woods, 2020, p. 98)

And in contrast the transportation firms were “poised to adapt” in the scenario they were dealing with due to constant preparatory work that preceded the incident that put them in a position to be able to quickly react:

“Upper management developed mechanisms for this shift prior to particular challenge events. As hurricane Sandy approached New York, temporary teams were created quickly to provide timely updates (weather impact analysis teams).” (Woods, 2020, p. 98)

“These mechanisms existed because this firm’s business model, environment, clientele, and external events regularly required adaptation as surprises were a normal experience.” (Woods, 2020, p. 98)

And in contrasting the cases, the paper again emphasises the importance of being able to adapt not only within a layer of work but also vertically throughout an organisation to push decision making and coordination onto layers that allow for timely reaction to the current situation:

“In the strategic agility gap, the challenge for organizations is to develop new forms of coordination across functional, spatial, and temporal scales—otherwise organizations will be slow, stale and fragmented as they inevitably confront surprising challenges.” (Woods, 2020, p. 99)

The paper then zooms out to a more high level view on systems and their messiness. Highlighting the fact that all development and operations of systems are limited by the fact that resources are finite. And thus preparatory vehicles like plans and automation can never cover the full extent of complexity of a given system:

“All systems are developed and operate given finite resources and live in a changing environment [5]. As a result, plans, procedures, automation, all agents and roles are inherently limited and unable to completely cover the complexity of activities, events, demands, and change” (Woods, 2020, p. 99)

This is a point that is also expanded upon in Woods’ 2023 paper “Resolving the Command–Adapt Paradox: Guided Adaptability to Cope with Complexity“) in greater detail to highlight the limitations of plans and automation.

As an essential capability in dealing with this predicament of having to quickly react to an always changing world that is only predictable in very narrow terms, Woods notes what him and Cook termed “SNFU” catching: The ability of people to adapt outside of standard plans:

“SNAFU catching, however technologically facilitated, is a fundamentally human capability essential for organizational viability […] people in some roles provide the essential adaptive capacity for SNAFU catching, though this may be local, underground, and invisible to distant perspectives [12].” (Woods, 2020, p. 99)

And with SNAFU being the normal operation, even though as the paper argues it is often rationalised away as a fringe phenomenon to put pressure on work-to-{plan,role,rule} and with that hindering the effectiveness and visibility of SNAFU catching:

“The compliance pressure undermines the adaptive capacities needed for SNAFU catching (initiative), creates double binds that drive adaptations to make the system work ‘underground,’ and generates role retreat that undermines coordinated activities.” (Woods, 2020, p. 100)

In the conclusion the paper then outlines a path out of the Strategic Agility Gap which is continuous adaptation:

“For organizations to flourish in the gap they need to build and sustain the ability to continuously adapt.” (Woods, 2020, p. 101)

It draws on lessons from Web Operations where outages and near outages are common in a world where systems are constantly changing and due to past success are enabled to grow and change continuously. It is also a field that has and is becoming more and more important as almost every system is becoming a digital system at least in parts. And in this complex world the limitations of planning are becoming more and more apparent with the underlying layer of resilience to not have them fail catastrophically all the time is SNAFU catching:

“Organizational systems succeed despite the basic limits of plans in a complex, interdependent and changing environment because responsible people adapt to make the system work despite its design—SNAFU catching. The ingredients are:” (Woods, 2020, p. 102)

With the mentioned ingredients being:

• anticipation
• contingent synchronisation
• readiness to respond
• proactive learning

With these behaviours and properties often being severely reduced by the pressure to work-to-{plan/rule/role}, compliance, and worst the fear of sanctions when any of these expectations are violated. And in summary:

“Strategic agility gap arises as organizations’ trajectory of improvement cannot match the emergence of new challenges, risks, and opportunities as complexity penalties grow (Fig. 1). To flourish in the gap requires organizations to build and sustain capabilities for SNAFU catching.” (Woods, 2020, p. 103)

Personal Thoughts

This paper was fascinating to me in a couple of ways. But having the visual of “a gap” in the capacity and capabilities of an organisation was to me an extremely useful way of framing the problem. The paper also casually gave me the best and most succinct definition of DevOps I have ever seen in a decade or so of blog posts and talks trying to define it:

“Reciprocity in collaborative work is commitment to mutual assistance.” (Woods, 2020, p. 103)

Neoliberalism has contributed to an environment where regulation and rule making is done on a central per organisation basis that often is implemented by non-experts and gets in the way of frontline workers while providing (next to) no improvements for them in terms of safety and incident reduction. The path to safety management after neoliberalism is through “the restoration of professional judgement” and increased and encouraged participation and compensation of workers at the sharp end.

Detailed discussion

The paper starts with a short introduction into neoliberalism and the definition that is used throughout the paper based on

“Most scholars tend to agree that neoliberalism is broadly defined as the extension of competitive markets into all areas of life, including the economy, politics, and society” (Dekker, 2020, p. 1)

With that as a starting point an example is cited from studying impact in the mining industry and summarising a set of lived experiences of safety management in a neoliberal context:

• “Rules and regulations that have become overly obstructive;
• Safety systems that encourage a dumbing down of individuals and a dilution of personal autonomy and discretion;
• Higher stress levels due to a sense of loss of control;
• Considerable wasted effort;
• Systems that have become far too complicated;
• Common sense and initiative that have been discouraged;
• Cynicism about slogans, stated priorities and the motivation behind rules;
• Safety staff detached from the front line—either by their inappropriate experience or because of their physical location being remote from the workplace (cf. Woods, 2006).” (Dekker, 2020, p. 2)

With that it is also discussed that statistically a reduction in workplace fatalities hasn’t happened under this shift in safety management. And the safety management that is often encountered today is positioned as a “corollary of neoliberalism” following from the push to deregulation, encouraging organisations to make up their own centralised safety management with a focus on responsibilization of workers and a tap into a free market of “safety services” that are introducing outside auditing, researching, training, etc as well accreditation and consultancy.

From this assessing the current state of neo-liberal tainting of safety management as a non-working setup, the discussion is then switching its focus on the core question of what safety management could look like after neoliberalism.

The first focus area then is what is called “the restoration of professional judgement” in 4 areas:

1. The management of complex risks The first area here is related to the familiar idea of “sharp end professionals” having an immense amount of expertise that can and should be tapped into:

“Highly practiced professional skills of pattern recognition, scenario formulation and mental simulation of the execution of possible decision options (Klein, 1998) have shown to be far better at managing dynamic, complex situations than imposing fixed rules that supposedly reduce uncertainty.” (Dekker, 2020, p. 2)

2. Deregulation and reregulation Here it is pointed out that deregulation in spirit is fairly aligned with the idea of worker autonomy and authority needing to be positioned where expertise is located:

“Deregulation aligns, in spirit and in principle, with the ideas and ideals of worker discretion and autonomy, and with the notion that decision authority needs to flow to where expertise sits” (Dekker, 2020, p. 2)

But with the caveat that this needs to be done intentionally and mindfully. And not in a way that leaves a vacuum of safety operations and practices:

“But an unanticipated effect of deregulation has been not a reduction but a displacement of rulemaking, documentation and inspection activities.” (Dekker, 2020, p. 2)

3. What governments can do after neoliberalism As for the questions of what governments can do this is split in two parts of reassertion of ownership as well as re-regulation. However this re-regulation needs to then take a new form, away from pure expectation of compliance and towards a Safety II mindset of safety management:

“Some regulators have become interested in more formally examining why things go right (consistent with Safety II principles (Hollnagel, 2017)), and trying to enhance or assure the presence of the capacities (or ‘resilience potentials’) in an organization’s people, pro- cesses and systems that make it so (Jacobsen, 2017).” (Dekker, 2020, p. 3)

4. What organizations can do after neoliberalism And finally the opportunities of organisations post neoliberalism which reads in this paper like the hardest (but also maybe most important) part of the shift. First of all the almost natural tendency of “solving safety with bureaucracy” needs to be tackled:

“as soon as safety is involved, there seems to be an irresistible push towards a wider scope of norms, procedures and processes, whatever the context” (Dekker, 2020, p. 3)

“A recent poll in the mining industry showed that the majority “of the workforce feels things are being imposed on them that add no value, wastes their time, adds to their frustration and, at worst, creates a disconnect by removing control over their work”” (Dekker, 2020, p. 3)

What this de-buraeucratizing of safety management looks like is again related to the first point of recognising front-line expertise:

“De-bureaucratizing safety means putting safety expertise closer to the nuances, ‘messy details’ and quotidian risks of actual practice—as well as to operational decision-makers (CAIB, 2003; Galison, 2000; Roe, 2013; Woods, 2006).” (Dekker, 2020, p. 3)

And also an emphasis on collaboration, relation, and empathy that is achieved through working together towards a common goal (safer operations) rather than imposing rules and regulations:

“deliberately avoided using bureaucratized safety systems, and instead built on their collective responsibility for mitigating risk by reframing official safety programs in terms of kinship—specifically the ties of relatedness crew members create with each other in their everyday work.” (Dekker, 2020, p. 3)

The paper then talks about participatory equality and workers’ compensation as a an area to change after neoliberalism. While not directly related to safety management they are outlined as requirements for changes to become sustainable.

“One is participatory equality: the actual, meaningful involvement that workers have in decisions about the design, preconditions, implementation, execution, circumstances, monitoring and remuneration of their work—including of course those aspects to do with safety.” (Dekker, 2020, p. 3)

And equally important is the shift way from the neoliberalism induced view of risk and injuries being a sole responsibility of the worker and attempts for improvements stopping there instead of looking at organisations as a whole:

“The individualization of workplace risk, or ‘responsibilization,’ re- fers to safety programs and violation notices targeted at workers, not companies, which has helped tilt the distribution of work-related injury costs in favor of corporate interests” (Dekker, 2020, p. 4)

The paper then closes with an overview of safety and global capitalism after neoliberalism in which studies are examined that found a correlation “between economic globalisation and the probability of industrial accidents”. And interestingly also a negative correlation to free speech:

“In particular, freedom of speech is negatively correlated with industrial disasters.” (Dekker, 2020, p. 4)

And the concludes with the conclusion that complex risk in safety after neoliberalism should be approached by

“trusting and enabling practitioner decision discretion; by finding a new balance between written guidance and risk appetite, between professional judgment and risk competence.” (Dekker, 2020, p. 5)

Personal Thoughts

I initially had quite a lot of problems following the argument from neoliberalism to essentially the description of Safety I management of incidents and safety as a whole. Originally I saw the inverse as more logical. And even though it is addressed in the paper as “Deregulation aligns, in spirit and in principle, with the ideas and ideals of worker discretion and autonomy, and with the notion that decision authority needs to flow to where expertise sits” (Dekker, 2020, p. 2)" it took me some mind bending to make this make sense in my head. I also don’t really have any background in learning or even thinking much about neoliberalism, so there’s a good chance I’m missing some stuff. But overall I liked how the paper made me think about these two concepts (safety management and neoliberalism) that I had never thought about in relation to each other.

Notes

Detailed discussion

The paper starts with an introduction of the usual approach for SLOs:

“Success- ratio is the fraction of the number of successful requests to total requests over a period of time (usually a month or a quarter) [” (Diwan et al., 2020, p. 1)

This plain calculation has the advantage that it’s very easy to calculate but comes with the bias of skewing towards most active users as they will represent more requests (successful as well as unsuccessful) in the overall ratio.

The general form of availability metrics are defined as

availability = good service / total demanded service

And the way service is measured here generally falls in one of two buckets. Either time based which means the time between failures is uptime and the time recovering from a failure is downtime. Which leads to a metric defined as

availability = uptime / (uptime + downtime)

This measure strongly relies on the definition and precise meaning of “failure” and “recovery”. Especially given situations of partial failure, the definition on whether failure means for all users or at least one user or something in between becomes important for the result of what it means for measuring availability. Here percentage thresholds are often introduced for the definition, to be able to say something like “failure means at least 5% of users encounter errors”. Which still comes with the problem of an arbitrary threshold. Within this definition a system with 5% error rate is treated the same as one with 0.0001% error rate given the chosen threshold as the delineation line. And the general downsides of this measure are:

• it’s not proportional to the severity of the system’s unavailability
• not proportional to users affected
• not actionable with not insight into source of failures
• not meaningful as they rely on arbitrary thresholds and/or manual judgements

For the other approach of count based availability which is much more rare and is based on the similar definition of

availability = successful requests / total requests

There is some popularity to this approach as it is easy to implement. However users ultimately care about time they were able to use a service and not the number of requests they were able to do. So this approach also comes with the downsides of:

• not meaningful as it’s not based on time
• being biased towards highly active users
• being biased due to different client behaviour during outages There is a quick discussion of synthetic probes for measuring system availability but this isn’t really discussed deeply as it comes with the downsides of not being representative of user activity and not proportional to what users experience. So for the goals of meaningful measure of uptime it’s not a good fit.

In order to improve on these

Over the years however I used it less and less, mostly because Twitter had started forcing its own URL shortening with auto-expansion when viewing a tweet on everything. And the experience of using a custom shortener was not on par with that. I’ve also almost lost the database of the shortener a couple of times because free tiers don’t usually come with backups. So I rigged up a quick GitHub Action that ran once a day, redis-dump-ed all the contents to plain text and committed them to a git repo as a low budget backup job. At this point I wasn’t really shortening anything anymore but wanted to keep the existing URLs functional. I had moved to Heroku’s own Redis service at that point and there was no real work involved to keep it running.

Fast forward to 2022 Heroku announced the end of the free tiers. And while I’m generally happy to pay for things I wasn’t convinced that just maintaining what was essentially by now a barely used URL lookup app was worth the $7/month for me. So I shut it down and thought about alternatives. I could run the app on my server that I use for a couple of things. But I really don’t want to run a ruby app + redis in my free time. I thought about implementing the shortener logic in Go and back it by something like sqlite or even just a yaml file. But again that felt like a lot of effort for not actually shortening anything.

And then I thought “this is just hosting 301 redirects, surely something nginx is good at”. And sure enough, after a quick internet search I found a stackoverflow post that provided a good example for managing a lookup map in a handful of lines of code. The core of it is basically:

# head -n 5 /usr/local/etc/nginx/mrtz_cc_redirect_map.conf
/-KmaJA 'http://lusis.github.com/blog/2014/04/13/omnibus-redux/';
/-vvREg 'http://hannahmontana.sourceforge.net/';
/-yW3mQ 'http://s3itch.unwiredcouch.com/1._tmux-20140719-180429.jpg';
/09nQKA 'http://s3itch.unwiredcouch.com/Projects-20141130-133209.jpg';
/0YK2gg 'https://speakerdeck.com/mrtazz/statsd-workshop-monitorama-2013';

# wc -l /usr/local/etc/nginx/mrtz_cc_redirect_map.conf
424 /usr/local/etc/nginx/mrtz_cc_redirect_map.conf


# cat /usr/local/etc/nginx/sites/redirect
map_hash_bucket_size 256; # see http://nginx.org/en/docs/hash.html

map $request_uri $new_uri {
    include /usr/local/etc/nginx/mrtz_cc_redirect_map.conf;
}

server {
  listen 94.130.5.59:443 ssl;
  server_name mrtz.cc;

  if ($new_uri) {
    return 301 $new_uri;
  }

  ...
}

So all I had to do was convert the plain text backup of my redis instance into the nginx map format, which was easy enough with this awk one-liner:

% head -n 5 backups/mrtz.cc/mrtz.cc.dump
SET     guillotine:hash:-KmaJA 'http://lusis.github.com/blog/2014/04/13/omnibus-redux/'
SET     guillotine:hash:-vvREg 'http://hannahmontana.sourceforge.net/'
SET     guillotine:hash:-yW3mQ 'http://s3itch.unwiredcouch.com/1._tmux-20140719-180429.jpg'
SET     guillotine:hash:09nQKA 'http://s3itch.unwiredcouch.com/Projects-20141130-133209.jpg'
SET     guillotine:hash:0YK2gg 'https://speakerdeck.com/mrtazz/statsd-workshop-monitorama-2013'

% awk '/guillotine:hash/ { split($2,a,/:/); print "/"a[3]" "$3";"}' < backups/mrtz.cc/mrtz.cc.dump | head -n 5
/-KmaJA 'http://lusis.github.com/blog/2014/04/13/omnibus-redux/';
/-vvREg 'http://hannahmontana.sourceforge.net/';
/-yW3mQ 'http://s3itch.unwiredcouch.com/1._tmux-20140719-180429.jpg';
/09nQKA 'http://s3itch.unwiredcouch.com/Projects-20141130-133209.jpg';
/0YK2gg 'https://speakerdeck.com/mrtazz/statsd-workshop-monitorama-2013';

And then chef out the nginx config and Let’s Encrypt setup for the domain to my server, change the DNS records to the server instead of Heroku¹. And voila:

% curl -sv https://mrtz.cc/-vvREg 2>&1 | grep Location
< Location: http://hannahmontana.sourceforge.net/

I really like this setup because running nginx is pretty straight forward for the small scale I use it at. And I care about keeping URLs working. So this makes me happy. I might at some point maybe want to start using it and adding URLs again. At which point I have to figure something out. But I don’t expect that to be any time soon (if at all).

But we also had this concept of “Senior rotations”, which to me always felt like an extension of onboarding. And it was one of the most fun things I got to do during my time there. The basic concept is that every Senior Engineer and above (back then that meant Staff, Principal, and Distinguished) was entitled to a 4 week rotation with a different team once a year. The idea was that for one month out of the year the more experienced engineers from any given team would be working on something completely different. Specifically we were encouraged to pick a team that was working on something very different than what our team was doing. And inversely every team was also hosting an engineer for a month once or twice a year (which is what in my experience it usually came down to).

The purpose of this - as it was explained to me - was multi-faceted. It was a way to make sure senior people gain and maintain connections across the organization, especially as the company was growing. Connections in the sense of relationships, but also which problems are being solved elsewhere, and what the impact is from changes in one part of the system to other parts of it. And having intricate knowledge of some parts of the stack or the product can often mean there are some good ideas ready to emerge when suddenly confronted with a different area. Another idea - though I don’t know how much that actually worked - was that it was a way to improve retention. For senior engineers that might feel like they’ve gotten into a rut it was a way to experience work on a different team and consider switching or find new excitement in their work. On the flipside it also meant junior engineers get to learn from more senior engineers from across the company. And it was a good way to build empathy for other teams as well. In addition the “borrowing team” needed to regularly plan for senior engineers to not be around. Which was good practice for handing projects off, documenting work, and generally increasing the vacation factor (the way better and less morbid version of the bus factor, because let’s be honest we should be planning for people going on spontaneous vacation much more than them being hit by a bus).

I’ve done two of those during my time at Etsy with wildly different experiences. And most of it came down to preparation. I did one rotation where I didn’t really plan and prepped for it. It was also a team that I was already closely working with in my regular day to day and I didn’t make sure I wasn’t gonna have to work on projects for my original team. It was during a period where there was a lot going on. So I wasn’t disconnecting from the chatter and questions for my team either. So all in all while I worked on some fun stuff and learned things, it definitely wasn’t really the change of pace and areas of focus that I wanted it to be.

The following year I really wanted to do another rotation. And as it happens we also had a reorg that meant my team was being disbanded and I was starting on a newly created team anyways. The perfect opportunity to not have to worry about any lingering projects or feeling like I abandoned my team mates during a time of high stress. And the timing matched up so that I would start on the new team after my summer vacation anyways. So it wasn’t gonna matter much if I started a month later anyways. This time I wanted to choose a team to rotate with that was about as far away from my day to day working on infrastructure as I could imagine. Which is why I was super excited to hear that the design system team was hosting senior rotators, so I immediately signed up.

I didn’t have many expectations going into the rotation other than wanting to learn about something completely different. Plus I’ve never liked the attitude that there are inherent backend and frontend engineers and that you have to be either one. It’s all just yelling at computers in different ways. So I was excited to learn more about the frontend part of the stack. My general project during that month was designing a versioning and deployment workflow for CSS (specifically SCSS) changes that we wanted to support going forward as an extension of the existing CSS build. The details are a bit fuzzy 7 years later. But basically it was a project that was somewhat rooted in infrastructure in a way that gave me a great starting point to talk to designers (who wrote large parts of CSS and JS changes themselves back then) and engineers about their requirements and workflows around CSS changes. I was in daily standups with the design system team, got to hear about the daily challenges of designers and frontend engineers, what goes into maintaining a design system, and most importantly got to ask all the “stupid” questions that came to my mind. Plus as my rotation mentor I had the fantastic Katie Sylor-Miller who is one of the most frontend savvy people I’ve ever met and I got to ask her all the questions as well. And I can’t count all the things I learned during that time. I had fantastic conversations about how to design for a brand in different shades of orange (Etsy’s brand colour back then), the implications of designing maintainable SCSS code organization, and just a whole bunch of things around design systems and frontend development for a big, busy website.

All in all I really enjoyed that rotation and all the things I got to learn through it and the people I got to know and hang out with for a month. I really appreciated that I got to get out of my comfort zone for a bit and in some ways “be a junior engineer” again. And it gave me insight into different parts of the company that I probably would’ve never gotten otherwise. As I mentioned, planning and executing senior rotations is a decent amount of effort. But I think it’s a really fantastic idea and I’ve benefitted a lot from being able to participate in it.

I’ve thought a lot over the years about why that is. What is so alluring and comforting about single cause incidents for a business. And conversely what is so hard about accepting the fact that we will never be able to prevent another incident from happening. We’ll never fully “solve” an incident and we’ll never be able to describe and map a system to the extent where we can know the full impact of every change in the future.

The base unit of work at a company is a ticket

What I’ve mostly come down to when thinking about this is the realization that the base unit of work in a corporate context is a ticket (or an issue, or whatever your tracking software calls it). The way you know what needs to be done and that you have done a certain amount of work is because there exists a ticket in a tracker. Many conversations (and “process optimizations”) in a usual work setting at certain times include a discussion about the fact that everything that is being worked on needs to be tracked in a ticket to make sure it’s all visible and surfaced. And once the work is done (or abandoned) the ticket needs to be closed to communicate its status (and sometimes status needs to be communicated on open tickets as well). These updates are then used to roll up into higher level tickets and summaries to higher up management to communicate to them what got done and what teams are working on. There is much more that could be said about this but the point I want to bring across is that the core measurement of work planned, to be done, in progress, done, or even abandoned is to have a ticket that can be closed.

Incident investigations are about learning

And now the contrast here is that reviewing and investigating incidents is about learning. Learning what was previously unknown and thus contributed to a surprise that manifested as an outage or reduced availability or data loss, or any other unwanted event that we commonly call an incident. And learning - as probably almost everyone has witnessed in some form before - is far from linear and sequential. Sometimes it’s very quick, but usually it takes its sweet time. Especially in learning through research, where there isn’t anyone already who knows the answer and who can say whether a hypothesis or an understanding is true or false, learning is far from linear. It’s full of dead ends, red herrings, misunderstandings, re-discoveries, reformulations, conversations, disagreeing opinions, and probably late nights and long weekends.

Trying to now wrap learning into a time boxed setting like work where it can be reflected by a ticket that can be closed is surely a challenge. The irony here is that a form of this already exists in arguably the main arena of learning: education. Every school and college setting knows the setup of students having to learn (and ideally understand) a topic in a given time and then pass a test to be able to mark it as done (or as learned). Essentially a trade off that attempts to summarize learning into a checkbox style format where the options are more or less either pass or fail. And many, many discussions have been had and continue to happen about this suboptimal setup where it incentivizes students to learn for passing the test and not for understanding the topic. It gives rise to many frustrations where someone who is able to recite the words or equations (potentially without having understood the meaning behind them) is given the same or a better grade than someone who took more time to dig into what something means and study additional material around the topic to improve their understanding - but didn’t do as good of a job demonstrating that in a test setting. And it leads to a considerable amount of people - not least in the software engineering world - actively despising education and its standardized testing.

But the truth of the matter is that learning can have more than one goal. There is no “true” and “false” learning. One way of learning has the goal of passing a test and the other has more the focus on establishing and deepening one’s understanding of a topic. Both are valid and they satisfy different requirements. But you should know which one you choose and what you will get out of it. You can’t not learn, but you can definitely learn different things depending on how you approach it.

Question of focus

Tying this back to corporate incident investigations we are presented with a very similar choice. Do we want to review an incident for understanding the complex interplay of contributing factors that allowed it to manifest? Or do we want to be able to just close out the ticket already and move on? Both are valid in their own way. Because there will honestly never be enough time to investigate every incident in full. There is other stuff that needs to be done as well and you’ll never get the 6 or 12 or 25 people that were closest to the incident and know the most about what happened in the same room to share all their knowledge and experience to untangle the full incident (which is already a trade off because ideally a facilitator should interview them individually to make sure there is no barrier to sharing).

However the huge downside and difference to the education setting is that there is no actual “passing the test” in work incidents. Just by doing the due diligence to be able to say we can close the ticket we don’t actually gain or pass anything. We just miss out on learning. So while it’s appealing to make it look (and feel) like an incident review was “completed”, following a linear, causal accident model to get to a single cause that lets us close the ticket, we just cheat ourselves out of valuable learning and insights.

Sidenote: MTTR

Viewing incident review as learning and incidents as a source for and opportunity to do so, also makes it a lot more clear how some common “measures” of incidents are not as useful as one might think. Let’s take the very popular metric MTTR (Mean Time To Recovery) for example, which generally is intended to denote how long it takes on average to recover from an incident. On the face of it, it makes sense. Because we want to be available as much we can and work on making sure incidents - when they happen - are as short as possible. However viewing incident handling (which comes before the review but has some overlapping properties) through the lens of learning basically lets us structure it in roughly the following phases:

1. Understand what’s wrong (e.g. too many 500 errors on the website)
2. Understand how this situation manifested (e.g. a combination of more traffic, a ramped up feature flag on a specific code path, and an upgraded app server version)
3. Understand what to change to mitigate (e.g. ramp down the feature flag for now)

Of course incident handling is more concerned with finding the most easy to change contributing factor that is still necessary but only jointly sufficient to give rise to the current incident and change it to make the incident go back from active to passive. So it’s less about getting a full (as much as possible) view on the incident - that’s what the review is for. Nevertheless in this view on incident handling it’s really three phases of understanding that we go through, so what MTTR really measures is MTTU (mean time to understanding). And thus we are back in the same situation. We are basically trying to force learning into an arbitrarily timeboxed, measurable, and summarized metric (similar to issues closed). Which again is something you can totally do. It might not be very useful and not serve you in the way you’d like it to do. Which makes it even more important to understand the limits and usefulness of a metric like that to not overly rely on it for the wrong reasons. Plus there are plenty more reasons why these measure are generally not all that useful.

Reading the book was a bit of a split for me. The author definitely does a good job (as far as I can tell) explaining physics concepts and theories in a way that is understandable and with the given topic in mind. And the fact that the book is structured as chapters where each answers a specific question also makes it a really quick and interesting read. However there have also been parts of the book where the tone bordered on obnoxious for me with the cliche undertone (or sometimes even explicit statement) that eventually everything can “just” be explained with physics and how - in the author’s view - many people (even other scientists) are incorrect in their assumptions and world views. Which I find more jarring than entertaining. But the core idea in the book that there are some views (e.g. the existence of a god) that are neither scientific nor un-scientific as they can’t be proven or disproven. But rather a-scientific as they are compatible with current science since nothing we currently know confirms or refutes them. I’ve found this to be a really good frame for thinking about the world. Overall I enjoyed the book with some caveats.

If your job is taking care of a running system, then you know how painful incidents can be. They are often sudden and unexpected. They are disruptive to your (and your coworker’s) workday. And they are a stark reminder of the fact that you are operating on inaccurate assumptions about the world your systems are operating in. Often in seconds you are forced to through a painful realignment of your understanding of the world. And at the very latest when the incident is remediated and the adrenaline starts to dissipate you are left with those irritating feelings full of hindsight and regret (especially if the incident was triggered through the deploy of a change). “How did we not foresee this?”, “Why didn’t we stop before running the command?”, “How could we let this error happen?”. “We have to make sure this never happens again!”

Prevention of incidents

This urge to prevent uncomfortable situations from happening again is absolutely natural. And we all go through it. We don’t want to be in this situation where we have no idea how something could suddenly break. Much less in a situation where we can’t be sure it won’t break again. And the next logical thing is looking for just even a single thing that makes sure we aren’t ever going to be in this situation again. And even if you already know that root causes aren’t a thing, it’s an incredibly tempting thing to try and find. Even Nietzsche said it over a hundred years ago:

To derive something unknown from something familiar relieves, comforts, and satisfies, besides giving a feeling of power. With the unknown, one is confronted with danger, discomfort, and care; the first instinct is to abolish these painful states. First principle: any explanation is better than none. Since at bottom it is merely a matter of wishing to be rid of oppressive representations, one is not too particular about the means of getting rid of them: the first representation that explains the unknown as familiar feels so good that one considers it true

— Friedrich Nietzsche, Twilight of the Idols (1888)

And so in the case of an incident we look for anything that lets us prevent this incident (or others like it) to happen again.

The problem with this kind of goal fixation on prevention (as noble as it is) is exactly that. It’s fixation on a single outcome. A single “good” state. And this means it severely limits how much can be learned from an incident. Because even as you have now realized something new about the system and the world and went through that painful realignment of your perspective, that’s still not the full picture. And it never will be. Our systems are highly complex and subject to constant change in both their code and operations as well as their environment. So even if we come up with something now that we are sure will prevent the incident from happening again, it won’t (and can’t) ever be a solution with the full view of the system. So anything we can come up with in this situation to “prevent” the incident from happening again has a high likelihood to be over specific to this current view of the world. Proposals for solutions in this situation often take the form of guard rails, check lists, over zealous automation, or other things to reduce “human error”. But those things - while well intentioned - often have the downside that they take flexibility away from the true source of resilience in a system: the human operator. And when the next incident comes around, there is a decent chance that these newly instated “improvements” are now working against operators and their ability to reason about a situation and improvise a solution. Effectively causing them to take longer with debugging and remediation in a state of reduced flexibility than they did before.

“The problem comes when the pressure to fix outweighs the pressure to learn.”

— Todd Conklin

And after all that investment there was likely only very little learned from an otherwise information rich incident. Plus there is usually no way of telling whether you actually prevented anything. Most of the time in running systems, incidents don’t actually happen. So the absence of incidents - especially the non-recurrence of a specific incident - is generally not a strong signal about the resilience impact of a single change. Often many contributing factors to an incident lie dormant for a long time only for an operator to exclaim while debugging an incident “how did this ever work?”.

Learning from incidents

So what’s the alternative? Surely we don’t just want to shrug off incidents and ignore them. That would be such a waste. Incidents are a great source of knowledge and learning. And I for one enjoy getting the chance to learn something.

In the situation of post processing an incident this means now is the time to keep an open mind and understand how we arrived at the current state of the world and our systems:

• Why are they set up like this?
• What are their current known limitations?
• What is easy/hard about changing them?

I won’t go into too much detail about how to facilitate learning from incidents here. I’ve had the fortune to collaborate on a document about this many years ago and I think it’s still a useful place to start.

But this is the discussion that should bring all your experts to the forefront (the yard?). The people that are operating the systems in question and know their in and outs. The ones that are operating alongside and within the behaviours and boundaries of those systems every day. They will be able to quickly pinpoint shortcomings, workarounds, and idiosyncrasies. But also more importantly slack and flexibilities, and processes and hacks that aren’t part of the day to day automation but are needed in edge case situations to keep the system resilient. They will be able to talk about how they decided to look at some metrics but not others to make sense of the state of the system. How they decided to go with one route for a possible fix and not the other. In a setting where learning is preferred over fixing and solving there is the chance for a lot of people to go on a journey of how to navigate a highly complex socio-technical system and make the whole organisation more resilient through the dissipation of knowledge. And while discussions of incidents often seem linear and algorithmic in hindsight, it’s also important to always remember that the operator chose one path to success amongst a myriad of other routes that may or may not have been taken. And what looks linear and algorithmic now was most likely nothing like that when it happened. Sometimes what comes out of the discussion might be more automation or it might be an additional guard rail. But the importance is that neither is it something that can be known beforehand, nor is it something that will be instated to keep the human at bay.

The point of debriefing and discussing incidents is not to keep something from happening but to make sure the tools (including automation) and support (including knowledge sharing and learning) are in place to “augment and compliment [..] human adaptive and processing capacities”.

So what to do?

Incidents don’t have to be painful. At least not once they are over and you’ve gotten some sleep to get ready to debrief them. You can approach them with a stoic attitude of knowing that you won’t ever be able to prevent them. That you will keep having incidents. They will likely change in nature and shape as your systems and understanding of them changes. But you won’t ever get rid of them completely. Along that same topic Lorin Hochstein recently gave a great talk at SRECon23 about why we will all keep having incidents. And it’s well worth your time, so go watch it. But approaching incidents with a mindset of learning makes it an exciting rather than painful situation. Because you’ll know you’ll never run out of sources for learning. And once you’ve realised what a good source for learning incidents are, it’s maybe even time to take a good look whether shallow incident data like “mean time to detection” and “mean time to resolution” (or the maybe worst offender of all “mean time between failure”) are actually helping your team approach incidents as a learning opportunity or maybe are incentivising an approach that foregoes learning for a better look of those metrics.

But truth be told I really enjoyed reading this book. It’s roughly divided into topical areas like justice, medicine, cars, and art and discusses the status quo (as of 2020) as well as improvements and downsides to the rise of usage of algorithms in general but also machine learning. The author takes a very nuanced look at all these and it actually made me feel less frustrated about using computers in some areas than I had been through the public discussions of AI this year. Highly recommend the book.

Overall I liked the book. The big core message of friendship on a deep level also takes work is something that really resonates with me. There’s too often the message in media and entertainment that the perfect relationship (romantic or otherwise) is effortless, which does us all a disservice as we assume we don’t have to put the work in. But this book talks very clearly about the work being part of a friendship and I really liked that.

In the field of software engineering there is a common theme of fiddling with your tools and setup until they are juuuuust right, and only then can the work of writing code commence. Or the next level of productivity be unlocked. I don’t think this is something particularly special to software engineering. It’s probably just a different form of cleaning up your desk before writing, sorting all of the paperwork before doing your taxes, or re-organizing the kitchen before cooking. Basically procrastination. And the hope that what’s missing for productivity and “getting in the zone” is just this one weird trick to improve the setup.

I’ve done this a lot as well especially early in my career, where I’ve had convoluted setups that were very intricate and fine tuned to what I thought made me the most productive. Be it tmux, or IRC notifications, or of course my way of managing tasks. But the downside was always that the setups became more and more brittle with everything that was added. A plugin would break behavior when updated, a tool doesn’t work or isn’t available on macOS or Linux, an integration breaks through an API change. And over time the upkeep of the setup starts to become a bigger and bigger chore. And I noticed that at some point I stopped bothering with it. My setup slowly “deteriorated” to the minimal working state that kept me productive. I didn’t use my fancy integrations anymore. I could hardly remember why I installed some of the editor plugins. And I actually was as productive if not more. And so the state I arrived at is that I write all code in terminal vim in tmux now and that basically any Apple laptop with even a small screen will do (my forever favorite being the 11" MacBook Air and I absolutely can’t stand having more than one display) and I can be set up within about 20 minutes by basically just configuring:

• low key repeat delay and quick type rate
• caps lock remapped to control
• git clone https://github.com/mrtazz/dotfiles && make install

And not much more. My dotfiles configure vim, zsh, some git aliases, and install some useful tools (via homebrew/linuxbrew) that I could do without but sometime enjoy using like fzf and ripgrep. They also get installed on every codespace I create (at GitHub that is my main development environment) so when I ssh into it, the terminal is set up in the same way. On macOS they install some apps I use like iTerm2, 1Password, Alfred, and the Phoenix window manager with configs that I haven’t really changed in years. They are also mostly niceties that I can probably more or less do without (except 1Password). E.g. for months when codespaces was new and only available through VSCode I wrote code in a fullscreen VSCode terminal window running vim within it. And I was basically as productive as ever (maybe even a bit more given I had access to quick and disposable codespaces). Even with browsers I don’t really use any extensions or anything and change them without even thinking about it. I had some work specific configuration in Safari break a while ago. And instead of spending ages debugging it, I just switched to Firefox and moved on with my work.

And that is more or less the pretty barebones setup I use to write code and which I describe as a “pedestrian programmer” programmer style when I get asked about it.

Programming

But this notion doesn’t stop at the setup for me. It’s also how I write code. As I’ve mostly worked in infrastructure engineering over the last decade (even though I’ve switched to a product platform team at the beginning of 2022), I’ve had to jump between many different languages in the same day. It’s usually some mix of ruby, python, shell, golang, javascript, PHP, and various config formats. And I use that same setup for all of it. Furthermore, I also write very similar code in all of these languages. I’ve mostly come to utilize the common denominator of syntax and code structure to implement things regardless of language (they are all C-style languages anyways so they aren’t vastly different). And only really start using language specific constructs where needed. So my python classes look like my ruby classes, look like my PHP classes. If I can help it I don’t use concepts like python’s decorators, or ruby blocks (or even heavy use of higher order functions that most languages support at this point) in code I define and control. In my mind, trying to convey the solution that is implemented in code in as simple constructs as possible makes it a lot more accessible and friendly to get started with. I hope that even if someone comes from having written python their whole career, my ruby and PHP still is very accessible. If it becomes a performance or maintenance problem, it can still be changed to use more specialized constructs and concepts later on. And maybe even more easily because the original structure is fairly simple. But for as long as possible I try to keep code as “pedestrian” as possible so it’s easy to read, follow along, reason about, and change.

Similarly I don’t really use design patterns a lot when I write code. I remember the days in university when design patterns were all the rage. And books about them the most important text one could ever read about programming. I also remember trying really hard to force the Singleton pattern into every university programming project because it’s what professors wanted to see. Nowadays I will try and solve the problem with as simple of an architecture as I can. And then only change to a more intricate pattern if it serves understanding or maintenance. I most of the time don’t recall what a design pattern does when I hear the name. Not that I don’t understand what they do or how they are useful. Or that I don’t end up implementing them along the way. But I’ve found it to be more confusing than helpful to throw design pattern names as jargon around instead of writing code in a form that solves the problem and comment it along the way.

Take away

My main point in all of this is not that it’s bad to have a very intricate setup, or that you shouldn’t take joy in fiddling with it. But it’s important to recognise when the hunt for the mythical “zero-friction” state gets in the way of getting things done (and sometimes a little friction is not a bad thing). On the other hand it also doesn’t mean that the most elite and true way to write code is just using vim and the most barebones setup there is. Neither would I advocate to never use design patterns or take time to build a sophisticated architecture. The point I’m trying to bring across with the description of my approach is that it doesn’t take intricate setups and complex architectures to write production code that solves problems. Everyone is different and different approaches result in the same useful code. There is no rule of “the better the programmer, the more sophisticated the setup”. I’ve worked with programmers I respect immensely that wrote code in Notepad.exe or used the default 8 line vim config. I also enjoy reading about other’s setups regardless of how complicated they are because there might be something in there I’d love to try. And when I work in a team where we decide on design patterns and a more complicated architecture or code style as a trade off for some other problems, I’m happy to go along with it. It’s just not my first choice.

Because in the end a setup doesn’t make you productive and code isn’t “the better solution” because it’s complicated. You are productive with it because it fits the way you (and the people on your team when it comes to shared code) think and approach problems. There are no unreal programmers, if you write code with whatever tools you like and in whatever shape you prefer, you’re still a programmer as much as everyone else. Even if you are - like me - a proud pedestrian programmer.

I very much still remember the emotions those Saturday mornings when the Green Ranger showed up and first wrecked havoc to the Rangers, summoning his Zord with the dagger flute, before finally joining the good side and the Power Rangers to fight evil and eventually becoming the White Ranger. I seems a bit silly now writing it all out all these years later. But back then it blew my kid brain and was the main topic of many debates on the school yard. Thanks for everything!

#art #powerrangers #greenranger #jasondavidfrank #watercolor

#art #sketch #sketchbook #watercolor #ink #movies #kate #instaart

I would definitely watch a TMNT show about a younger Splinter pre-Turtles that just runs around a does badass ninja stuff. Got inspired for this by the recent absolutely amazing @bosslogic Metal Gear Solid cyborg ninja posts for the MGS anniversary. And thought which other popular fighters would fit such a pose.

#art #sketch #sketchbook #ink #watercolor #splinter #tmnt #turtles #instaart #comicart

#art #sketch #sketchbook #comicart #kang #avengers #marvel #watercolor #ink

#art #sketch #sketchbook #comicart #watercolor #conan #conanthebarbarian

The following assumptions are being made with this:

1. The ada0 disk was faulty and already replaced
2. ada1 is running in the zpool and working
3. Both ada0 and ada1 are the same size and have the same layout

Replace the disk itself

1. First up we restore the disk layout from ada1 onto ada0 and verify:

# gpart backup ada1 | gpart restore -F ada0
# gpart show ada0

2. Given this is a zroot pool to boot from, we need a bootcode

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

3. Now we just replace the (missing) disk in the zpool with its replacement:

# zpool replace zroot ada0p3 /dev/ada0p3
# zpool status zroot

At this point the zpool is being resilvered to make sure all data is on the new disk. Depending on the amount of data this can take a while. And while it’s running zpool status zroot shows something like:

# zpool status zroot
  pool: zroot
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue May 10 07:15:33 2022
        416G scanned at 144M/s, 382G issued at 133M/s, 417G total
        384G resilvered, 91.66% done, 00:04:28 to go
config:

        NAME              STATE     READ WRITE CKSUM
        zroot             DEGRADED     0     0     0
          mirror-0        DEGRADED     0     0     0
            replacing-0   DEGRADED     0     0     0
              ada0p3/old  REMOVED      0     0     0
              ada0p3      ONLINE       0     0     0  (resilvering)
            ada1p3        ONLINE       0     0     0

errors: No known data errors

Configure encrypted swap

The zroot setup from the baseinstall also comes with encrypted swap. So this also needs to be configured on the new disk:

1. Check setup and options from the existing swap partition

# geli list ada1p2.eli
Geom name: ada1p2.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 128
Crypto: accelerated software
Version: 7
Flags: ONETIME, W-DETACH, W-OPEN, AUTORESIZE
KeysAllocated: 1
KeysTotal: 1
Providers:
1. Name: ada1p2.eli
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 4096
   Mode: r1w1e0
Consumers:
1. Name: ada1p2
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1048576
   Mode: r1w1e1

2. Set up the new swap partition with the same options

# geli onetime -d -e AES-XTS -l 128 -s 4096 /dev/ada0p2

3. Turn new swap partition on

% swapon -a

Notice: When running swapinfo the old swap partition still shows up as a kinda ghost partition. I haven’t experienced any problems with that and it usually goes away on the next reboot.

# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/#C:0x86      2097152        0  2097152     0%
/dev/ada1p2.eli   2097152        0  2097152     0%
/dev/ada0p2.eli   2097152        0  2097152     0%
Total             6291456        0  6291456     0%

#art #ink #sketchbook #moonknight #marvel #comicart

This is such a fun game and it reminds me a ton of being a kid and hanging out with my best friend to play games on his mega drive

#art #watercolor #drawing #sketchbook #videogames #cherry #streetsofrage #switch

Part of this tendency towards simple setups also means that I don’t want to have the management of the few plugins I do use be cumbersome and involving understanding the intricacies of some new tool. And given that vim plugins for the most part also don’t have a ton of dependencies (at least not the ones I use), my needs for managing plugins can be reduced to a small list of requirements:

1. Have plugins I want defined in an authoritative list
2. Have a single command to update plugins to the newest version

Enter Make

So after having gone through various stages of managing my plugins like git submodules and a ruby script that was doing a lot more than it needed, I was wondering if I could just use one of my favorite tools make to do the job for me. After all one of the things it’s really good is, is making sure a defined set of files exists. Which is kinda all that vim plugin management is. And after thinking about this a little bit, I came up with the following things I would need to implement.

1. A way to define a list of plugins with name and URL
2. A way to download all plugins and put them in the right location
3. A way to make sure plugins that don’t exist anymore are removed

So let’s see how this can be done with make. First we need to define a list of plugins. This can be easily done by just storing them in variables, e.g.:

plugin_supertab  := https://github.com/ervandew/supertab/tarball/master
plugin_syntastic := https://github.com/scrooloose/syntastic/tarball/master
plugin_fzf       := https://github.com/junegunn/fzf/tarball/master
plugin_fzf.vim   := https://github.com/junegunn/fzf.vim/tarball/master

This was easy and if you’re wondering about the plugin_ prefix, you’ve noticed an important part of how to turn this into an easily usable list of plugin definitions. Within make there exists a meta variable called .VARIABLES which contains the list of all defined variables in the current Makefile. And since we have chosen a specific prefix for all plugin definitions, we can now get all defined plugins in a programmatic way by filtering all variables with the prefix and then stripping that prefix to get the name we decided to give that plugin via something like:

# this filters out all variables with a plugin_ prefix and regards them 
# as plugin definitions
ALL_DEFINED_PLUGINS := $(filter plugin_%, $(.VARIABLES))
# from the defined variables list we only extract the name
ALL_PLUGINS := $(subst plugin_,,$(ALL_DEFINED_PLUGINS))

Now we have a list of plugins (by their names) that we can use to construct a target. I want something easy and nice to type, so I went for a target called install-plugins:

.PHONY: install-plugins
install-plugins: $(patsubst %, $(PLUGINSDIR)/%, $(ALL_PLUGINS))

The target is PHONY because it doesn’t itself describe a file that gets generated, so it’s always “out of date”. But the interesting part of the target is its prerequisites on the right side. There is a path substitution there that defines file targets for all plugins in the plugin directory (I use vim’s built in plugin management, so I have the plugin directory defined as PLUGINSDIR := pack/plugins/start). This now means make knows that it needs to create all the files for the generated plugins paths (e.g. pack/plugins/start/syntastic) in order to fulfill the install-plugins target. However so far there is nothing telling it how to do that and in order to change that, we define a wildcard rule that matches the file path of those plugin definitions:

$(PLUGINSDIR)/%: $(PLUGINSDIR)
	@echo "Installing $@ from $(plugin_$*)"
	@install -d $@
	@curl -Lfs $(plugin_$*) | tar xz -C $@ --strip-components=1

This rule matches every directory under the plugin path via the wildcard character % . And because this is an implicit rule, make provides an automatic variable named $* that contains the stem of the match, which in our case is the plugin name. From there we can make sure the directory exists via install -d $@ ($@ is an automatic variable that matches the whole target), get the URL for the plugin by reconstructing the original variable we defined for the plugin from the stem ($(plugin_$*)) and then run curl and tar to unpack the plugin into the newly created directory. You might have noticed that the rule also has a prerequisite on $(PLUGINSDIR) on the right side. Which is easily satisfied via this rule:

$(PLUGINSDIR):
	install -d $@

So now we have all the pieces together to define plugins and install them. However we also want to make sure we don’t keep old plugins around that aren’t defined anymore, or files that were removed from plugins in newer versions. And this is done via deleting the plugins before installing new ones:

.PHONY: clean-plugins
clean-plugins:
	rm -rf ./$(PLUGINSDIR)/*

And then we introduce one more convenience task to update plugins by first removing all of them and then installing the ones we want:

.PHONY: update-plugins
update-plugins: clean-plugins install-plugins

And with that I can easily update all my vim plugins and commit the update to my dot file repository via the following command sequence:

make update-plugins
git add pack/
git commit -m "update vim plugins"

And the full example of the Makefile looks like this:

PLUGINSDIR := pack/plugins/start

# plugin definitions
plugin_supertab  := https://github.com/ervandew/supertab/tarball/master
plugin_syntastic := https://github.com/scrooloose/syntastic/tarball/master
plugin_fzf       := https://github.com/junegunn/fzf/tarball/master
plugin_fzf.vim   := https://github.com/junegunn/fzf.vim/tarball/master

# this filters out all variables with a plugin_ prefix and regards them as
# plugin definitions
ALL_DEFINED_PLUGINS := $(filter plugin_%, $(.VARIABLES))
# from the defined variables list we only extract the name
ALL_PLUGINS := $(subst plugin_,,$(ALL_DEFINED_PLUGINS))

# this will install all plugins via the wildcard matching target below
.PHONY: install-plugins
install-plugins: $(patsubst %, $(PLUGINSDIR)/%, $(ALL_PLUGINS))

.PHONY: clean-plugins
clean-plugins:
	rm -rf ./pack/plugins/start/*

.PHONY: update-plugins
update-plugins: clean-plugins install-plugins

$(PLUGINSDIR):
	install -d $@

$(PLUGINSDIR)/%: $(PLUGINSDIR)
	@echo "Installing $@ from $(plugin_$*)"
	@install -d $@
	@curl -Lfs $(plugin_$*) | tar xz -C $@ --strip-components=1

Final words

I’ve been using this way of managing my vim plugins for a while now. And I’m really liking it. It’s small, portable, and easy for me to reason about. I don’t think there will be a need for this to change anytime soon since it’s pretty feature complete for me. From a purely aesthetic perspective it’s not super nice that it removes all plugins every time it runs just to put most of the files back right away. But that doesn’t bother me because it’s fast, reliable, and simple.

If you’re curious about more of my setup, you can find my dotfiles on GitHub where I’ve done much more with make .

Bonus update 11-2023: automated plugin updates

Since writing this I’ve also incorporated an Action in my dotfiles repo that updates all my plugins once a week. So the next time I pull my dotfiles (or create a new codespace) I have an updated version of my vim plugins. The Action definition for this looks like this:

name: vim-plugin-update

on:
  workflow_dispatch:
  schedule:
    # run once a week on Wednesday
    - cron: '30 3 * * 3'

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: update vim plugins
        run: cd vim && make update-plugins

      - name: commit and push changes
        run: |
          git config user.name Github Actions
          git config user.email actions@noreply.github.com
          git add vim/pack
          git commit --allow-empty -m "update vim plugins"
          git push

I started 2020 with a very different expectation of the year to come. And I think it’s safe to say that I’m not alone in that one. But fast-forward 3 months and I found myself in the middle of a lockdown and a pandemic. Everyone is at home all the time, it’s working from home all the time, no coffee shops, no meeting friends, and that for the foreseeable future. The days are fairly unstructured and pretty chaotic as we try to make the best of this situation. In which I found myself really embracing journaling into various paper notebooks and it has made a tremendous difference for my mental health.

Some lines a day

Towards the end of 2019 I had bought a “Some lines a day” journal from Leuchtturm1917. And while at the beginning of the year, I wasn’t too sure what to put it in there, and lots of entries were just of the “had coffee today and got some stuff done” variety. I now really appreciate having this low-pressure prompt to get my thoughts out of my brain every morning.

The way the journal is designed is that it has 365 pages, one page per day. The pages are then divided into 5 parts so that the book can be used for 5 years. Once you’re through with the first year, you can see what you wrote in the year(s) before when getting to subsequent entries. It’s basically an analog version of the “On this day” feature you can find in every photo and journaling app these days.

The really nice part about this is that there isn’t really a ton of space for each day. So, it’s very low pressure. If you want, you can write a single sentence, and it feels like you did all there was to do. Or you can write fairly small and get a couple of good thoughts on paper before the space fills up. This means that for me, I can sit down in the morning and put whatever is on my mind onto paper, without a goal or a requirement. A literal brain dump.

Long Form Journal

A month into the first lockdown, I wanted to have a place to continue these thoughts from the “Some lines a day” brain dumps. So I ordered a simple soft cover, lined, A5 notebook from Leuchtturm1917 again. And once it arrived, I just poured my thoughts and anxieties into it. Every morning when I could see that some lines wouldn’t be enough, I continued whatever thoughts I had in this new journal. It felt a bit awkward at first because I wasn’t sure how to deal with which things go in there. It was different from only writing some lines because filtering felt like a built-in thing with so little space each day. But in the long form journal there was no limitation. It took me a while to get used to the journal and be ok with writing whatever crossed my mind with no judgement or self censoring. But it was (and still is) incredibly helpful to calm myself down in the morning and try to make sense of all the thoughts in my brain.

Again, I try not to force any requirements or expectations on me with this journal, either. If I thought there was more to write but I am done after 2 sentences, that’s fine and I’ll close the journal for the day. Or pick it up again in the afternoon to continue some thoughts or add new ones. I allow myself to have days (or weeks) where I don’t have the patience and calm to write in there. Or write multiple pages a day. The purpose of it is to make sure I don’t keep things rotating in my brain that make me distracted and anxious, but dump them onto paper. Even if I ran out of “some lines” that day.

Bullet journals

Which then brings the third notebook into focus, which is my Bullet Journal. I wrote about this before, and it has seen a couple of different implementations since writing the original post.

In the Bullet Journal, which is a dotted A5 Leuchtturm1917 notebook for me, I describe “the runway” of my responsibilities. Meaning that I plan out what the months, weeks, and days (often with explicit time blocks) should ideally be. Which of course never actually ends up being what reality looks like.

Make the plan. Execute the plan. Expect the plan to go off the rails. Throw away the plan. - Leonard Snart

But that’s ok. It’s the map, not the territory. And sometimes even just a compass. The important thing for me is that it’s a place where I put structure into the day and don’t just let it happen. I’ve learned about myself over the years that while I sometimes feel like it would be nice to just not have any plans or to-dos for a day, these unstructured days quickly turn into dissatisfaction for me. And I feel like I’ve not done anything. This is why I try to be intentional about my days. At least during the week. Weekends are still mostly unstructured in the sense that I don’t plan out the whole day but maybe only one or two things.

A very major thing the Bullet Journal is instrumental in for me is what I call “Inbox anxiety”. Depending on how much and what kind of things I have going on, I find myself in periods of dreading opening E-Mail (especially in the morning) because of the additional and new responsibilities I will find in there. And to some extent this also goes for physical mail. So the way I tackle that is that I open the Bullet Journal and the mail client on my phone. And for everything I see in there that needs follow-up - or any other action from my side - I write a to-do item into the journal. Including some potential context, or related to-dos, and so on. And I’ve found that this helps me feel grounded and not overwhelmed in times when there is more incoming responsibilities through my inboxes than I’d like to.

As the kind of complement to that, I also try to log even seemingly mundane things I do throughout the day. Especially on days that don’t go as planned, or are super stressful and I end up not feeling super great about them, it’s helpful to take a break and realize where all the time went that day. And what kind of things I actually did get done. And even if it still means I didn’t get much done, I have a record of the day and it doesn’t feel that lost. Especially throughout this pandemic, where every day has a tendency to feel the same, it’s useful to have something that shows me that this isn’t quite true. And all the days are indeed different, if only in subtle ways.

What I don’t really put into the Bullet Journal anymore are backlogs (or “someday” lists in GTD parlance). I realized that with 2 notebooks a year for bullet journaling, there are a bunch of things that often get migrated over and over again because I want to get to them eventually, but there is no time pressure. And it was tedious to rewrite them all the time. It just made me feel like the collections I had those items in weren’t useful and actually demotivating to me. So I moved those into digital tools to keep the Bullet journal the place for front-of-mind things.

Reading journal

Towards the end of 2019 I also bought an “Ex Libris” reading journal from Leuchtturm1917. This is the least used one of my regular journals. But while I don’t use it consistently, it’s been an absolutely fun addition. It mostly came about through me realizing that I don’t super enjoy Goodreads for more than tracking when I’m reading and finishing a book. I don’t like the review interface, and the app is generally pretty slow. And it never really intrigues me to go back and browse through the books I read and what I thought about them the way a paper journal does.

In closing

As difficult and challenging as 2020 was, and the ongoing pandemic continues to be, this process of having notebooks for many different ways to get things out of my head is immensely helpful to me. And as a continuation of leaning more on analog tools that I started in 2018 it keeps me from being completely petrified and procrastinating in the morning on many days. It also provides a way to really think about what thoughts and anxieties are rotating in my brain and have a conversation about it, even if I’m basically just talking to myself. And despite it being slightly inconvenient to carry around 3 or so notebooks sometimes, the benefits I get from having them around vastly outweighs that inconvenience for me. I even got myself a nice new pen to enjoy the process of sitting down and writing even more.

The author makes it very clear how much research has gone into the theory and how it explains many observations that can’t be explained by the classical view on emotions. It’s definitely a theory that’s a lot less intuitive than the classical view, but I’ve still found myself nodding along at some passages, while being very perplexed at others trying to understand concepts like simulation and prediction of the brain. And what it means for my day to day perceptions.

#drawing #painting #art #watercolor #sketch #sketchbook #dinosaur #triceratops #tonedpaper #ink #instaart

With the official availability of GitHub Actions last year I decided to move some of my CI jobs for my personal infrastructure over from my private Jenkins server to Actions. Both in an effort to clean up the setup a bit, but also to not have to maintain and rely on running Jenkins myself so much anymore.

My personal infrastructure runs out of a single monolithic repository that contains chef cookbooks, provisioning code, kubernetes resources, terraform code, and also dns configuration (which will serve as the example in this post) in a subdirectory. That is why a lot of the examples I post here are related to having multiple deployment targets in a single repository.

The used to be situation

For a couple of years now I’ve been managing my DNS configuration with [octodns]. It’s a really nice tool written by the GitHub engineering team to manage DNS zones across different providers via yaml files. My zones are configured in yaml files in the dns sub directory of my infrastructure repository. There’s a small Makefile that has tasks for verifying and deploying configuration. And it usually got deployed via a push to the default branch of the repo and a top level Jenkinsfile that mapped directory paths to Jenkins jobs like this:

stage('determine subjob to build') {
    try {
      sh "printenv"
      echo "Got params: ${params}"
      foundJob = false
      changedFiles = sh(script: "git diff --name-only ${params.prevSHA} HEAD", returnStdout: true).trim()
      if (changedFiles =~ /^jobs/){
        build job: 'create-jobs', wait: false
        foundJob = true
      }
      if (changedFiles =~ /^dns/){
        build job: 'dns', wait: false
        foundJob = true
      }
      if (!foundJob) {
        echo "No subjobs to build for " + changedFiles
      }
      sh "/usr/local/bin/ci-notify --job=${env.JOB_NAME} --build=${env.BUILD_NUMBER} --success"
    }
    catch (err) {
      sh "/usr/local/bin/ci-notify --job=${env.JOB_NAME} --build=${env.BUILD_NUMBER} --failure"
      throw err
    }
  } 

In case the job that was run on push failed or I just wanted to rerun the deployment, I also have a helpful Slack bot to help out with that:

And this worked really well for a couple of years. I don’t have a ton of changes usually in my personal DNS setup, so whenever a change was needed, this setup was more than enough automation to keep me happy. But at some point I got annoyed by the fact that basically all automation for my infrastructure was dependent on Jenkins being up (and I had to make sure it was).

Hello Actions

Fortunately around that same time I got really annoyed running Jenkins, GitHub Actions went GA with the CI offering which prompted me to look more into how I could use it to maybe replace my Jenkins setup.

As a first step here I wanted to change as few things as possible. So Actions would literally just replace Jenkins, running the deployment logic whenever I pushed to the default branch of my infrastructure repository.

In Actions this was done by restricting the job to changes in the dns subdirectory or changes to the workflow file:

on:
  push:
    paths:
      - 'dns/**'
      - '.github/workflows/dns.yml'

The job itself would then run the same sequence of make check to do a dry run of the changes, and then make update to deploy the changes.

Pull Request CI integration

This was already great and a perfect replacement of the setup I had before. But it wasn’t using the power of Actions to their full extent. The most obvious one being that this wouldn’t really work with pull requests. Sure the job would run, but every change pushed to a PR would automatically be deployed. Which also can be a nice workflow, but wasn’t what I wanted. I wanted to have more control over deploys, essentially a human 👍🏻 that the changes are actually good to go. And as the signal for this I decided on the merge button. So once I was happy with the changes and wanted to see it deployed, I just had to merge the PR and make sure the automation keeps deploying on the default branch.

In order to do that, all I had to do was add if: github.ref == 'refs/heads/master' to the step that was running the make update deployment. And voila, the make check dry run is run on any PR now with nice and proper GitHub checks integration and any merge to master triggers a deploy now and makes sure the changes go out.

Branch based deploys

I ran with this setup for a while. And again it worked perfectly fine. However I was also running into situations where the actual deploy failed even though the PR check was fine. Because there is no place like production. And all the tests will never be a full replacement to catch all the things before hitting production. That’s just how it is. But it meant that I would only find out after the merge that something was off. And then I had to open another PR to fix the problems which usually was just a small one line change. And I got annoyed by this. Plus Pull Requests have a really nice integration with the deployments API which I was missing out on.

So I wanted to have all these nice things as well. The first step to get there was to find another trigger for deployments that signals the automation that this code is fine to deploy. Pull Requests don’t have a ton of ways to interact with them. It basically comes down to comments or labels. And because there is a way to trigger Actions on labels, that’s what I went with.

In Actions configuration this looks like this. First I had to make sure the job runs when the PR gets labeled (the other 2 opened and synchronized make sure the automation is run on any changes pushed to the PR):

on:
  pull_request:
    paths:
      - 'dns/**'
      - '.github/workflows/dns.yml'
    types: [opened, synchronize, labeled]

Then I decided on a label name - in this case deploy requested/dns - and made sure the deployment logic only ran when the PR actually was labeled with that. I did this by making deploy a separate job in the Actions definition and have it be guarded by if: contains(github.event.pull_request.labels.*.name, 'deploy requested/dns') similar to how the guard on default branch worked before.

Now to have that nice PR timeline integration with the deployments API I wrote a small ruby script which is configured via environment variables in the job.

env:
      DEPLOYMENT_REPO: ${{ github.repository }}
      DEPLOYMENT_ENVIRONMENT: dns
      DEPLOYMENT_DESCRIPTION: dns updated via octodns
      DEPLOYMENT_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      DEPLOYMENT_SHA: ${{ github.event.pull_request.head.ref }}

In order to record deployments and their status changes and run the deployment creation step right before the deployment logic like so:

- name: start deployment
  run: ruby bin/gh-deployment.rb create

And then the at the end of the job there are these two steps that are run on success and failure of the job respectively:

    - name: record deployment failure
      run: ruby bin/gh-deployment.rb failure
      if: failure()
    - name: record deployment success
      run: ruby bin/gh-deployment.rb success
      if: success()

Which makes sure the deployment status is properly reflected in the Pull Request timeline:

Wrapping up

This is now a workflow I really enjoy. It’s extremely similar to how we work at GitHub, which is nice because I don’t have to rethink how things are done when I change things in my personal infrastructure. Plus I get all of the testing and feedback on my PR and can act there with changes before merging the code. I also can work on more than one PR at a time, and even have a PR for my chef changes and one for my dns changes and have them be deployed and tested and incorporate feedback before merging. Or do it in the same PR because the pre deployment checks will run as soon as there are changes in the respective subdirectory. And I can add a deploy requested/dns and a deploy requested/chef label to have the automation for both deployments run.

After finishing the book I took some time to think about how a Zettelkasten style note taking setup would best fit into my day to day. And how I would want to interact with such a setup. And I’m still in the process of moving my old notes into the system and figuring out how it works out for me. So while I can’t say anything yet about how effective the things are that I learned through the book, I found it intriguing enough to make me adapt the method and try it out.

But still decided to color it a couple of days later. Watercolor and watercolor pencils. And a little bit of ink for some small details. That counts for #inktober, right? 😆 . . . #art #sketch #sketchbook #drawing #painting #watercolor #watercolorpainting #watercolorpencils #ink #markers #lebronjames #basketball #🏀 #👑 #tonedpaper #fabercastell #strathmorepaper #royaltalens #instaart

The book is structured in a way where only the first two chapters are really about outlining itself. The advantages of writing an outline versus writing “at the seat of your pants” as well as different approaches to outlines (e.g. mind maps, longform writing, bullet points) are discussed.

The following chapters then are a whirlwind tour through planning a novel and how to apply outlining to areas like plot planning, character development, or creating a backstory. At the end of every chapter there is an interview with a writer about the value and their approach in outlining.

I really enjoyed reading the book. I learned a lot about structuring my writing and what kind of things go into planning a novel. It’s very accessibly written and the chapters all have a good length so it never feels like they are dragged out or missing content. I could have done without the writer interviews as they didn’t really add a lot to the book but had a couple of interesting anecdotes. I don’t know if there would be anything new for a seasoned author in the book, but I can definitely recommend it as a quick intro into planning a novel.

After having spent a lot of time on learning how to draw in the last 2 years, there weren’t a lot of things that were really new to me in this book. However it’s the book I wish I had 2 years ago when I got started. It does a great job laying down the fundamentals for getting started with sketching. And it definitely inspired me to do more quick sketching and drawing from life.

#art #drawing #sketch #sketching #pencil #ink #spiderman #marvel #comicart #instaart #learntodraw #study

I have to admit that occasionally while reading the book I was fascinated by the things I was learning while at the same time not really knowing what it has to do with dark matter. Nevertheless I extremely enjoyed this book and the things I learned reading it.

#art #sketch #sketchbook #watercolor #painting #drawing #tonedpaper #shaft #richardroundtree #samuelljackson #tv #movie #instaart

#art #painting #sketch #watercolor #gouache #woodworking #lumberjack #axe #instaart

#art #sketch #sketchbook #painting #drawing #sketching #watercolor #ink #tonedpaper #brushpen #greengoblin #normanosborn #marvel #spiderman #comicart #instaart

#art #sketch #sketchbook #painting #watercolor #comicart #drdoom #victorvandoom #esadribic #ivesvorcina #secretwars #marvel #thanos #avengers #instaart

#art #sketch #sketchbook #ink #watercolor #drawing #painting #clarkkent #superman #christopherreeve #dccomics #comicart #movieart #movies #instaart #moleskine #fabercastell #royaltalens

#art #sketch #sketchbook #fountainpen #ink #watercolorpencils #tonedpaper #strathmorepaper #abby #lastofus2 #videogames #videogameart #instaart #drawing #sketching

#art #scrapbook #sketchbook #sketch #drawing #pen #ink #fountainpen #batman #michaelkeaton #dccomics #instaart

#art #sketchbook #sketch #drawing #painting #tonedpaper #watercolor #ink #brushpen #magik #illyanarasputin #xmen #marvel #comicart #instaart

#art #sketchbook #sketching #anatomy #lamy #lamyjoy #fountainpen #instaart

#art #sketch #sketchbook #drawing #painting #ink #watercolor #watercolorpencils #spiderman #spideysense #comicart #marvel #avengers #instaart #videogames

#art #sketch #drawing #painting #ink #watercolor #brushpen #harleyquinn #dccomics #suicidesquad #movieart #comicart #instaart #stayhome

#art #sketchbook #sketch #drawing #painting #watercolor #ink #stratos #motu #mastersoftheuniverse #heman #instaart #stayhome

#art #sketch #sketchbook #painting #drawing #ink #watercolor #thewasp #hopevandyne #marvel #avengers #instaart #stayhome

I thought this was gonna be a quick fun sketch. And it turned out to be a pain for me to get that face to a somewhat not creepy state. Still fun. That movie was one of my favorites as a kid.

#art #sketch #sketchbook #ink #watercolor #mask #themask #jimcarrey #movies #instaart #stayhome #moleskine #drawing #painting

#art #sketch #painting #drawing #finalfantasy #tifa #finalheaven #videogames #instaart #ink #watercolor #stayhome

#art #sketch #sketchbook #ink #watercolor #brushpen #ghostrider #johnnyblaze #marvel #comicart #instaart #stayhome #painting #drawing

Left the quarantine stronghold today for the weekly-ish grocery shopping tour.

#art #sketch #sketchbook #painting #drawing #watercolor #ink #vangoghwatercolor #fabercastell #moleskine #instaart #stayhome

#art #sketch #sketchbook #ink #watercolor #watercolorpencils #hercules #hades #disney #cartoons #instaart #stayhome #drawing #painting

#art #sketch #sketchbook #drawing #painting #ink #watercolor #darkwingduck #megavolt #elmosputterspark #instaart #stayhome

#art #sketch #painting #drawing #ink #watercolor #saturdaymorningcartoons #countduckula #instaart #stayhome

#art #sketch #sketchcard #ink #watercolor #darkwingduck #letsgetdangerous #zwoeinsrisiko #instaart #stayhome #drawnearlyeveryday

The overall book is structured in chapters about the cognitive revolution, the agricultural revolution, the unification of humankind, and the scientific revolution. I’ve never thought about historical periods in that way and it was interesting to see them presented in that way. The book covers the usual things like evolution of Homo Sapiens with a bigger brain than other species, the early periods as hunter gatherers, the emergence of agricultural and the connected settling down.

The book also discusses cultural things like a common belief (e.g. religion or capitalism) being a necessary establishment to allow for communication (and collaboration to some extent) across larger territories than just family or tribe boundaries. For any evolutionary step the trade-offs in those changes are also discussed. Especially the cruel(-er) aspects of human dominance on the planet.

I learned a lot reading this book and it definitely made me think about history and human evolution from other perspectives than usual.

Wikipedia’s triumph seems to defy the laws of behavioral physics

One of the more prominent examples of intrinsic motivation in the book is contribution to open source. At that part it shows that the book was written over 10 years ago. Because the discussion of open source is solely focused on the positive effects of honing skills, furthering one’s career through the work, and the intrinsic “feel good” motivation of giving back and contributing. However over the last 10 years the problem of burn out with open source maintainers has become a major topic that is completely absent here. And there is also no real acknowledgement of maintainers vs contributors and the difference in demand, work, and rewards. But I don’t think that detracts from the points the book is making, merely something I immediately jumped to when reading these packages as some missing nuance in the argument.

… rewards can perform a weird sort of behavioral alchemy: They can transform an interesting task into a drudge. They can turn play into work

Large parts of the book then deal with the detrimental effects of the “carrot and stick” approach to management. It was interesting to read about flow state, rewards, and how - depending on how they are used - they can be actually detrimental to motivation. This part contained an interesting short lesson in management history and that the idea of creating space for autonomy for workers has come up a couple of times in the past but never got a lot of traction. And that the new way of granting autonomy and letting workers have more urgency over their work is improving motivation. However there are also a couple of dangerous ideas the author arrives at. That can easily be read as management being a problem, rather than the fact that it is a craft that needs to honed just as any other work

Perhaps management is one of the forces that’s switching our default setting and producing that state [of passive inertia]

Finally there is a discussion about mastery and putting effort into something that I found very interesting based on Carol Dweck’s - a psychology professor at Stanford - research. The discussion of mastery being a mindset rather than a set goal resonated a lot with me. Especially the idea that intelligence is not a set entity within people but a resource that can be increased via learning.

That is the nature of mastery: Mastery is a mindset

And finally the idea that mastery is an infinite pursuit. That even if you spend a lifetime doing something, you can never truly master something. There will always be more to learn, more to practice, more to … master.

Overall I really enjoyed the book. Even though there are definitely parts that haven’t aged quite so well over the last 10 years and with all the things we’ve learned from tech startups left and right misunderstanding the importance of structure, management, and Human Resources. There were many parts that I really enjoyed reading and learned a bunch of things from.

So I read the whole thing with not a lot of expectations. And overall I didn’t hate it. It was slightly obnoxious at times, and Dostoyevsky seems to definitely have a thing for drama. Plus it has all the problems you can expect from a man during the 19th century when it comes to respecting women. The ending made up for a lot though.

I thoroughly enjoyed reading „A Field Guide to Getting Lost“ even though (or maybe because of?) I often found myself getting lost myself in an essay and wondering what’s the point she’s trying to be bring across. It was also at times deeply emotional to read for me as I was reminded of personal loss (or the fear of it) with many of her essays.

However once I started reading the book I immediately got sucked in and really enjoyed it. The book is very well researched and there are a lot of references to other works and research that are interesting to follow up on. It also contains a good mix of background information and actionable things to do for minimizing digital factors in once life. That makes it really easy (and somewhat also creates the desire to try) to give the exercises for minimization a spin.

My fear of already being sold on the concept and thus the book feeling like preaching to me as the choir were completely unfounded. If anything I got a confirmation and reaffirmation to my own practices of minimizing digital influences on my life.

I especially enjoyed the discussions of axioms as a kind of personal value system. And thus the root of any logical reasoning that you apply to arguments. This was the part for me that really tied the theoretical approach of logic and the emotional and moral foundations of one’s own beliefs together.

Overall however I wanted this book to be better. While I really enjoyed the logic part, there was one thing that made it really hard for me to get into a flow while reading. The author kept jumping between theory and examples way too frequently for my taste. She also rarely used lighter “introductory” examples. But most of the time jumped straight into heavy examples around racism, sexism, and other topics like that. Which made it much harder for me to get an understanding of the logical foundations as I was trying to keep the theoretical ideas in my head while also dealing with the emotional reaction to such heavy examples. I don’t think that those didn’t have a part in the book at all, but rather for me it would have worked better to reserve these heavy examples for the end. At which point I had familiarized myself with the concepts that got introduced, had some simple examples to apply them, and could focus on a more trying example as a completion of the topic.

First let’s get this part out of the way: here are the scenarios where I’ve seen having a technical manager being a big advantage:

1. As a very junior engineer without any mentorship setups in place
2. During onboarding where I didn’t have a good lay of the land or relationships to lean on to ask about system overviews yet

However these situations are edge cases to me which don’t pose actual problems. A good non-technical manager in this scenario will just make an introduction to an engineer who knows the answers and is willing to mentor a junior engineer or new hire. If there isn’t such an engineer, then there are much bigger problems in an organization than technical vs non-technical managers.

And this actually ties into the main point why a technical manager is a silly requirement in my mind: As an engineer you’re already surrounded by a ton of people with deep technical knowledge (i.e. all the other engineers). It’s almost impossible to have a problem or question and there not being someone around to help.

However it’s less of a likelihood to have someone around to help you with non-technical problems. How to structure work into plannable pieces, how to give feedback, how to put your work into the larger context, how to improve the team, etc, etc. And these are much less special and unique topics to technology where you need a strong technology background than most people think. In a healthy organization the senior engineers will also be able to help with this to some extent. But they will also need to have learned it from someone. And need to have an avenue to improve their non-technical skills.

And as you become more and more senior, being able to excel in these non-technical areas becomes more and more crucial to be an effective engineer. With seniority comes problem solving on a larger scale that needs more breadth in impact and much less depth. Focusing on a specialist skillset of just technology will more and more limit you in being able to do this important kind of work. And at some point the only way to improve is working on your non-technical skills, for which you need a good non-technical manager.

This doesn’t mean that technical managers can’t teach you these kind of things or are necessarily bad at it. But in my experience when you have technical managers who are good at it that’s the case despite their technical background and not because of it. It’s way too common that the technical background actually gets in the way. That they consider themselves more as used-to-be-engineers than managers. That they are tying every conversation back to a technical topic and focus on that instead of the reports actual growth. And at worst they try to still write code themselves and debate technical decision making instead of taking a step back and let others grow on those challenges. And don’t get me wrong, I know many great technical managers that don’t do these kind of things and are great at their jobs. But what I value about them is exactly that. They are managers first.

Requiring a strong technical skillset of managers is also a huge waste of resources. Given that a manager essentially ends up with two jobs that way, they will struggle to be able to do a good job in at least one of them (more likely in both). They are constantly having to shift focus from the management side of things to try and stay up to date on technical topics that they will never really have hands on experience on anyways. Taking time and energy away from their actual job: managing. And nobody really enjoys having a conversation with a manager who hasn’t written production code in years yet still engages in technical problem solving and decision making conversations late at night because they think they have to also keep their technical chops up to date. While at the same time clouding their view of the higher level context because they get caught up in technology details.

And it’s not only about the manager. It already starts at hiring. Requiring technical aptitude from managers means that you waste precious interview time on technical questions that are even less related to what their work will be than any of the ridiculed inverting a binary tree whiteboard exercises every engineer groans about. Instead you could use all that time to talk extensively about management topics. And how they will help grow teams and organizations.

At the end of the day management is a lot more removed from the actual day to day practice of writing code than most people think. And the problems to solve and challenges to tackle for managers at technology companies are much less unique to technology than we all like to think. A good manager (regardless of technical background) is able to recognize that and work with engineers to make them better by complementing the skills they already learn every day while doing their main job.

The whole book is absolutely fun to read and I can highly recommend it for learning and inspiration.

#art #sketch #sketchbook #dailysketch #inktober2019 #arttrober2019 #inktober #batwoman #dccomics #illosketchbook #copicmarkers #brushpen #ink #instaart

There are definitely good things in there about balance, not seeing yoga practice as a competition, listening to your body, taking time, etc. But as a fairly scientific person it’s hard to read over the parts where the last 100 years of modern medicine are ignored.

One of the things that definitely did strike me as interesting were Drucker’s points about taking notes and writing things down in general:

If I don’t write it down immediately, I forget it right away. If I put it into a sketch book, I never forget it and I never have to look it up again

I’ve had this experience so frequently, especially as a kid in school. That when I wrote something down (e.g. homework for the day) I rarely needed to look it up again but remembered what I had to do. Because I would also often remember it either way, I sadly drew the wrong conclusion there (namely that writing things down isn’t worth the hassle). Which stifled my note taking skills and a lot of my learning later on. It wasn’t until a couple of years ago that I realized that it was precisely that process of writing things down that made me memorize my homework.

A good part of the book also is about figuring out which version of yourself you want to be. Which I think is a really important aspect of maturing as a person and figuring out why you’re even trying to get better:

That is the mirror test. Ethics requires that you ask yourself, what kind of person do I want to see in the mirror in the morning?

As a manager, one of your primary jobs is to foster a foundation of trust on your team

In the book Lara gives invaluable insights about the human side of management. About the fact that as a manager you are tasked with making a group of humans work together. And that only works if you understand what humans value, how they tick, and if you’ve put the work in to build and foster a foundation of trust on your team.

This common thread of understanding humans and using that to be effective at work (and to be honest also outside) guides you through the whole book . Be it mentoring, coaching, sponsoring, providing (and receiving) feedback, or communication. Every chapter is written with humans in mind. And that is what makes this such an outstanding book. Throughout the chapters you will learn how to meet your team, grow your team, set expectations, communicate effectively, and build resilience. Every chapter is to the point, clearly written, and extremely actionable.

This book is a must have regardless whether you are a manager, want to become a manager, reporting to a manager, or just in general work with other humans. Lara Hogan puts down so much useful insights and wonderfully helpful guiding questions and suggestions that I don’t think there is anyone who wouldn’t benefit from reading it. Plus it’s about 100 pages long. You can literally start reading this book on Friday and will be better equipped to do your job on Monday.

Trying something new(-ish)

I needed to change things up. And the solution for this couldn’t be another app. It needed to be different. And it turns out this is a pretty normal thing for humans. We link memories (which things to remember to do basically are) to locations via the hippocampus.

This is the reason it’s important to have a designated place for each of our belongings - the hippocampus does the remembering for us if we associate an object with a particular spatial location.

— Daniel Levitin, The Organized Mind (p. 91)

I’ve been carrying a Moleskine notebook with me since early 2008. Early on I had already used it as a todo organizer before switching to Things and eventually Omnifocus. I’ve used it on and off for random things (rarely enough for it to last 10 years). And it’s been the testing ground every time I wanted to get back to taking more analog notes. I’ve also backed the Spark Notebook on Kickstarter and used that with a lot of success for a while. So when I was looking to change things up from my digital routine I remembered having read about the Bullet Journal method and decided to give it a try.

Getting started with a Bullet Journal

For getting set up I started reading the website first and watched the canonical intro video linked from there. But being used to this elaborate GTD setup I wasn’t convinced that a minimalist way worked for me. I read a lot of fairly popular posts on getting started with bullet journaling from websites like this and this one and a ton of other blog posts to understand how this is being used by different people. And then I bought a new notebook and some pens and started with my own.

And I absolutely overdid it. I used a ton of color and differently sized pens to denote headlines, priorities, etc. I had 2 different systems (dot stickers and sticky labels) do denote important pages. And I added a ton of modules and collections like trackers for workout, meditation, water intake, and reading time. I had very elaborate monthly and weekly spreads, trying to recreate the organizational cockpit that I always wanted Omnifocus to be. I put way too many things to do in, areas of focus with color coded headings, and complicated time blocking details. My daily spread had a daily plan bar that included all my meetings and time blocks for the day. My weekly spreads were as complicated and stuffed, at some point even including which days to take out the trash. Bringing me to up to an hour of just setting up my page to get started for the day. All to combat the feeling of not getting things done and falling of the wagon again.

Of course once the initial excitement had worn off I fell back into seeing maintaining this complicated thing as a chore and neglected it. And I ran into the same problem I had with Omnifocus of having a layout that was very tuned to my workdays. On the weekend or when I was taking vacation, it wasn’t useful. And I hardly interacted with the journal. Leaving me again with the guilt of “having fallen off”. One important difference though was that on those weekend days and during time off where I couldn’t bother to get into my complicated setups, when I did use the journal it resembled a lot more the original idea of the Bullet Journal. And instead of giving up and changing back to Omnifocus, I stuck with it.

What my Bullet Journal actually looks like now

One of those vacations was at the end of last year. During that time I reduced my usage of the journal to basically only a weekly spread. Mostly because there wasn’t much to keep track of. And I realized it still worked for me. I still put all my todos and appointments in there. And it adapted to the difference in usage wonderfully. I was also about to start my third Bullet Journal, having journaled more than twice as much as the previous 10 years combined. I bought the official Bullet Journal book to learn more about the ideas and philosophies behind the original approach given I had more belief it could work for me. And aside from all the other interesting things in the book, the thing that really changed the way I thought about it was that it’s still supposed to be more like a journal than a GTD system.

After finishing the book I slimmed down my Bullet Journal to the useful bare essentials. I kept the original monthly layout I had already been using but stripped down the monthly task list to a literal list instead of different areas with colored headlines. The 2 page weekly spread turned into a single page of tasks I want to get done over the course of the week. And the daily spread is no longer a plan bar for a meticulously planned out day. It now just starts with the date headline and serves 90% as a journal for recording the day rather than a pre-planned skeleton of how I think the day will go. Because one of the big reasons why I was often abandoning the journal was because they day almost never turned out as planned. Making me feel like the journal was less useful.

I kept marking the future log (which for me is the combined calendex/alistair method), monthly and weekly spreads, as well as important collections with dot stickers. That way I can quickly find e.g. the page with the last monthly spread if I want to look something up.

And another big insight from the book was that I’m now leaning on migrations a lot more than I used to even though I don’t do daily migrations anymore. I scan the last pages for the current week in the morning for things that still need to get done and if they are a priority I move them to the current day. However that rarely happens and it mostly a measure for me to not forget about priorities. I do weekly and monthly migrations where I thoroughly go through the pages and migrate items, add additional context, put things into the future log (or the topic specific collections for things like personal, work, apartment, etc that serve as a sort of backlog). But otherwise I really just start a new headline every morning and start journaling.

In Closing

Switching to paper for organizing my todos, thoughts, events, and planning things has been absolutely wonderful for my stress levels and mental health. Especially after trimming the process down to the minimum. I’m no longer stressing about the perfect setup, but use the journal in the way that makes the most sense for me in the moment. I still use a reminders list on my phone for things on the go or when I don’t have the journal with me to migrate over later. I’m much more focused and calm about organizing things when I’m able to close my laptop and just open the journal, it feels much less noisy. Using pen and paper so much every day also lead me to occasionally doodle on pages and discover my interest in drawing and art which has been another huge source of joy for me.

#art #sketch #sketchbook #dailysketch #dailydrawing #2019draw365 #brushpen #ink #pigmamicron #bengrimm #thething #thing #fantasticfour #marvel #comicart #instaart

#art #sketch #sketchbook #dailysketch #dailydrawing #2019draw365 #brushpen #ink #reedrichards #mrfantastic #fantasticfour #stretch #marvel #comicart #instaart

#art #sketch #sketchbook #dailysketch #dailydrawing #pigmamicron #brushpen #mgsv #metalgearsolid #bigboss #snake #videogames #instaart

#art #ink #drawing #pigmamicron #copicmarkers #jeangrey #xmen #marvel #comicart #instaart

I’ve come to view this from a perspective of formulating a hypothesis and establishing factors of confidence to confirm or refute this hypothesis. This sounds very abstract and theoretical at first. But bear with me for a moment here. The basis for all delivery is a change (or patch, diff, commit, change set, whatever you wanna call it). This change is meant to improve something. Add a feature (or establish the base for one), fix a bug, improve performance, increase visibility, or just clean up some technical debt. This means you’re going to production to make the world better. However given the complex nature of the systems we deploy software to, you won’t actually know if your change is a net positive until it’s running in production. And even then you often only know a couple of hours or even days later. So all you have when you’re in front of your editor writing some code is an idea about what will make the word better. A hypothesis.

The job of a delivery pipeline now is to help you get confidence. Confidence that your hypothesis holds. Or confidence when you have to refute it. However all of the complex interactions of systems means you don’t get to have that single unified proof that your code is what you want it to be. Your ability to make a decision about your change is based on many small factors of confidence. And the delivery pipeline should give you tools along the way to acquire those factors of confidence in reasonable time and effort. It usually starts with a very quick feedback loop and something akin to a unit test. You can write them quickly and they can be verified quickly (individually that is. Running large numbers of unit tests on CI is still a not so easy problem). You then usually move on to test that are more expensive with a longer feedback loop. Like an integration test. Maybe a QA environment. A staging environment. Smoker tests in production. Canary deploys. And so on. All of those things (and this is hardly an exhaustive list) are intended to give you confidence in something. That your logic is correct, that your code works well with other API endpoints, that it interacts with other code on the site in a way that doesn’t break the whole thing, that it doesn’t put too much load on downstream systems, etc, etc. And ideally all these things in place will give you a nice set of guardrails that make deploying to production an enjoyable experience.

However given these tools are merely a snapshot of your understanding of the system at the time and what confidence is needed to make a change to it, the delivery pipeline needs to be constantly maintained and re-evaluated. Maybe system growth now means that the trade off of running a large array of unit tests and the time it takes, doesn’t pay off in the confidence it provides. Or maybe it does and this means you need to think of something to make running unit tests faster. Maybe a new additional service means you now need to add a set of smoker tests. Whatever it is, the most important thing is that you know why any of these tools to assert confidence are in place. Who are they for and what are they telling you? The last couple of years have seen the rise of a huge number of fantastic delivery systems. Often highly opinionated or infinitely configurable. Sometimes both. It’s easy to just take one of them and cargo cult what they bring with it. And if you don’t already have an established system, this is a fine approach that will certainly make you end up with a better setup than you had before. However I encourage you to look closely what your delivery pipeline is made up of. And what kind of things it gives you confidence in. Do you often see failures after deploys because of surprises in your logic? Maybe you’re missing some unit tests. Are you spending tons of time on unit tests that essentially only re-test the framework code of the tool you’re using? Maybe you don’t need those tests and can free up a lot of engineering time. Whatever it is, your delivery pipeline needs to give you confidence in changes in your stack. You know best what kind of things need to go in there. And spending some time to think about that will give you a lot of insight and pay off when it comes to improving your delivery pipeline. And it’s also tons of fun!

PS: I’ve had many discussions about those things with many people over the years. And they all helped me figure out how I think about delivery and make sense of my rambling thoughts. So if you’ve ever chatted with me about deployment and/or delivery, I’m extremely grateful you took the time and I really enjoyed our chat.

#art #sketch #sketchbook #dailysketch #drawdaily #2019draw365 #ink #pigmamicron #pentelbrushpen #copic #copicmarkers #henrymccoy #beast #xmen #marvel #comicart

#art #sketch #sketchbook #dailysketch #drawdaily #2019draw365 #pentelbrushpen #pigmamicron #ink #comicbookart #dccomics #constantine #johnconstantine #comicart

I’ve gone through the process of thinking about vision for a team before. At Etsy we use a structure called “VMSO - Vision, Mission, Strategy, Objectives”, to organize and structure teams and departments in what their purpose is within the company and what they contribute to the business. It draws a lot of inspiration from the ideas in this blog post by LinkedIn CEO Jeff Weiner called “From Vision to Values: The Importance of Defining Your Core”. The rough overview is that vision is the 30 000 foot view, the high level idea on the horizon that (almost) never changes. The world we want to see exist. The mission is derived from it and describes what the team does to get towards the vision. And then it gets more concrete with strategies how to get there and concrete objectives we want to fulfill. It’s not an easy process and definitely takes a whole of brainstorming, suggestions, throwing away suggestions, refining and merging ideas, and consensus building to get there.

Before this season of change, on my old team, when we were tasked with creating a VMSO for ourselves we always got hung up on the vision. It was always that high level thing that never quite matched the work we were doing. We would meet once or twice and always seemed to end at the same dead ends: “Our work is too multifaceted to be captured by a single statement”, “It’s hard to explain what we do”, “We do anything that needs doing”, “We keep things running”. If you’re working on a general purpose infrastructure team, this might sound familiar to you. It seemed like it was just impossible to come up with a single vision for the team, so we always left it at a half baked, cheesy feeling idea. And of course at that point we didn’t manage to derive a good mission from the vision either. Not to speak of strategy or objectives. I didn’t feel too bad about that at the time. As we had a fairly broad vision statement, it let us basically take on anything we wanted. And to be honest, I loved working on that team. Although we were always working on separate things, we were a bunch of engineers with the same mindset and approach to work. We had a great team dynamic and our team meetings were a ton of fun. I couldn’t imagine working on a different team.

And in the middle of this work the first part of the reorg happened and our team got dissolved. I was really upset. While I was fully onboard with the reasoning and goals of the reorg, I couldn’t understand why our team got ended and most of our roadmap dropped. It felt like our work had gone completely unvalued. But then I had a long 1-on-1 with my then Engineering Director Jason Wong. We talked about all of it, he gave me a ton more context. And he made me understand how a team that does “a little bit of everything” is really hard to fit in organizationally. He asked me flat out what the purpose and vision of the team was in the org. Where was the team going? What would it look like in 2 years? And I couldn’t give him a straight, simple answer. I was a very senior engineer on the team and I had no answer. This was the moment where I managed to connect (some of) the dots. And tie together our lack of a comprehensive vision to the downsides of our operating model. We ended up supporting way more things than we could, leading to long periods of maintenance work and almost none of the iterative improvements we planned for at the beginning of the year. We had no way of saying no to work because we didn’t have a good reason to reject the work. We had weeks where our work summary would basically just be “clean up”. Which I love doing and is valuable work, but not if it takes up 90% of someone’s time. We agreed on a vision that was defined by the work we were already doing and not by what we wanted the work to be.

And at the time I failed to see the big downside of this: it let us take on anything we wanted. While this sounds like fun at first, it makes a lot of things really hard. We continuously worked on 6 different projects as a team of 7 engineers. There was hardly any collaboration possible and we ended up with single points of failure because single engineers would end up being the only ones knowing about a particular system. Once we had hit the limit of reports for a manager (7 at the time), we needed to hire another manager but had a really hard time figuring out how to split the team because there was no clear structure. And boy was it hard to give the elevator pitch for the team in those interviews. We were a team that was ever expanding its work areas to catch things and never managed to retract back and focus on our core. We were aware of those problems and we always thought we will figure them out with time. For the time being it felt better to keep fixing things and worry about the rest later. Succumbing to the always existing, intriguing feeling that something that you can fix needs to be fixed right now.

And in the middle of 2016, all of this was suddenly gone. And after that very intense and honest 1-on-1 with Jason I felt I knew what I had to do. I joined a new team. I kept thinking about team focus and organizational structure. And when we set up to create a VMSO I went full in and went with the process. I talked a lot to Vanessa Hurst and Lara Hogan, both also Engineering Directors at the time, about VMSOs, team structure and direction. Both of them know incredibly well how to build engineering organizations and gave me so much insight and food for thought how to approach this task. I thought hard and good about what I as an engineer on the team wanted the team to be. And what I don’t want it to be. What were the things that I wanted this team to contribute to the business? What did I not want to bring around anymore and let stay in the past? I wanted to have a vision that I can align goals and work to that would provide focus and effectiveness for the team. And after a week or two with many, many VMSO meetings we ended up with a result that I was really happy with. It was also the first thing we delivered as a newly created team which helped immensely with team identity building. And in the following months to come I found myself often referring to the vision when the question came up of whether our team should be doing a particular bit of work or take over a certain ticket.

Since then I’ve also worked on the VMSO for our whole organization of Systems Engineering. And it was an even harder challenge to find something that matches the purpose of a dozen teams and gives them something to align their work to. But it was again a valuable lesson and a time spent building the structure for something I really want to be part of and make contribution to the business.

These past 18 months have been an incredibly intense learning period for me. Most of the things we did on my old team were the right things to do at the time. We worked on a lot of exciting and important projects that enabled others to build on top. And we were incredibly successful. But we also missed the point where the team needed to change to a different operating model to grow with the business. To align it to where the infrastructure needed to go. I learned to not think about work on an infrastructure team as just “keeping the lights on”, “fixing broken things” or “administering the machines everything else runs on”. But actually take the time to think about what to really contribute to engineering. What the state is I can see on the horizon and not just the work I know needs to get done right now. The two or three things that will make a difference over a laundry list of things that would be nice to do.

At work we have a group of people which I’m part of that work on making sure we have good frameworks in place for blameless postmortems and organizational learning as a whole. Part of that is moving past only investigating failure (via postmortems) and also look into investigating successes (via retrospectives). So in a similar way to how I’ve spent time understanding the unhelpful concept of human error, I wanted to learn more about the theoretical concepts of successful retrospectives. Unfortunately this was completely the wrong book for this. It is a great and very practical read for retrospectives in the agile sense and how to run successful meetings in general. However I wasn’t looking for that so I constantly kept thinking when we are going to dive into the meaty, theoretical stuff. This is in no way the authors fault and I would highly recommend the book as inspiration for improving your meetings. But for the theoretical underpinnings of retrospectives as an organizational learning tool I’m still on the lookout. Let me know if you have recommendations :).

The Basics

Every project I start gets this Makefile with some basic setups and variable definitions that I always want.

export GO15VENDOREXPERIMENT = 1

# variable definitions
NAME := coolthings
DESC := a nice toolkit of helpful things
PREFIX ?= usr/local
VERSION := $(shell git describe --tags --always --dirty)
GOVERSION := $(shell go version)
BUILDTIME := $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
BUILDDATE := $(shell date -u +"%B %d, %Y")
BUILDER := $(shell echo "`git config user.name` <`git config user.email`>")
PKG_RELEASE ?= 1
PROJECT_URL := "https://github.com/mrtazz/$(NAME)"
LDFLAGS := -X 'main.version=$(VERSION)' \
           -X 'main.buildTime=$(BUILDTIME)' \
           -X 'main.builder=$(BUILDER)' \
           -X 'main.goversion=$(GOVERSION)'

For the most part this just defines a whole bunch of meta data that gets compiled into the binaries via linker flags. This is a pattern I have seen in a lot of Go projects and I really like that this is somewhat of a standard thing to do. Especially with the static nature of Go binaries, the more helpful information you can compile into the binary the better it is when you have to figure out where a binary comes from.

I also always have a handful of tasks defined that are helpful for running tests and such, especially to have a uniform and documented way how they are run locally and on CI.

# development tasks
test:
	go test $$(go list ./... | grep -v /vendor/ | grep -v /cmd/)

PACKAGES := $(shell find ./* -type d | grep -v vendor)

coverage:
	@echo "mode: set" > cover.out
	@for package in $(PACKAGES); do \
		if ls $${package}/*.go &> /dev/null; then  \
		go test -coverprofile=$${package}/profile.out $${package}; fi; \
		if test -f $${package}/profile.out; then \
		cat $${package}/profile.out | grep -v "mode: set" >> cover.out; fi; \
	done
	@-go tool cover -html=cover.out -o cover.html

benchmark:
	@echo "Running tests..."
	@go test -bench=. $$(go list ./... | grep -v /vendor/ | grep -v /cmd/)

These make heavy use of go list to determine existing packages to run tests for. The rules also exclude the vendor folder as I don’t want to run those tests and the cmd folder which I will describe more in the next section.

Structure for multiple binaries

Go has this defacto standard of how to structure code if your build produces multiple executables. Since your main entry point in the app is always the main package and there can only be one per directory (which is also true for any other package btw) you need to separate different executables by directory. The pattern here is basically to have a cmd folder that contains subfolders for each executable which in turn just contain a main.go file. This is a pretty nice pattern, once you get used to it and is a convention that lets you easily create make rules for building those executables via the make wildcarding support.

CMD_SOURCES := $(shell find cmd -name main.go)
TARGETS := $(patsubst cmd/%/main.go,%,$(CMD_SOURCES))

%: cmd/%/main.go
	go build -ldflags "$(LDFLAGS)" -o $@ $<

This piece just finds all main.go files under the cmd folder and creates targets from them located at the top level of the repo. Then there is a rule to build those targets via a rule that ties them back to the source file via wildcarding again and runs go build with the linker flags from before.

Of course it’s good habit to provide man pages for your tools. So we can rig up a similar set of rules for building man pages for each executable:

MAN_SOURCES := $(shell find man -name "*.md")
MAN_TARGETS := $(patsubst man/man1/%.md,%,$(MAN_SOURCES))

%.1: man/man1/%.1.md
	sed "s/REPLACE_DATE/$(BUILDDATE)/" $< | pandoc -s -t man -o $@

all: $(TARGETS) $(MAN_TARGETS)
.DEFAULT_GOAL:=all

This lets us write man pages in markdown under the man/man1/folder named as ${cmd}.1.md and again uses wildcards in make to generate them top level via an implicit rule. I also added an all target there which is the default and builds all binaries and man pages.

Over time I’ve come to the conclusion that it’s really a good practice to have your main.go files be as slim as possible. Ideally all they should be concerned with is flag parsing, calling a method from your library packages, and formatting and printing the output to the terminal. Any actual logic should live in library modules somewhere else in your repo. This maintains a good code layout to extend, makes sure code is reusable, and provides good conventions for testing.

Installation

So now that we have rules to build the binaries, we also want to be able to install them to the PREFIX we have defined at the top. Go comes with an install command already (go install) which will put binaries in your $GOPATH/bin but there is no need to have to rely on that. Plus on a multi user system you want to provide tools for everyone anyways. Also let’s be real, go install is not a replacement for a real package manager. Just because go builds are fast and produce a static binary doesn’t mean it’s not a good idea to be able to build packages. Plus you want your man pages to be installed with your software as well of course. So let’s write some generic install commands:

INSTALLED_TARGETS = $(addprefix $(PREFIX)/bin/, $(TARGETS))
INSTALLED_MAN_TARGETS = $(addprefix $(PREFIX)/share/man/man1/, $(MAN_TARGETS))

# install tasks
$(PREFIX)/bin/%: %
	install -d $$(dirname $@)
	install -m 755 $< $@

$(PREFIX)/share/man/man1/%: %
	install -d $$(dirname $@)
	install -m 644 $< $@

install: $(INSTALLED_TARGETS) $(INSTALLED_MAN_TARGETS)

local-install:
	$(MAKE) install PREFIX=usr/local

We’re adding the PREFIX to all targets and man targets here to generate the paths to install. Then we write another implicit wildcarding rule that has the original targets as dependencies and performs install commands to put them into the prefix. This is a quick and easy way to have a generic make install target and also lets us easily add a local install target that we can use as a dependency for building packages later on.

Dependencies, Oh My!

If you’ve spent time with Go and make before, you will maybe have noticed a flaw in the building step of the Makefile so far. To revisit, we are building binaries from the source in the cmd folder with this implicit rule.

%: cmd/%/main.go
	go build -ldflags "$(LDFLAGS)" -o $@ $<

However this only tells make about the first level of direct dependencies for the binary to the cmd source. Chances are you are using library and vendored code in those. This means while go build technically knows about all dependencies, make doesn’t. And it will refuse to rebuild the binaries if something other than the cmd source changes. This is annoying but fortunately also fixable. A simple fix would be to just not have dependencies in make for the executables and mark them as .PHONY so that they are always regarded out of date. This pushes all dependency resolution back to the go tool chain which is nice, but kinda defeats half of the purpose of a Makefile as it will just run all the commands all the time. To be clear, in practice this is a fine solution and the downsides are mostly academic with the speed of a usual go build.

However it’s fun to figure out how to make things work and while we’re here already, lets utilize make to its full extent and make it aware of all dependencies. The details for the make side of things I got from this awesome blogpost which gives a great overview over automatic dependency management in makefiles. So now all we need is a way to get a list of all dependencies for a go source file. And of course, go files to the rescue again! As it not only lets us print packages for passing to the test runner, but also can print out all dependencies of a file. And with its -f parameter it also supports basic templating for printing out the results. Utilizing that we only need to do a small amount of post processing to print it in make dependency format and we are good to go.

# source, dependency and build definitions
DEPDIR = .d
$(shell install -d $(DEPDIR))
MAKEDEPEND = echo "$@: $$(go list -f '{{ join .Deps "\n" }}' $< | awk '/github/ { gsub(/^github.com\/[a-z]*\/[a-z]*\//, ""); printf $$0"/*.go " }')" > $(DEPDIR)/$@.d

$(DEPDIR)/%.d: ;
.PRECIOUS: $(DEPDIR)/%.d

-include $(patsubst %,$(DEPDIR)/%.d,$(TARGETS))

%: cmd/%/main.go $(DEPDIR)/%.d
	$(MAKEDEPEND)
	go build -ldflags "$(LDFLAGS)" -o $@ $<

The makedepend command here grabs all dependencies that come from github (which was a good enough approximation for me to filter out the std lib), cuts off the project prefix and appends /*.go to each dependency. With the go rules of having a package per folder, this also is pretty accurate most of the time and only occasionally serves false positives to result in a rebuild. We then adapt the implicit build rule to require the dependency file as well but also rebuild it on each build. And BOOM our Makefile knows almost perfectly a out all source dependencies.

Packaging and Documentation

I always aim for providing packages and good documentation for my Go projects. But I’ve already written about those things more generally here, so if you’re interested in the details of it, give that blog post a read. The important part is that the Makefile also holds the logic for building docs and packages, so they can be easily triggered from CI.

Cleanup

Since it’s also always good to make it easy to clean up artifacts and generated intermediate and output files, all makefiles also get some clean up tasks.

# clean up tasks
clean-docs:
	rm -rf ./docs

clean-deps:
	rm -rf $(DEPDIR)

clean: clean-docs clean-deps
	rm -rf ./usr
	rm $(TARGETS)
	rm $(MAN_TARGETS)

.PHONY: all test rpm deb install local-install packages govendor coverage docs jekyll deploy-docs clean-docs clean-deps clean

Equipped with those Make tricks I’ve been having tons of fun building Go code. Some of that is surely more involved than it has to be and especially the dependency resolution stuff is very bonus round. But it’s been super interesting to rig it up and I learned a lot of things about Make. And in the end that’s what it’s all about for me. (Besides having projects with a super nice to use structure :)

The short answer is that I optimize for the present and for mutability, which in itself is probably a completely useless answer. So let me try to elaborate what I mean by this. My day job is working in infrastructure engineering, specifically on a team that works on making writing code and deploying it as much fun as possible. This means while I’m technically a software engineer, the lines between software engineering and operations are blurry at best in my day to day work (which is a good thing and I very much enjoy it). I have worked on a bunch of systems, designed some of them, see almost all of them break in various ways and participate in as many architecture reviews as possible to give input on other people’s system designs. The main goal of my work however is to contribute to engineering happiness. This means I’m very aware of the intersection of technology and humans using it. In addition to that, working mostly on internal things means when the things I work on break, a lot of my coworkers are blocked from getting their stuff done. This can be petrifying. When I set out to write a new thing or fix up an existing one, I can test it out for my workflow but can also inevitably see how it could and will break for someone with a different workflow, editor, or set of dotfiles. And what makes it worse is that I won’t notice immediately and when it breaks for someone I might be in a meeting, unable to help right away. And I really hate breaking things.

So there I have 2 choices. Ship a thing that’s gonna break in some way. Or don’t ship anything. And the way I make myself be ok with shipping something that is flawed is by of course making sure I do a reasonably extensive attempt of testing it. But also make it as easy as possible to change or adapt later, to decide whether my original trade offs are still the right way to go and to rip things out if not. But not necessarily by trying to cater for all possible future use cases (the famous “premature optimization”) and sure as hell not by writing throw away code. Because everybody can tell you the only thing harder than building something is decommissioning it, and that goes doubly so for throw away code. How I try to achieve this is by dropping all of my context into documentation and automation in some form. This means code comments. Documenting my thought process on the JIRA ticket that relates to it. Writing thorough detailed commit message about my change. Writing unit tests that are being run on CI. A proper README. A thought through Chef recipe. A Makefile with all important tasks. A runbook. Those kinds of things. So that when someone else has to go and fix something (that could be future me or a coworker), they don’t have to spend minutes to hours to get up to speed on the context, decisions, and trade-offs I had and made to understand why I opted for this solution to the problem. So I’m very happy to write a 20 line commit message that links to the ticket and the CI/try run and mentions the people I consulted while working on this - even for a single line of change. I’m excited to add unit tests even when I’m “only” writing a vim plugin or a shell script. And I’m excited when I get to write man pages.

Because if I’m honest, yes I can see how things are flawed and can break in the future. But I don’t think I can accurately judge the impact of that flaw down the line. How severe will it be to reboot a bunch of things for a security update? How annoyed is the developer really gonna be about this tooling change? How pissed is my coworker gonna be to get paged for the thing I built? But also, what is someone gonna be able to build on top of or inspired by this? What am I free to do until the flaw really becomes a problem? And is my coworker gonna be ok with with all of this as they have learned something from it and had all the context available to fix it and make it better? Because the one thing I do know about current me is that there are gonna flaws in any solution. And I know one thing for sure about future me or my coworker encountering the flaw in the system: If they have the same context as I had when I deployed it, they are gonna be a lot happier, more empathetic as to why I made those choices, and be able to more quickly build on the existing solution. And if documentation, automation, and tests are in place it looks a lot less like some thrown together piece of code but more like the thought through project and honest attempt to fix a problem that had to make trade offs that it is. And up until now they trade offs were good ones and enabled a lot of other things that were impossible to tell before the fact.

So I guess the way I work through those feelings of overwhelming and paralysis is by making sure I can be damn proud of the work I’m doing in the present. And make sure it’s as easily adaptable as possible when the future comes around.

Of course this is just my personal way of dealing with it. And it is highly influenced by my character and the team I get to work with. And I hope I didn’t deviate too much from John’s original point in the blog post and that this post makes sense in some way. None of this actually makes the reality of flaws and dread of having to deal with them go away. But it gives me an anchor in the present and something to focus on to get things done. And a way to feel more prepared when the flaws do surface. Because I’d like to think of change as inevitable but also a good thing. Change you’re not prepared for however is when it feels most like a flaw.

In August last year I decided to finally scratch an itch I had for quite a while. The details aren’t super important here, just that it’s a shell script and that there was no sort of of pressure around it which made me take the time to write unit tests for it. That meant for me researching what existed in terms of frameworks and how people are generally approaching this. And unsurprisingly I found a number of ready to use unit testing frameworks, most of them modeled after the familiar patterns you can find in test frameworks for other languages. However I was also curious what a minimal testing framework for bash would look like. After all, all my script would be doing is create some files and directories on disk with some specific content. So I could verify it all with grep and test. So I decided to also use this side project to try and write my own minimal bash unit test setup. And while I mostly ended up doing integration testing for the script, it still made me think quite a bit about the basics of unit testing.

Unit Testing 101

One of the first questions I always get when someone hasn’t really come across a lot of unit testing is “what is a unit?”. And while technically you can probably argue for a unit being a lot of things, the most helpful one I’ve always found to be:

a unit is a function

This simultaneously gives a very concrete answer and also a starting point of what to do. When writing unit tests, start testing functions. This of course occasionally leads to the next question “what is a function?” and more often to the debate of how to make a function testable. For the first question I’ve again found this very reductionist answer to be the most helpful:

a function is a reusable piece of code that turns defined input into defined output

This is somewhat close to what you learn in school about functions in math and has helped me a lot with how I think about writing code.

Writing our first tested Bash code

Now with those definitions out of the way, there’s a plan on how to make a shell script unit testable:

1. Refactor your code into functions
2. Write tests for those functions

There’s a bit of a lesson to learn about side effect free functions but we can short circuit that by saying the only things your functions should rely on are variables passed into it. And it should always echo its results to STDOUT. This heavily reduces the possibility for side effects in bash functions but also limits the functions that absolutely have to do something other than just taking input and printing results to an absolute minimum. Your logic can live in the other functions most of the time. And those are the ones you can unit test. So now let’s write some functions and tests for them.

Let’s say we want a function to output the number of characters in a string. It could look something like this:

function num_chars {
  echo "${1}" | wc -c
}

It’s a very contrived example and you’re basically testing that wc works correctly. But it’s a useful example here to show some things. Notice how the function only acts on variables passed into it and prints the result to STDOUT.

Now let’s write a unit test for it.

function test_num_chars {
  local res=$(num_chars "foo")
  if [ ${res} -ne 3 ]; then
    echo "failed to assert that ${res} is 3"
  fi
}

And that’s it. That’s all you really need to do to write a simple unit test in bash.

Of course adding more tests now generates a lot of repetitive work. So we can write a helper function to do the assertion part of the test.

function assert {
 eval "${1}"
 if [[ $? -ne 0 ]]; then
   echo "${FUNCNAME[1]}: failed"
 else
   echo "${FUNCNAME[1]}: passed"
 fi
}

This helper function takes an argument which is a statement to evaluate. And depending on whether the eval exits with 0 or not, the test is regarded as passing or failing. It then prints out the result accordingly. FUNCNAME in bash is an array that holds the current execution call stack. And thus the first entry in it is the current function and the next one is the calling function. This gives us a nice way to determine which test is being executed and make it part of the output message.

And with this helper function in place, our test now looks like this.

function test_num_chars {
  local res=$(num_chars "foo")
  assert "[ ${res} -ne 3 ]"
}

Now we can already define a couple of tests and run them by calling the functions we defined. However that also gets very repetitive fast and you always have to remember to actually call the function when you define a new test. So let’s also write a helper function to do this for us.

function run_test_suite {
  for testcase in $(declare -f | grep -o "^test[a-zA-Z_]*") ; do
    ${testcase}
  done
}

This helper gets all the currently declared functions (via declare -f), looks for the ones starting with “test”, and then simply executes them.

Now all you have to do is call run_test_suite at the end of your file and all new test functions are automatically picked up as long as they start with “test”.

Fixtures for Tests

Now a lot of times in shell scripts you actually want to interact with files on the file system. And it’s not really feasible to always have everything just be variables to be passed in. In this case you can adapt your script by setting the base of where the files are you want to interact with. Something like this:

FILEBASE=${FILEBASE:-/usr/local/foo}

function list_files_with_a {
  ls "${FILEBASE}/a*"
}

Now you can set the variable in your test suite before you source your shell script with the functions to test. That way FILEBASE will already be set and the functions use it as their base. If you know create a directory for those fixtures in your tests directory, you can easily mock out file system details in a controlled way and test for them.

Dependency Injection in Bash

One of the most important things for me to get better at unit testing in general was understanding dependency injection. Writing code in a at that would let me completely drive function behavior based solely on what I’m passing in. And if I have to call an external resource make it so I can pass in the expected return value and only if it’s not set, call the external resource. A simple example could look like this:

function get_url {
  local url="${1}"
  local res="${2}"

  if [ -z "${res}"]; then
    res=$(curl -s "${url}")
  fi
}

Now you can use this function as you would normally do with 'get_url "https://unwiredcouch.com". However in tests you can also pass in a second argument which will be used as a locked out response instead of actually curl-ing the URL.

Wrapping up

In this short set of examples I hope it got somewhat clear that it can be straightforward to write a quick unit testing setup for shell scripts from built in functionality and start writing tests. I’ve also shown some techniques to write more testable bash to begin with. If you’re interested in reusing the code if pushed the (slightly more improved) version of this I use to GitHub. It provides nicer output, more details and properly returns a non-zero exit code if something failed, so you can run it on CI. If you want more functionality or more advanced testing support, I’ve listed some alternatives in the README.

And on a slightly related note, you should start using shellcheck when writing bash. It’s such an awesome way to get feedback about how to write better shell scripts and I’ve learned tons already just from the errors, warnings, and suggestions popping up in my VIM quickfix list.

But the most important part is that testing isn’t magic and doesn’t have to be complicated. You can get started immediately with just the basics of any language. And especially starting to write tests for shell scripts is lots of fun.

Fast forward 25 years or so I have gone to a bunch of LAN parties as a kid, went to university to study computers and eventually got a master’s degree in computer science. I’m running most of my own infrastructure, have built my own home router, had DNS servers from a Dutch ISP take zone transfers from a computer running in a camper van toilet, and upgraded PHP on a big e-commerce website without downtime. It’s fair to say I’ve learned a couple of things and know my way around computers most of the time. And yet I am still deeply uncomfortable breaking things.

What’s wrong with breaking things?

Ironically I now work in an industry that basically worships breaking things. From famous company mottos like “move fast and break things” to phrases that get quoted out of context like “ask for forgiveness, not permission” everybody seems to love being able to break stuff. What it doesn’t take into account is that breaking things doesn’t happen in a vacuum. Your actions always impact others. Even if you’re on-call, the nature of our complex systems means that nobody has a perfect overview over all interactions. And nobody can be sure they will be the only one to get paged and not someone else downstream who is just sitting down to eat with their family. And claiming you can is in my opinion more an unhealthy sign of hubris than healthy engineering. More likely than not it’s gonna ruin someone else’s day.

In addition the romantic picture of the engineer who is not afraid of breaking things and thus disrupting whole industries on the way is not an evenly distributed one. As Kate Losse already wrote in “the unbearable whiteness of breaking things” it’s usually just the white men again who are able to get away with it. For everyone else this is likely gonna end less well.

It’s also a very unhealthy and non-collaborative way of approaching things. It assumes a very negative default instead of working together. And it keeps reinforcing a stereotype that only works because there’s a team of people picking up the pieces once the magic disruptive engineer is done.

But it’s the only way to learn

Now you might say “Hold up there. Breaking things is the only way to learn. You don’t know a technology until you’ve seen it break.” And I partly agree with you there. However there is a big difference between doing gamedays where things are turned off and shut down in a controlled environment, where everybody got a heads up this is going on, and systems are observed as a team to learn how they behave. This is a great way to learn about technology. And I encourage everyone to do this. Equally if things do break it’s paramount to investigate those incidents in a blameless way to maximize the things to learn from it.

However: Just testing in prod. Not bothering to write unit tests. Not going through staging before going to prod. Rolling out a change to all servers at once just to save some time. Those are the things no one learns a lot from. Other than the fact that you can quickly make a day awful for a bunch of people. And that you might be a shitty coworker.

Let me be very explicit. I don’t think you can prevent every failure from happening. I don’t think people should be punished if something breaks during their daily work. Things break. There’s nothing you can do about that. What I don’t condone is approaching everything with the attitude that it’s ok to actively break things. That it should be the default. That your need of changing something is more important than someone else’s need of not being interrupted. That it’s ok to lean all the way towards efficiency and away from thoroughness. This is not disruption it’s just lack of empathy. The default should always be to try and not break anything. There should be a way to make it as easy as possible to catch errors early on. To test things before they go to prod. To get confidence in something without disrupting someone else’s day. And if there isn’t, maybe this is something to spend time on making better first. It beats breaking things by a long shot. And is actually something to be proud of.

And I think this doesn’t have to be. All of this talk about egoless programming and throwaway code doesn’t mean you can’t take pride in what you create. As a programmer (and that includes ops people, security engineers, designers, etc - if you commit to a repo, you’re a programmer; don’t let anyone take this away from you) you now have access to a myriad of wonderful tools and services that make it so much fun to write and use software.

The README

I almost feel like this goes without saying, but you should take some time to write a proper README. It might take 15 or 30 minutes for you to write it. But if 2 other people don’t have to spend 10-20 minutes figuring out how your project is supposed to be used or if it even solves their problem, it has already saved time. The usual things like usage examples and installation instructions should go in there. Plus as you probably know, GitHub shows READMEs very prominently in a nicely rendered way. So your project already feels a lot nicer to use.

And while you’re there, also create a CONTRIBUTING.md. GitHub will show it whenever someone is creating a pull request. So you can put some information in there how you would like to receive contributions which can act as some helpful guidelines and make it a lot less scary and awkward to contribute.

Unit Tests and Code Coverage

It’s no secret that I’m a fan of writing tests. And it has really become more fun over the last years as frameworks and best practices have improved. In basically all languages there exists now at least one unit testing framework that is easy to use. Some languages even come with one in their standard library. So there is no real reason to not write tests. After all you are already testing your changes manually. Why not have the computer do the tedious work? If you’re looking for some introductory material on testing, we have open sourced our Testing 101 material we use at Etsy to teach testing. The point here is not that you will never have bugs in your code because you write tests. You are gonna reduce the number of bugs for sure. But more importantly it provides some confidence factors for contribution and sets a visible expectation of what things are being automatically tested. Beyond that it provides example code for how to use your code and codifies the intent you had while writing the original functions. It also automatically serves as a first client for your API outside of the intended use case the code was written for. Thus often uncovering a good chunk of design problems.

And while you’re at it, add code coverage as well. Coverage is one of those tools that most people either love or hate. But the main point for me is that it sets expectations for which parts of your code are regularly exercised through tests. Not more not less. It’s also not a simple way to make sure you never have bugs. It can’t, as it’s a tool that is concerned with syntax and not semantics of your code. But what it can do is make you think about code paths more explicitly. And through that make you think more about how to test things. And also add instructions about how to run the tests into the CONTRIBUTING.md file so prospective contributors don’t have to guess or search.

Continuous Integration

Once you have tests, the next logical step is to run them on a continuous integration service. I love Travis CI for this but there are many others out there. Most services now support GitHub pull request status updates which makes it so much less work to maintain external contributions to your project as you’ll immediately see whether or not the pull request passes tests. But the most important bit about hooking up a CI system to run your tests are the fact that you’ll know it works somewhere else besides your laptop. Plus it gives you a platform to trigger many other useful things (which I will talk about in a bit) after your tests have successfully run. And for that you can even just use the Jenkins setup you probably already have at work or any other CI setup really.

Code style and Static Analysis

Another thing I really enjoy is the renewed rise of static code analysis tools. And I’ll just throw code style checkers in that same bucket. I’ve met quite a lot of people who hate coding style checkers. The arguments usually go like this: “if you can’t read an if statement with a missing space before the curly brace I don’t want you to write production code anyways”. It’s amazing how many people have strong opinions on things they claim to not matter. The point here is not so much the correct way of writing code but rather a consistent way. Chances are your code is being read by a good number of people, depending on how important/popular it is. Having a consistent style makes it easier to read. Code without style guidelines can be like reading a book where every page was printed in a different font. It doesn’t matter for functionality but it would be a lot more annoying to read. Plus there is literally almost no overhead. Most languages have editor plugins now that will format text for you, some languages even come with tooling. But what it shows is that you care about this code to be readable and accessible. And that you have an easy way of making these coding styles visible and applicable. Static analysis is usually a less contestant topic. There are usually a lot of things that aren’t immediate problems but are helpful fixes. And it’s as well a sign of you caring about the quality of your code.

Codeclimate is a wonderful service I use to do this. It has static analysis plugins for a lot of languages and is super easy to set up. It integrates with the Github status API and shows changes for every commit and pull request. That way you can have a computer enforce things like indents, formatting, and problems that a static analyzer can find and you can concentrate on the logic and spirit of the change.

Packaging and Deployment

These are topics very dear to my heart. A good packaging and deployment setup makes the user experience of software so much better. Not having to think about where to copy that one file, no need to curlbash some weird script, having things come from the package manager you already use. All those things make it a wonderful experience to get started on a piece of software. And the state of things there also only has gotten better over the years. A lot of the language specific package managers now have nice tooling around creating and uploading install packages for their platform. Some like packagist even go so far as to just fetch things for you from GitHub and create releases on tags automatically. There is literally no reason to not have your PHP project on there. But even ruby gems and Python modules you can upload easily in an automatic way from your CI system. Travis CI has a whole section on deployments and integrations with most of the popular services. So do other CI platforms. fpm has made it ridiculously easy to build packages for Linux. And with packagecloud you can host them in an amazingly accessible and user friendly way. You can even have your packages built and uploaded from your CI system as well with something like this:

NAME=restclient-cpp
VERSION = $(shell git describe --tags --always --dirty)
BUILDER = $(shell echo "`git config user.name` <`git config user.email`>")
PKG_RELEASE ?= 1
PROJECT_URL="https://github.com/mrtazz/$(NAME)"
FPM_FLAGS= --name $(NAME) --version $(VERSION) --iteration $(PKG_RELEASE) \
           --epoch 1 --license MIT --maintainer "$(BUILDER)"
           --url $(PROJECT_URL) --vendor mrtazz \
           --description "C++ client for making HTTP/REST requests"
           --depends curl usr

# build rpm and deb
fpm -t rpm -s dir $(FPM_FLAGS)
fpm -t deb -s dir $(FPM_FLAGS)

# deploy to package cloud
package_cloud push mrtazz/$(NAME)/el/7 *.rpm
package_cloud push mrtazz/$(NAME)/debian/wheezy *.deb
package_cloud push mrtazz/$(NAME)/ubuntu/trusty *.deb

Or use their integrated deployment provider which his even less setup work.

Documentation Deploy

And speaking of automatic build and deploy. The same goes for documentation. One of the most genius features of GitHub in my mind is the fact that every repository can have a gh-pages branch whose contents are getting published as a website under http://username.github.io/reponame. This makes it extremely easy to host a documentation page for your project. And with GitHub’s CNAME support you can even have a custom domain for your project point to it. The fact that it’s just another branch in your repository means that you can easily automate the deployment of docs alongside your code for example from (you probably guessed it by now ) your CI system. Thanks to GitHub pages this is as easy as:

# generate docs
install -d docs
echo "projecturl: $(PROJECT_URL)" >> docs/_config.yml
echo "basesite: http://www.unwiredcouch.com" >> docs/_config.yml
echo "markdown: redcarpet" >> docs/_config.yml
echo "---" > docs/index.md
echo "layout: project" >> docs/index.md
echo "title: $(NAME)" >> docs/index.md
echo "---" >> docs/index.md
cat README.md >> docs/index.md

# deploy to github
cd docs
git init
git remote add upstream "https://${GH_TOKEN}@github.com/mrtazz/$(NAME).git"
git submodule add https://github.com/mrtazz/jekyll-layouts.git ./_layouts
git submodule update --init
git fetch upstream && git reset upstream/gh-pages
git config user.name 'Daniel Schauenberg'
git config user.email d@unwiredcouch.com
touch . && git add -A .
git commit -m "rebuild pages at $(VERSION)"
git push -q upstream HEAD:gh-pages

If you run this from Travis CI with an encrypted GH_TOKEN environment variable, make sure to suppress command echo-ing for the git remote add command as it will otherwise write your token to the log in plain text.

And even if you just start with publishing your README, you have a nice website in place already to build upon. Maybe add doxygen or other reference generation to it. Add a better getting started guide. Maybe someone else contributes their notes. It’s more likely the easier it is to do. And it makes documentation contributions look more like the first class contribution they are. And less like a nice side addition. Which is something almost every project can benefit from.

Build and Automation

With all those wonderful things in place and hooked up, you also want to optimize for the common part of contributing. The local feedback loop. So while it is awesome to have all those services hooked up, it should also be obvious how to run and test them while working on something. This is where build automation via a tool like make comes into play. If you don’t have to look at your Travis config how to run tests but instead can just run make test or make coverage to get coverage information or even make packages to have debs and rpms build locally it’s a lot more fun to contribute. And it’s not that much more work. When you hook up those things anyways, you can then just run the make commands from your CI system as well. Which also makes it a lot easier to debug if it goes wrong.

Show it off

And finally please let everyone know that you have all those things in place for your project. Most CI systems and other services now support an HTML embedded badge that shows the build status, code coverage percentage or static analysis results in a little image. It is green when things are ok and red or yellow otherwise. Which lets everyone know the current status of your project immediately when loading the README on GitHub or the website of your project. For everything else there is shields.io which lets you create custom badges via a simple URL structure so you can have the license you use, the location of the packages and other things that are not red/green right up there.

Do I really have to do all of this?

I’ve given a lot of examples for things you can or should do to make your project nicer to use. And there are a myriad more, Heroku deploy buttons, npm dependency checkers, slack links, etc. I’ve mostly focused on a very specific set of things I use regularly for my projects. And it’s also very much focused on open source repositories or at least repositories hosted on Github.

I’m very aware that not all of these things always apply to or make sense for a project. Some languages don’t have the support of your coverage platform. You want to use another CI service. Your code is hosted in your corporate network and you don’t think you have the time to set all of these things up.

The real answer here is to always try to strive for this. A lot of setups can literally be copy and pasted once you’ve done it for one project. And again while all the services mentioned are public ones, there are a lot of integrations you can emulate in house with your existing CI and deployment system and some HTML. If you only do a third of the things I described here your project will already be in much better shape. And people will be more happy to (have to) use your code. Which in my mind is something to be proud of.

So without further ado, here’s my reading list for 2015:

The Whole Woman by Germaine Greer

As mentioned last year I started reading this book in 2014 and finished it early 2015. I overall liked it and it was really good in giving me different ways to think about feminism and how the whole system works together to enable sexism and exploitation. It’s also a good resource to understand better how closely related feminism and capitalism really are. However it comes with a really serious trigger warning. Germaine Greer is known to have very transphobic/cissexist views and this book is no exception. It is restricted to one chapter but those opinions - which I don’t share at all - are definitely in there. So if this is a trigger for you, it’s probably better to skip this book.

The pattern of devaluing women’s contribution is as old as human civilization

Feminism is for Everybody by Bell Hooks

I’ve known about this book for a while now, but up until early 2015 it was only available in print. And since I don’t really like owning physical books and read exclusively on my Kindle and iPhone I hadn’t bought it yet. So when I found out there is a Kindle version now, I immediately bought it. As expected, the book is really good and gives a good primer on feminism and the historical context from the author’s perspective. It reads less extreme to me as Greer which is very much in line with Hooks’ other writing. Definitely highly recommended for learning more about feminism.

Feminism is a movement to end sexism, sexist exploitation, and oppression.

Designing for Performance by Lara Hogan

My coworker Lara wrote this book last year and it was a lot of fun watching her process and how she knocked out that book. Since then it was on my list of books to read. Especially since I tend to shy away from frontend things in my day job and want to get better at not doing that. The book is a wonderful introduction into web performance especially from a design view. It gives very solid technical details on a lot of things like browser rendering and image formats that I only had very superficial knowledge of before. I really enjoyed it and the book lead me to reduce the page weight of this blog by 92% which was tons of fun to do as well.

The largest hurdle to creating and maintaining stellar site performance is the culture of your organization.

Manage your Day-to-Day by Jocelyn Glei

This book sparked my interest while I was looking for improving my daily routines. I was often just starting the day as it happened often leaving me feel disorganized, unproductive, and imbalanced. Reading “Manage your Day-to-Day” gave me a lot of ideas of what things to try and add to my daily routine. And also to try and even have a daily routine. Something I picked up again through this book was journaling and while it has been on and off for the last couple of months I really enjoy it. The book was not mind blowing for me but I enjoyed reading it and definitely would recommend it if you are looking for inspiration for your daily routine.

It takes willpower to switch off the world, even for an hour.

Leading Snowflakes by Oren Ellenbogen

I’ve heard about this book ever since it was released and a lot of people I know speak very highly of it. And they weren’t wrong, I basically devoured the book in a weekend. It’s very well written and has a ton of actionable advice for engineers becoming managers. But I would argue that this description really limits the value of the book. I have no intention to become a manager at the moment however the book was really interesting and helpful for me. I think it’s a great read for anyone looking to grow more into a leadership position.

The Highly Sensitive Person by Elaine N. Aron

I had no idea about the concept of highly sensitive people until I read this article. It has a pretty click-baity headline but it really hit home for me. So I decided to learn more about it and this book was the most prominent resource to pop up in my search. It’s a really good book with a lot of great psychological insights and explicit case studies. At times the way high sensitivity was described was a bit too feel-good for my taste. At other times I would almost throw my kindle across the room as the author managed to really sneak up on and hijack my sensitivity. The book focuses a lot on what usually goes wrong during childhood for highly sensitive people and makes it a point to relive memories and traumas through the lense of high sensitivity. This is a practice I really enjoyed although it felt a bit much to me at times as I consider my childhood to have been a happy one. On the other hand I started to do this practice with every day situations at work to help me understand why I feel what I feel in different situations. I identify myself as a highly sensitive person and the book was an extremely good read to help me understand better what this could mean for me and my days.

Recoding Gender by Janet Abbate

I have a very complex relationship with the profession of “software engineering” and how it’s often defined in a non-inclusive way and as the profession of the golden children of society. Part of that is that I had always known a bit about the origins of programming and that a majority of programmers used to be women. But I didn’t know a lot about it which is why I was excited to read this book. And it was great! The book walks you through the beginnings before and during WWII and what programming meant back then. It discusses how the emerging industry in this field changed job prospects and economic chances for women. But it also discusses how the image of a programmer changed as more and more men participated. It’s full of historical facts and documents and a more than wonderful read. It sparked a lot of thoughts for me and changed the way I think about my profession even more.

the traits that managers found most problematic in programmers were those stereotypically associated with men

You had me at ‘Hello World’ by Dona Sarkar

I found this book through Camille tweeting about the fact that she was also interviewed for it. “You had me at ‘Hellow World’” is a collection of interviews with industry leaders from successful companies about the many aspects of leadership and mentoring. It’s a pretty lightweight read and a great resource to get some insight how successful people talk about those topics. It does a great job in conveying how important skills outside of writing code are. And it provides good examples of how to use those for your advantage.

Nonviolent Communication by Marshall B. Rosenberg

This has been recommended by many people I work with as a wonderful resource about positive human communication. And as - especially in a growing engineering org - communication is one of the most important skills to try to master, I decided to finally read this one. It’s a very interesting book with an approach to communication that is rarely taught especially not to men. It focuses on a collaborative rather than a competitive style of communication and the goal to reach agreements over winning arguments. The examples in the book are often pretty extreme coming from the author’s work as a diplomat. And even though those are great to demonstrate how this way of communicating can work in the most extreme cases, it also shifts its focus a lot on explicit diplomatic style discussions. There are more examples that are more directed towards every day situations and even though the author is very explicit about this being useful in regular work meetings as well, I had a very hard time understanding how to practically apply those lessons in a meeting for example. That being said however it made me think a lot more about the way I communicate and what I’m saying versus what I want to say. I have also applied that way of communicating successfully at least once since reading the book. And I look forward to try it out more.

The Retrospective Handbook by Patrick Kua

At work we have a group of people which I’m part of that work on making sure we have good frameworks in place for blameless postmortems and organizational learning as a whole. Part of that is moving past only investigating failure (via postmortems) and also look into investigating successes (via retrospectives). So in a similar way to how I’ve spent time understanding the unhelpful concept of human error, I wanted to learn more about the theoretical concepts of successful retrospectives. Unfortunately this was completely the wrong book for this. It is a great and very practical read for retrospectives in the agile sense and how to run successful meetings in general. However I wasn’t looking for that so I constantly kept thinking when we are going to dive into the meaty, theoretical stuff. This is in no way the authors fault and I would highly recommend the book as inspiration for improving your meetings. But for the theoretical underpinnings of retrospectives as an organizational learning tool I’m still on the lookout. Let me know if you have recommendations :).

Cybersexism: Sex, Gender and Power on the Internet by Laurie Penny

This short book by Laurie Penny is a very good read about sexism in the age of social networks and the omnipresent Internet. It does a great job at talking about how a lot of familiar concepts of “offline sexism” are reinvented online and no news to women. It’s short and insightful enough to recommend reading it without hesitation.

Perhaps one reason that women writers and technologists have, so far, the calmest and most comprehensive understanding of what surveillance technology really does to the human condition is that women grow up being watched.

The Boy Kings by Katherine Losse

The biography of Kate Losse about her time at (earl stage) Facebook is in my mind a must read for any software engineer and especially if you’re a man. It gives an extremely good insight view into what happens when young men are suddenly in charge of a ton of money. But more importantly it talks very bluntly about how engineers are treated differently from most other employees for our supposed gift to turn any idea into gold with code.

Technology carries with it all the biases of the people who make it, so simply making the world more technical was not going to save us.

The Art of Mindfulness by Thích Nhất Hạnh

This is another super short read and the de-facto introductory book to mindfulness meditation. There’s not a lot to say here. It’s good, give it a read as it’s short enough to not matter if you end up not liking it. I started meditating regularly after reading it and it has been a great experience.

Men Explain Things to Me by Rebecca Solnit

This collection of essays titled for the aggressive tendency of men to always have to explain things to women while assuming they have no idea what they are talking about. The first essay brings this to a point by telling a story of a party where a man mansplains to the author the book she herself wrote. Without having actually read it. The book than continues with more essays that talk about a lot more darker things like discussing domestic violence. The over arching theme is that the credibility of and respect towards women is continuously diminished to maintain the status quo and its power imbalance. Some of the essays towards the end of the book are not easy to read but it’s more than worth it.

Credibility is a basic survival tool.

Don’t Make Me Think by Steve Krug

I’m one of those engineers who used to happily claim to not have any frontend skills and just not be good at design. I came to loathe this thinking over the years and decided that if I can’t do something I want to learn at least the basics. This is one of the reasons why I read “Designing for Performance” as mentioned above. Thankfully I also work with a ton of talented designers and one of them is Jessica Harllee. I talked to her about suggestions to get started with learning about design. And she said I should read “Don’t make me think”. And she wasn’t wrong. The book is a wonderful introduction into usability and design. The beauty of it is that while reading it, all of the things mentioned are total no-brainers. But you have to remember it while designing things. The other interesting thing for me was that while all of the examples in the book are web based (with some brief stints into mobile) I could totally think of CLI apps I’ve written in the past that totally do the wrong thing design-wise. Definitely a recommended read.

The Internet of Garbage by Sarah Jeong

In this book Sarah Jeong - a journalist trained as a lawyer at Harvard Law School - talks about the problem of online harassment. It’s another short but really good one. I’ve learned a ton about copyright law and the limitations of current legislation when it comes to online harassment. But also things that do work and what things could be attempted. It’s a very sobering look at the current state of social networks, online harassment and tooling and legislation to help fight it. Definitely worth a read if you spend any time on the internet.

Yet Another Graphite Dashboard

However since I wasn’t using Librato anymore, I now had to find a way to get nice overview dashboards for all of my metrics. And I looked into the usual suspects. But all of them seemed to need a very elaborate setup and running an additional application server besides mod_php which I was already running for Nagios just for some graphs embedded on an HTML page didn’t seem like a thing I wanted to embark on. I always liked the way we approach dashboards at Etsy a lot. It’s basically a PHP framework that gives you a nice way to create graphs from Graphite or Ganglia and combine them into dashboards. However it was a bit overkill for my use case and I would have to write all the code for a typical collectd host anyways. So I wrote my own little PHP script to generate a list of graphs from a config file. And it was really nice, took me 20 minutes, was a lot of fun, and did everything I wanted it to do. I decided to just use Twitter Bootstrap for the UI, which means it also looks nice on my iPhone and it’s aptly named Yet Another Graphite Dashboard.

Chef integration and additional metrics

Now that I had this nice way of viewing dashboards, I wanted to have more graphs. I have long made a choice to track as much as possible in Chef for my personal infrastructure. And graphing is no exception here. Setting up the initial collectd install is a bit manual as I depend on some options that are available in the ports but not the official package builds (my infrastructure is still all FreeBSD). But the configuration and graphing additions are all fully chef-ed. I took the way we have Ganglia set up at Etsy as the role model. We have a setup chef-ed to every box that runs all scripts prefixed with gmetric- in a certain location on a minutely cron. This means in order to get a new set of metrics, you just have to write a shell script that ends up calling gmetric and put it in Chef. And a couple of minutes later graphs for all boxes will magically appear in Ganglia. I did the same for my collectd setup via collectdctl and it looks a little bit like this:

* * * * * for SCRIPT in $(ls /usr/local/collectd/collectd-*); do command ${SCRIPT}; done

This means I can now easily add new metrics by dropping a script in there that utilizes the collectd CLI tooling. However since collectd has a very specific type setup, each script also needs a corresponding configuration in a custom types db. I also track this in Chef so it’s not too big of a problem. An example script to track disk temperature looks like this:

#!/bin/sh

PLUGIN_NAME="disktemp"
HOSTNAME=$(hostname -f)
SMARTCMD="/usr/local/sbin/smartctl"
COLLECTD="/usr/local/bin/collectdctl"

for disk in $(ls /dev/ada* | grep -o "ada[0-9]$"); do
  TEMP=$(${SMARTCMD} -a /dev/${disk} | awk '/194 Temperature_Celsius/ {print $10}')
  ${COLLECTD} putval ${HOSTNAME}/${PLUGIN_NAME}-${disk}/celsius_current interval=60 N:${TEMP}

  if [ $? != 0 ]; then
    echo "ERROR ${0}: ${HOSTNAME}/${PLUGIN_NAME}-${disk}/celsius_current interval=60 N:${TEMP}"
  fi
done

Another thing that Ganglia gives you for free is a section for additional metrics that just appear as soon as you send them with an optional group name to group them by. In order to emulate that in my setup, the recipes for each collectd script are also defining node attributes with the Graphite graphs they are generating and how they are supposed to be displayed. This made a lot of sense to me as when I’m writing the scripts I have the generated metrics in my head anyways. And it’s easy to just drop them in a node attribute. So for tracking disk temperature for example, the recipe looks a bit like this:

cookbook_file "/usr/local/collectd/collectd-disk-temp.sh" do
  source "collectd-disk-temp.sh"
  owner "root"
  group "wheel"
  mode 0755
end

node.default[:yagd][:additional_metrics][:disk_temperature] = {
  "Disk Temperature" => "collectd.#{node[:fqdn].gsub(".","_")}.disktemp-ada*.celsius_current"
}

The next step was now to not have to manually edit the config file for my dashboards anymore. I now had all the data I needed in Chef, so all it took was generating the config file there from a Chef search and all graphs were magically appearing as soon as both the node to monitor and the dashboard host had run Chef. This can take up to 20 minutes worst case (I run Chef every 10 minutes) which is really not a big deal for me. The Chef search code that does this for me looks like this:

hosts = []
nodes = search(:node, "domain:*unwiredcouch.com")

nodes.each do |computer|

  this_computer = {}

  this_computer[:name] = computer[:fqdn]
  this_computer[:cpus] = computer[:cpu].nil? ? 0 : computer[:cpu][:total]
  this_computer[:apache] = computer.recipes.include?("apache")
  this_computer[:interfaces] = computer.network.interfaces.keys.select {|k| !k.to_s.start_with?"lo" }
  this_computer[:filesystems] = []
  computer.filesystem.each do |k,v|
    name = v[:mount] == "/" ? "/root" : v[:mount]
    # cut out leading '/'
    name[0] = ''
    # substitute '/' with '-'
    name.gsub!("/", "-")
    # substitute '.' with '_'
    name.gsub!(".", "_")
    # and add to array
    this_computer[:filesystems] << name
  end
  this_computer[:additional_metrics] = computer[:yagd][:additional_metrics] unless computer[:yagd].nil?

  hosts << this_computer

end

template "#{dashboards_dir}/config.php" do
  source "yagd.config.php.erb"
  owner "www"
  group "wheel"
  mode 0775
  variables( :hosts => hosts )
end

And this is the accompanying erb template that gets rendered into a PHP file to serve as the configuration for my dashboards instance.

<?php

$CONFIG = array(
    'title' => "dashboards",
    'navitems' => [
        'Hosts' => '/hosts.php',
        'Graphite' => '/graphite.php',
        'Twitter' => '/tweets.php'
    ],
    'graphite' => [
      'host' => "https://graphite.example.com",
      'hidelegend' => false
    ],
    'hosts' => array(
      <% @hosts.each do |host| %>
       "<%= host[:name] %>" => array(
         "cpus" => <%= host[:cpus] %>,
         "apache" => <%= host[:apache] %>,
         "interfaces" => <%= host[:interfaces].to_json %>,
         "filesystems" => <%= host[:filesystems].to_json %>,
         "additional_metrics" => [
           <% host[:additional_metrics].each do |name,values| %>
           "<%= name %>" => [
             <% values.each do |title,metric| %>
               "<%= title %>" => "<%= metric %>",
             <% end %>
           ],
           <% end %>
         ]
       ),
    <% end %>
    )
);

The Cleanup

After all of this configuration, my dashboard setup was working beautifully and I added more and more graphs I was interested in. But it was still more or less the 20 minute PHP code I had initially thrown together. This was technically fine and I didn’t mind it too much. But at the same time I thought it might be nice to bring it in a state where it’s usable for others. So I decided to take some time to clean it up and make it more generically usable. So I refactored the code, added unit tests to run on Travis CI and hooked it up to Code Climate so I could have a computer tell me how I can improve the code quality.

With this refactor in place it’s now fairly easy to get a dashboard page that shows the status of all hosts:

<?php

require __DIR__ . '/../vendor/autoload.php';

include_once("../config.php");

use Yagd\CollectdHost;
use Yagd\Page;

$page = new Page($CONFIG);
echo $page->getHeader($CONFIG["title"],
    $CONFIG["navitems"]);

foreach($CONFIG["hosts"] as $host => $data) {

    $fss = empty($data["filesystems"]) ? [] : $data["filesystems"];

    $server = new CollectdHost($host, $data["cpus"], $fss,
                               $data["interfaces"]);
    $server->setGraphiteConfiguration($CONFIG["graphite"]["host"]);
    echo "<h2> {$host} </h2>";
    $server->render();
}

echo $page->getFooter();

You can also inject a selectbox into the header to have it be possible to just select a single server instead of always displaying all of them. This makes it really nice to be able to just browse to something like https://yagd.example.com/hosts.php?hostname=foo.example.com and get a quick overview of how that that host is doing. Plus it gives you a URL you can link to from anywhere else.

Also since this is just plain PHP, entirely driven by the config file, it’s possible to have a per cluster view by passing a URL parameter like ?cluster=name and then changing the include_once() code in that example to include a different config file based on that. And since Chef already knows or can know all that data (maybe each cluster is its own role? ), it’s just a matter of writing some ruby to generate different sets of config files for the dashboards.

Summary

Writing yagd has been a lot of fun. The initial version took - as I already said - about 20 minutes to write and I learned a ton of things while refactoring it into a usable PHP module. You can install it via composer from packagist if you want to try it out and use it for your own dasboards.

However the point of this is not so much that I wrote yet another dashboard framework, but rather how easy it was to get this going. Sure it’s not super trivial to get your infrastructure into Chef if it’s not. And it also takes some time to install Graphite if you aren’t familiar with it. But with those things in place, you have all the building blocks to quickly whip up a nice dashboarding solution with some simple PHP.

As much as I love how many frameworks and libraries there are to already solve those problems for us, I think it’s a good practice to occasionally go back to the basics and think about what the simplest solution is I actually need. In my case this was showing graphs on an HTML page in a somewhat structured way. I didn’t need anything more fancy. And there was no reason to try and find the dashboard solution that would do that, preferably in PHP so I don’t have to set up yet another application server, which most likely solved way more problems that I actually had.

If you want to give yagd a try, I would love to hear what you think. I currently track 4 servers and 2 jails with it, but for this and other reasons it won’t be the solution to all dashboarding problems. Nor should it. The way more important thing in my mind is that it’s solving a very specific problem I had, in a pretty simple way. And in addition served as a side project for me to learn a lot of things about writing PHP, setting up phpunit, using codeclimate, and creating a reusable package on Packagist I didn’t know before.

So why is this important? Everybody does not drink coffee at some point, right? It’s important because it’s part of something I’ve tried to continuously include more and more in my life over the years. Taking timeouts and making time for reflections. Humans tend to be creatures of habit. If you want to achieve something that needs constant work, build a habit out of it. You want to read more books? Spend 30 minutes every morning with just reading. You want to lose weight? Have a strict habit of going to the gym every Monday, Wednesday, and Friday. Want to learn a language? Start memorizing words every day on your commute. If you search Google for “building habits” you will find a myriad of websites, articles, and tutorials of how to hack your life and yourself into a more productive version by making things a routine. And it’s true, building habits is a very effective way of incorporating new things into your daily life.

However the same also goes for negative habits. Smoking, drinking a beer every day after work, eating too much unhealthy food, constantly immersing oneself in the stressful routines at work, not going to the gym because the day already feels too planned out every day. Once something has become an ingrained part of your day, it’s really hard to notice whether it’s there for fun and enjoyment or actually harmful, and it’s even harder to get rid off.

The way I’m trying to battle those things becoming too much of a routine is by taking timeouts and reflect on my choices. Because that’s what they all are. Choices. I do reflect a lot without necessarily taking a timeout, however when I do take a timeout from something, I automatically also reflect on what impact that thing has on my life and whether I like it or not. Over 9 years ago I thought deep and hard about why I was smoking and decided that it wasn’t worth it and stopped. I had multiple times in my life where I completely stopped drinking alcohol for months or even years because I had stopped and reflected on whether I really want to drink this beer because of its taste or because of habit. I generally try to eat vegetarian by default and then have meat twice a week and fish/seafood two to three times a week. I periodically stop and think about whether this is still the case and whether it has been for the last couple of weeks or months. If it doesn’t then I think about why and if it’s because I decided to or just because I got carried away in another routine.

But this doesn’t only go for negative things in my life. I have a very specific set of tools that I rely on heavily for my work. I write code and words only in vim, I have all my todos in OmniFocus, I exclusively use iPhones, OSX on my laptops and FreeBSD on my personal servers. And I really like it. My setup is as close to ideal as I can imagine. However I still occasionally stop and revisit those choices. I have used Atom for a bit when it came out and have tried to use emacs as well. I’ve switched my todos over to a different app or even plain text notes for a month, I every now and then wonder if I want to run FreeBSD on my laptop again, and I’ve switched completely to an Android phone for a week last year.

Of course this is partly because of the ever present nagging of optimizing things. But it’s also because I want to make conscious choices about the things I use and be mindful about the way I consume. Just because I have used a tool for 5 years doesn’t make it the perfect one. Just because I have been doing things in a certain way every day doesn’t mean it’s the best way to do it. The hard part is recognizing which things are even part of a routine. This is why it’s important to me to have a lot of time for reflections. I try to have multiple times a day where I can just think. This is mostly right after I get up in the morning and prepare breakfast, on my commute, when I do the dishes, or in the shower. I don’t reflect on my habits every single time but I do have the time to do so every day. It’s also the reason why I really value vacation days and completely unplug from work when I go on vacation. This forces a timeout of all work related things on me. It’s so easy to take things for granted and assume that’s how it has to be instead of thinking about what you want it to be. I’ve cut down on mailing list memberships, push notifications, and times I check and respond to email just because I was on vacation and had a timeout from it all and when I came back reflected on why I was getting all those notifications and if I really had to.

Taking the time to stop one of your routines or habits for a week or a month and reflecting on whether it actually makes your life better has been a wonderful way of improving my quality of life. It made me cut out things completely that turned out to not bring me the joy I thought they would. It made me find new things that I thoroughly enjoy now. And it made me appreciate the things that continue to be part of my life even more. Because come Monday I will drink coffee again. And it’s gonna be wonderful because I know why I do drink it.

Credibility is a basic survival tool.

I have given a bunch of talks over the last 2 years. Most of them were last year where I felt like I was at a different conference every month. I have given regular conference talks, keynotes, and was on panels. Some of them were as short as 15 minutes with the longest one having been 90 minutes of me talking. The topics range from cultural talks about how we approach things like deployment or blameless postmortems at Etsy to somewhat technical talks about our stack and tools we use and have built. Before 2013 I had never given a public talk, save for some presentations at work and university with a somewhat mixed audience of coworkers or other academics. This means when I had my first talk over 2 years ago, everything was very new to me, I had no idea what I was doing and it took me a long time to get anything done. However I have learned a lot since then and have refined my skills to a level where I enjoy and feel comfortable preparing talks, creating slides, and speaking in front of an audience. And giving talks has been one of the most amazing experiences in my career so far. So I hope this post is useful in helping more people to also improve their presentation skills and encouraging more to also give talks. While you’re reading through this I also want you to always keep in mind that all of those things are very much a function of my character. I identify as an introvert and a highly-sensitive person (I haven’t been tested for it, so I don’t have conclusive proof that I am either one). A lot of things in how I prepare and give talks are to make this experience as good as I can for me as well as my audience. So while some things might work for you as well, I encourage you to take this as inspiration to find your own way.

Finding A Topic

The most important part about finding a topic is to remember that talks are about sharing knowledge and whatever seems obvious to you, is very likely pretty interesting to a lot of people outside of your company.

I used to have a pretty hard time with this as I did think all my work was obvious and nothing interesting. I was fortunate however that the way Etsy does deployment was something still very popular when I started giving talks and still is to this day. And since I also work on deployment tooling as my day job it was a very natural thing to talk about for me. So when I got asked to submit a talk proposal for my first conference, I could take the topic of Continuous Deployment and turn it into a talk that was appropriate for what the organizers were looking for. Since then I have given many variations of that talk. Depending on what a conference was looking for I could put more focus on the cultural aspects of it, the tooling, or how it fits into the bigger picture of software development and collaboration at Etsy.

However coming up with different topics has been a challenge for me. I gave two differently themed talks last year. One was focused on the Etsy monitoring stack and the other one on how we tackle blameless postmortems. What makes them both a little bit special is that they are not really related to an actual project I had at work. They are about existing setups and ongoing work we are doing to improve things. While this is great for getting out of the mindset that all your current work is obvious and nobody would be interested in hearing about it (two things that are basically always false), it can be hard to find your story arc in such a talk. Since you don’t really go from a problem to the analysis and research part, to the implementation and then the solution, you have to find another hook for your audience. In the case of the monitoring talk I chose Etsy’s technical architecture. We are for the most part running a monolithic LAMP application which seems surprising to a lot of people. So this was a good introduction in to the talk. For talking about blameless postmortems I just chose something everybody can relate to: failure. Most people in the audience have seen their stack break under surprising conditions and Etsy is no exception there. So I chose a familiar scenario to talk about how we deal with it.

Something that has helped in the past with finding a topic for me was talking to coworkers that are working on slightly different things and asking them what they think would be interesting to hear a talk on. There is also often an opportunity to follow trends on Twitter to see what kind of problems people are interested in and give a talk on how you tackle that. And if there is something you think is interesting, there’s a high chance others will to. And even if it seems obvious to you, people love to hear about how others tackle problems. So don’t think just because you feel like your work isn’t totally novel that others won’t be interested in it.

Writing The Abstract

When it comes to writing the abstract, there are two things I try to optimize for:

1. Organizers should quickly get an idea if the talk is a fit
2. Attendees should quickly know if they want to see the talk over another one

Writing the abstract for the talk proposal used to be a huge undertaking for me. I wanted to make sure all my ideas are captured and the conference organizers knew what they were getting when they accepted my talk. So I ended up with proposals that were sometimes 2 pages of a fully fleshed out talk (that would change until I give it anyways) and really elaborate. It took me a while to realize that conference organizers get a ton of proposals and they don’t have the time to read a novel of a proposal just to decide whether or not the topic is interesting (if you are a conference organizer and have gotten such a proposal from me: I’m sorry). So I eventually learned to concentrate on the main story arc and message the talk will have. So now when I write an abstract, it’s about 6 sentences to at most 2 paragraphs of text. It contains the title, the super high level outline and what the audience will be able to take away from it. Because the other part of this is that conferences often put the abstract on their schedule page and attendees use it to decide which talk to go to if it’s not a single track conference. So I want them to be able to decide within 30 seconds whether or not my talk sounds interesting. And not bore them with 2 pages of things they might not even be interested in.

Plus focusing on the main idea has a big benefit when writing the outline and the slides for the talk. I can always go back to the abstract and check whether or not my talk actually conveys the message I wanted it to bring across. And if I notice I drift from my original idea, I can correct that easily. There have also been occasions where I changed the story arc and message of the talk slightly as I found one I liked better while preparing the slides. This is ok most of the time, however if it turns into a completely different talk I’d check with the organizers if they are fine with this as well. If not, maybe I have a new proposal for a different conference :).

Prepping The Talk

The time leading up to the conference and preparing the talk is kind of a tricky one for me. I have a process there that works great for me, but which I wouldn’t necessarily recommend for anyone. The main reason for this is that I don’t write anything down until a week or so before the conference and I also don’t do any dry runs. However I want to emphasize that this is not because I think I don’t need all of those things. I’m very convinced my talks would be better if I did them. It’s mostly because of how my brain works and some of my personal anxieties.

Just because I don’t write anything down doesn’t mean I don’t think about the talk. As a matter of fact in the weeks before the conference I’m mostly forming ideas and shaping things in my head which then end up on the slides. I do think a long time about things before taking actions, this is just my nature so my talk prep follows this. Then about a week before I give the talk, I start to write my ideas down as slides. Refining them until (sometimes literally) I go on stage. I have used Keynote for a long time to do this. I sometimes wrote down my ideas as a Markdown outline in vim and then create slides in Keynote from this. However as much as I preferred writing the outline in vim, it being twice the amount of work - as I had to basically do the same thing in Keynote then - lead to me more often than not just starting in Keynote. Then in fall of last year I found Deckset. A wonderful OSX application that lets me write my slides completely in Markdown and then creates a beautiful presentation from them. Since then I have gone back to writing the outline of my talks in vim and then slowly transforming it into slides.

And as I said before, I never do dry runs. That’s not because I don’t think they are a good idea. They are and you should absolutely do them. However for me they never fit into my schedule. Because I work on the slides until right before I have to give the talk, there isn’t a version I’m confident in showing people early enough for dry runs. In addition to that if I prepared my talk sooner so I could do a dry run I would constantly think that I’m not giving my best because I’m not using all the time there is. Plus talking in front of people takes a lot of preparation for me (as you will discover later). So dry runs take a huge amount of energy. That being said it is something I’m not really happy about and want to work on getting better at in the future. There are so many things that other people notice about your talks that I think it is one of the things I’m doing that is keeping my talks from being better.

Slide Design

For the slide design I have come to heavily rely on Deckset to do the right thing. I’m a big fan of having only a simple statement or message on a slide to carry the story of the talk. Even when I was using Keynote I tried to have as few things as possible on each slide. Keynote makes it really easy to go overboard with effects, information, shapes, pictures, movies, bullet points, etc. I had a pretty good slides template that I got from a coworker and that has basically been adapted for almost all Etsy engineering talks by now. This made it pretty fast for me to iterate on slides. I would put the outline headings on a single slide in bold font and then fill in slides in between with content aiming for 1.5 slides per minute. When it comes to slide design I usually choose between either

• a short statement or quote
• a picture (optionally with a statement or title)
• an animated gif

That’s it. Nothing more complicated than that. I sometimes have a short bullet point list but I try to keep that rare. If I can’t say something on a slide within those constraints, I very likely should rethink it or split it across multiple slides. I do use some font styles to emphasize words in a statement that I think should have more focus. But all in all I try to keep it simple. And Deckset makes that a lot easier with its constraints (generated from Markdown, no custom themes, etc) than Keynote. So I actually end up being able to iterate on slides much faster. I spent a lot of time trying to find the right pictures and animated gifs for my talks, often I even switch them out right before I give the talk (more on that later). Usually I look for things that are somewhat humorous and make the talk less dry and more enjoyable. I have a big weakness for pop culture references and so it’s not unlikely that my talk includes references to Gossip Girl, Vampire Diaries, Black Sabbath, Iron Man, MacGyver, or various internet memes. This is also how I ended up giving a talk at the USPTO with the Backstreet Boys and Avril Lavigne being part of my slides. I usually end up having enough slides for my talk (to satisfy the 1.5 slides/minute ratio) by the day before the conference or the day before I leave for the conference.

Travel (optional)

This should probably be its own blog post as there is so much I’ve learned about traveling in the last year. However this only has its own section here as I want to emphasize that I optimize for minimalism and worry free travel when I go to conferences. This means I only have my backpack, which can hold my clothes for at least a week, my laptop with all the cabling, and my Aeropress, grinder, and beans because I love good coffee and don’t want to think about where to get coffee when I’m in a hotel. I plan my travel so I’ll be at the conference on the day before I give my talk. Usually I end up working on the slides more on the plane and thinking about what message I want to bring across with each slide.

The Day Before

When I arrive at the conference, I check into the hotel, make sure to ask where breakfast is served the next day and when and then meet up with the organizers and try to get a feel for the atmosphere of the conference. This is all for me to get acclimated with the new environment and to minimize surprises and things to worry about. I try to find the room I’m going to give the talk in and if there is a talk I really want to see I attend it. But I don’t sweat that too much, if I have the feeling that I don’t yet feel good about my slides or that I will be more calm by hanging out in the hotel room, I will do that instead. After all I’m here for giving a kick-ass talk and I will do everything to make sure this is what’s gonna happen. Either way I try to be in my hotel room at 10pm at the latest. If I’m there sooner, I’ll go over my slides again and make some adaptations based on what atmosphere I picked up from the conference. By 10pm I’m usually exhausted from travel and/or jet lag and will fall asleep.

The Day Of

I get up between 6 and 6.30am, read a book or browse twitter or my RSS feeds to not yet think about the talk. As soon as the hotel breakfast restaurant opens, I’m heading down there to have an extensive and relaxed breakfast. I go that early for two reasons:

1. I want to have as much time as possible to enjoy breakfast.
2. Most likely there aren’t a lot of people around yet, so it’s quiet

After breakfast I go back to my room and go over my slides again. At this point I usually just do some minor changes. But I have also reworked a good chunk of the talk during this before. So there are no rules except that I have to feel good about the talk. I might look for a better picture or gif to support the message of some slides or reorder them a bit to make the flow of the overall talk better. I also make coffee as it’s another thing that makes me calm (oddly enough) and feel good. I keep working on my slides until 90 minutes or so before my talk. Then I try to take my mind of the talk for a bit, shower, listen to some music, get dressed (I always wear my Etsy Engineering t-shirt so I don’t have to think about what to wear) and head to the conference. I always try to be at the conference 30-45 minutes before my talk to acclimate myself with the atmosphere. This is a lot easier if the conference is at the same hotel obviously but the same goes regardless of where it’s at. I then find the room I’m giving the talk in, if nobody is speaking there before I find an organizer or other staff member and get set up. My favourite speaking slots are the first ones in the morning. The conference is still a bit empty, the rooms usually are. I just take 15 minutes or so to get a feeling for the room and watch the people coming in. While attendees are sitting down I try to spot 3-5 people in various locations that are evenly spread out and remember them. Those are going to be the people that I try to make eye contact with during the talk. Then 10 minutes or so before I actually am supposed to go on, I go to the restroom, even if it’s just to wash my hands, to have another couple of minutes of quietness before I’m supposed to talk to a room full of people.

Actually Giving The Talk

Once I’m on stage and have my slides up there and am ready to give the talk. Nothing matters except for the talk. I try to be my most energetic, friendly and enthusiastic self. I emphasize when I think how good or bad something is that we are doing or trying to solve. I might try to make some jokes about certain things that people can likely relate to, like how naming this is hard or how computers sometimes don’t do what you think you told them to do. I remember the 3-5 people from before and while I’m talking I switch between them, trying to make eye contact. Generally I’m really bad about making eye contact even in conversations within small groups. So by adhering to this pattern of looking up people beforehand I can just follow it without worrying whether or not I’m actually capturing the audience enough or staring holes into the air. I have my speaker notes on my laptop (I try to present with my laptop if at all possible) but I only usually have notes for the most crucial things or if I’m not confident I get some things right. English isn’t my first language so if things might get tricky with remembering an important word I write it down. Otherwise I tend to improvise on slides a bit. I generally know what I want to say and not having a strict set of notes tends to make it less dry and more lively for me. I tend to always have the last 10% of talk time open for questions. So once I’m done with my slides I let everyone know that I have time for questions. But also make sure it’s clear that I will be around after the talk and also have my (work) email address on the slides. So if someone in the audience has a question but doesn’t want to ask it in front of a lot of people, there is more than that one setting to ask about my talk.

After The Talk

If possible I try to stay in the room for a couple of minutes so people can come up to me for questions. If not I’ll try to be around the conference somewhere and - although I have the urge to - try not to disappear immediately. I won’t however attend a talk right after as I’m too hyped up and overstimulated from just having spoken to a room full of people. Once people are not coming up to me for questions anymore or it otherwise doesn’t seem like I’m running away from anyone I try to find a quiet corner and check Twitter to see how people reacted to my talk and what things they tweeted from it. This is a good indicator for me which parts resonated with people and which didn’t. I don’t over obsess on this but it’s nice to read about how people liked your talk and gives me a better feeling about all this preparation and over stimulation having been worth it. I then upload my slides, usually have a page written for it on my site which I publish and then try to enjoy the rest of the conference. I tend to only go into hiding for a bit and not go to my hotel room as there is a big chance I won’t come back to the conference. So I only take as much time as I need to be able to recharge and enjoy the conference again.

The Takeaway

Preparing and giving a talk is something I do very thoroughly. I do all those things mentioned here (which may seem like a huge set of preparations) because it helps me be and feel prepared. A lot of the preparation for a talk is psychological for me. As I said in the introduction, I’m very introverted and often have somewhat strong reactions to new and unknown environments and people. So having this framework helps me immensely feeling less overwhelmed.

However it’s worth noting that this is the absolute ideal plan. I try to make it work like that but if any of the things don’t work according to plan it’s not a catastrophe. I’m able to give the talk regardless, this is just the dream set up. It also varies a lot depending on how big the conference is and what kind of talk I’m giving. If it’s an internal lunch talk at work, it’s fairly low stress for me now and I don’t have that much prep time that I need. But mostly because I now have a ton of experience giving talks and it’s less scary than it was 2 years ago.

This year however I have decided to take a break from giving talks as it was just a bit too much last year. I’m looking forward to giving more talks next year and have spent this year so far helping others to give talks by connecting them to conferences I have spoken at, acting as a sounding board for talk ideas, giving feedback on abstracts and proposals and answer as many questions as I can about the process and nature of giving a conference talk. Learning how to give talks and giving them until I felt pretty comfortable doing it has been a great experience and definitely one of the most amazing things I get to do as part of my job.

Thanks to Bethany Macri and Lara Hogan for reading drafts of this and giving feedback

Technology carries with it all the biases of the people who make it, so simply making the world more technical was not going to save us.

Perhaps one reason that women writers and technologists have, so far, the calmest and most comprehensive understanding of what surveillance technology really does to the human condition is that women grow up being watched.

the traits that managers found most problematic in programmers were those stereotypically associated with men

The Basics

In order to make writing for me as low barrier as possible it has to work with the tools I use every day. For me this mostly means vim and git. I try to really embrace plain text for everything and store those files in git - even for my accounting data. For quite a long time I just used my regular (terminal) vim setup that I had configured for writing code to also write prose. And it worked remarkably well. However it also always had this programming feel to me with all the important status bar information that was actually way more distracting than useful when not in a coding mode. It’s also a lot harder and uncomfortable to focus on only writing with default left aligned layout in vim. I would open vim in full screen to not have any distractions and my text would always hang on the left third of the terminal.

I was looking for something like a write room style setup. Fortunately a lot of other people have also had that problem and there’s a huge variety of distraction-less writing plugins for vim now. After some searching I’ve settled on vim-zenroom2. It’s based on Goyo and gives me a beautifully decluttered vim session where I can focus on only the text I want to write and nothing else. I spent some time trying to get the vim status bar to display the word count of the current buffer. But it was always kind of janky. So I decided to not care that much and just run :!wc - w % when I really want to know how much I’ve written. And as I also exist inside of a tmux session most of the time, I also disable the status bar there when in writing mode. I rely heavily on the terminal bell changing the status bar for notifications and I don’t want to be interrupted by that when I’m writing.

I do all my writing in Markdown. I really like that it’s basically plain text with a little bit of markup and super close to HTML. That makes it easy to just write and more importantly read without having to constantly render and preview it. I actually write out all of my drafts before looking at the rendered version most of the time. A helpful setting in vim there is to enable spell checking for Markdown files. In my vimrc I have autocmd FileType markdown setlocal spell set, so that every time I open a Markdown file I get spell correction automatically. This is especially helpful as I do most of my writing not in my first language. So vim tells me immediately when I’ve written the German spelling for an English word and lets me correct it.

When I want to focus on writing I also tend to only work on my 11" MacBook Air and not connect it to the Thunderbolt Display. The bigger screen tends to distract me more than it helps. However I do have an iTerm profile with a bigger font for writing. The default for coding, IRC, email, etc for me is 11pt. And I switch to 18pt for writing.

Blogging

For blogging I use jekyll. I switched to it 6 years ago when I wanted to have a blog again and it works pretty well most of the time. Rumour has it that I’m constantly trying to replace it with a simple Makefile but that may or may not be true. The repo is open source on GitHub but I host the actual site on my own server.

I used to have all my drafts in the jekyll _drafts folder, as it made the most sense to have it all in one place. Whenever I had an idea for a new blog post I would create a file in there with the yaml frontmatter, set the title and published to false and jot down some bullet points. However since I didn’t want the blog posts I may or may not write be in the git repo on GitHub, I never committed them to git. Which started to annoy me as I didn’t really have a commit log, backup, etc for my drafts. I also had the whole blog cloned into my owncloud folder, which meant that it would always try to sync all the jekyll files and git objects although it wasn’t really necessary as I only really cared about having my drafts everywhere for easy access. So I decided to move my drafts into a separate repo, that is just pushed to one of my servers. From there I can just clone it wherever I want, edit, and push it back up.

The way I create drafts in there is still the same. I add a file with the yaml frontmatter (I actually have a vim-stencil template for it). And then add some bullet points, headlines and ideas I have about that blog post in there. Basically a small outline which I will use to evolve the post. I will then most likely not touch it for quite a while. The way I write most blog posts is that I have the draft with some notes in the repo for weeks or months and every now and then think about it and come up with some new things to add and new ways to phrase ideas. Then at some point I just sit down and write it all down. So my writing flow really happens in spurts.

Once I’m happy with the draft, I move it over to the jekyll blog, make some minor adjustments so it looks good in HTML and publish it by rsyncing the generated _site folder to my server. I then git rm the file from the drafts repo and add the URL where it’s published in the commit message.

Journaling

Another outlet for my writing is journaling. However it’s a lot more infrequent and random for me. The setup is kinda similar, although I don’t use jekyll for it. I basically have a journal git repo that holds a file current.md where I write down my thoughts for the day. At the end of the month I move the file to an archive format in the form of YYYY/MM.md and touch the file again to now hold the entries for the current month. This means I have a setup in the repo that holds a folder per year and a markdown file per month in each folder. There is also a Makefile to generate a single HTML file from all entries to make it nicer to browse.

I try to write something into my journal every day but it’s been one of the harder habits to keep up. Similarly to how I have a hard time taking notes in meetings, I’m a person who thinks a lot about things. But since it all takes place in my brain I never remember to actually write it down. In order to make it more low barrier to write journal entries I have created a specific iTerm profile just for that. It opens a terminal in a new window, and immediately runs /usr/local/bin/vim -c Goyo /path/to/current.md instead of a shell. Thus I get a vim session with my journal immediately. I also have an Alfred workflow to open iTerm profiles directly. So I can open Alfred, type itp journal and get the iTerm session for writing. This makes it a lot easier and low barrier to journaling.

When I’m done with the entry, I commit it to git and immediately push. This way I have my journal available wherever I want it but especially on my phone to jot things down on the go.

Mobile

I don’t write a ton on my phone but it’s nonetheless a crucial part of my writing flow. I used to have all my writing things in my ownCloud folder together with my notes and synced to my iPhone via Notebooks and webdav. However this has gotten a bit more tedious than I want it to be. Webdav is not super fast and if there is a merge conflict, I don’t get all the niceties of git to resolve it. Plus I don’t need it to check all the files when I really just want to open my journal and add an entry.

So with the move to separate git repos for my drafts and journal I also started to base my mobile flow more on git. I use Working Copy on the iPhone to get access to all my git repos. It’s a really great mobile git client and for journaling I just open the current.md file in the app directly and commit and push it back up. The editor is pretty decent and more than apt for quickly jotting things down.

For writing longer, actual blog posts I’ve come to really like Editorial for iOS. It’s an extremely nice plain text editor with a great Markdown mode. And since both Working Copy and Editorial support the iOS share extensions, I can open a file from Working Copy in Editorial and then save it back to Working Copy for committing the changes to git when I’m done. I’m not gonna ditch the laptop for it anytime soon, but it’s a great way to write when I suddenly am in the mood but don’t want to get up and get my laptop. And I even wrote the first third of this blog post completely on the iPhone.

Future Improvements

I’m really happy with my setup, it works great for me and is based on the tools I know and trust. It’s a very frictionless setup that makes it easy for me to write down my thoughts and have them available for look up and further editing.

The things I definitely want to improve on is to write more. I have a lot of things on my mind and whenever I write things down, it makes things a lot clearer for me. But I have to remind myself to actually do it. One thing I want to establish is more of a writing routine. And even if it’s just a couple of hundred words every day in the journal. Because writing is a ton of fun and it’s one of the things I really really enjoy.

I had no idea about the concept of highly sensitive people until I read this article. It has a pretty click-baity headline but it really hit home for me. So I decided to learn more about it and this book was the most prominent resource to pop up in my search. It’s a really good book with a lot of great psychological insights and explicit case studies. At times the way high sensitivity was described was a bit too feel-good for my taste. At other times I would almost throw my kindle across the room as the author managed to really sneak up on and hijack my sensitivity. The book focuses a lot on what usually goes wrong during childhood for highly sensitive people and makes it a point to relive memories and traumas through the lense of high sensitivity. This is a practice I really enjoyed although it felt a bit much to me at times as I consider my childhood to have been a happy one. On the other hand I started to do this practice with every day situations at work to help me understand why I feel what I feel in different situations. I identify myself as a highly sensitive person and the book was an extremely good read to help me understand better what this could mean for me and my days.

I’ve heard about this book ever since it was released and a lot of people I know speak very highly of it. And they weren’t wrong, I basically devoured the book in a weekend. It’s very well written and has a ton of actionable advice for engineers becoming managers. But I would argue that this description really limits the value of the book. I have no intention to become a manager at the moment however the book was really interesting and helpful for me. I think it’s a great read for anyone looking to grow more into a leadership position.

It takes willpower to switch off the world, even for an hour.

After I was done reading it, I decided to try out what I’ve learned and see if I can make my blog faster. So I installed YSlow and ran it on an example blog post. I don’t often have a lot of images in my posts. It’s mostly text and maybe some inline code snippets. So I decided that this was a good representative post for testing my improvements. And the YSlow results weren’t great. In my current setup loading this simple page resulted in the following YSlow result:

12 HTTP requests and a total weight of 157.2K bytes with empty cache

At least I now had a baseline to compare my changes to. The book has a very good chapter on how to cleanup CSS and how it can affect page weight and page load times. So I started there. I didn’t really have good structure in my CSS as it has basically organically grown since 2009. And whenever I wanted to change something or try out something new, I just added the CSS for it on top of the existing one. I’ve rarely gone back and cleaned things up. This lead to me having 3 CSS files being loaded for every page. The main style declarations aptly named style.css, a CSS file which contained definitions for all the code highlighting I’m using on my site pygments.css and a third file from when I started playing with fonts that would declare font-faces and such and was named - you guessed it - fonts.css. When I added those files it made sense to me to have them separate. They were taking care of different things, contained no duplicate code, I was basically treating them like includes in a programming language. So I took a look at the files and found a lot of old clutter. I was still loading two fonts which I hadn’t used in forever. And I also had recently reworked my header but was still loading the font that gave me icons for popular social media sites which I had previously used for linking. Those together were already around 75kB and 3 HTTP requests for nothing. So I removed the fonts I wasn’t using, cleaned up some other CSS that I had found that was unused and combined the 3 style files I had into 1. And that already gave me a huge jump in optimization. Just by removing things that I didn’t need and combining CSS files I went to:

7 HTTP requests and a total weight of 65.4K bytes with empty cache

I was already amazed and excited and of course now I wanted to do more. So I installed a CSS minifier and decided to load only minified CSS. This gave me another 15 kB in improvements:

7 HTTP requests and a total weight of 50.0K bytes with empty cache

Another thing that is mentioned in the book as a common way to improve performance is to load images for the size you actually want to display them at and not bigger. I didn’t want to go through all the images I had in some random blog posts but still do something for my baseline performance. And since I load my avatar from Gravatar into the header on every load, I looked at that and saw that I was requesting the image in size 142x142 (Gravatar gives you the handy URL parameter ?s=142 for that) but was only displaying it in 100x100. So I changed the parameter to 100 and squeezed a couple more kilobytes out of my site:

7 HTTP requests and a total weight of 47.7K bytes with empty cache

Now I was kind of at a stopping point. I had collected all the low hanging fruit. My page was only a third as big as it was before and I was down 5 HTTP requests that didn’t have to be done anymore. This was already huge for me. Looking at YSlow now told me that I was spending another HTTP request for loading JavaScript from Twitter to embed their social media buttons. And that this also added about 35 kB to my page weight. At this point about 75% of my total page weight. Of course as a first course of action, I tweeted about it and thought about whether or not I really need or want the share buttons on there. Fortunately Corey responded to my tweet with essentially what I was thinking. That I don’t really need those buttons for anything. So I removed the buttons, bringing my total YSlow results to:

6 HTTP requests and a total weight of 12.4K bytes with empty cache

And I was blown away when I actually saw the number. Just by following some simple advice from the book, I was able to cut the number of HTTP requests in half. And slim down my page weight to ~8% of what it was before.

Learning new things is fun

Optimizing my site has been a lot of fun. Of course if you are an experienced frontend engineer, all of those things are not surprising and probably the first thing you would try. But not working in that field every day, I was surprised by the impact such a cleanup and restructuring can make on even a small site like mine. I had tried to dabble a little bit in getting better at frontend engineering and performance before. However I had never had a good introduction to give me a place to start. Learning new things can be overwhelming and especially if you don’t know where to start, everything can feel like it’s probably wrong. Lara’s book gave me a great introduction into all the different topics of frontend performance and if you are interested in this as well, I can highly recommend it.

For now I might not be able to squeeze out more performance just from reducing the page weight. I’ve since installed mod_deflate and mod_expire on my web server to improve transfer size and caching for my site. However I still feel like structuring HTML and CSS in a good and fast performing way is something I don’t have a good grasp on. And it’s definitely something my site can benefit from, if even for more clarity next time I want to change anything. So this might be the next thing I’ll tackle in learning more about frontend engineering.

The largest hurdle to creating and maintaining stellar site performance is the culture of your organization.

Feminism is a movement to end sexism, sexist exploitation, and oppression.

A while ago I wanted to find a better way to keep on top of my finances. They aren’t crazy in any way and I have a very normal, regular, non-exceptional financial situation as an employed engineer. However I felt like I could get more data and better insights into everything. This wasn’t really on my mind for a while though. I kept the idea in the back of my head but wasn’t actively looking into any way to make it happen. Then one day I was reading my RSS feeds where I amongst others subscribe to the wonderful “The Setup” blog. If you don’t know about it, it’s basically an interview series where people talk about the tools (hardware and software) they use to get their job done. I like reading about the tools others use and get inspired to try out different things. And in there I was reading an interview with a Debian developer who mentioned working a lot with text files and git. And he also said he was doing his accounting with git and ledger and I was immediately intrigued.

I started to investigate the tool and read blog posts about how others were using it. Ledger actually has a very comprehensive documentation, so I started there. Read about how to use it, the basics of double-entry bookkeeping, and what kind of information I could get out of it. I then also found an interesting post where someone wrote a tool - reckon - which parses CSV and formats it into ledger format. It even uses Bayesian machine learning to suggest accounts to use for each entry, minimizing the work that needs to be done manually even further.

Diving into the deep end

So I decided to give it a try and take the upcoming tax return I had to file as the motivation to get it done before that. I downloaded CSV data from my bank accounts (and learned that the allowed time to back is 2 years, no data before that), installed ledger via homebrew and the reckon rubygem and started to import data. This was a bit tedious at first, as reckon didn’t support backspacing and thus editing mistyped accounts. I fixed that in the gem and sent a pull request like a good open source citizen and procrastinating software engineer. And after a couple of hours I had all my data from 2014 and (most of) 2013 in the ledger data format. I played around with the reporting options and really liked it. It was super flexible I could quickly fix and change things by opening the file in vim. So I decided to properly structure it and go full in with ledger.

All in

I created a git repo with directories for the raw csv files (in case I needed to regenerate any data at some point or look up something else) and for each year since 2013. In those per-year directories I have a file for checking, credit card, cash and opening balances. And a top level ledger dat file per year that includes the appropriate files.

cassie:accounting[master]% cat 2015.dat
include 2015/opening_balances.dat
include 2015/checking.dat
include 2015/credit_card.dat
include 2015/cash.dat

cassie:accounting[master]% ls -1 2015
cash.dat
checking.dat
credit_card.dat
opening_balances.dat

Checking and credit card should be self explanatory as they just hold the entries for those accounts. Whenever I withdraw money at an ATM, I book it to an Expenses:Cash account as I expect to spend that money. Otherwise I wouldn’t have withdrawn it. But this also means that I don’t have a ton of visibilty into what I spend cash on. That is why I have a file called cash.dat. When I spend cash on something and remember, I note it down on my phone in a text file which syncs to my computer. And when I’m doing my monthly accounting I can pull up this file and just write proper ledger data entries for the contents of this file. I then note that those expenses come from the account Expenses:Cash to keep everything correct. The next file special file is opening_balances.dat. Because I have a file per year, the data only reflects postings for that year. In order to still get accurate balances, I run the equity command (ledger -f ledger-old.dat equity) on the old year’s data and write that into the opening balances file as coming from the account Equity:OpeningBalances. This is a bit of hack, but it illustrates a major advantage of ledger. It doesn’t care about what the accounts are named. This means you can give them names that mean something to you and ledger won’t have a problem with that. The documentation even gives an example for tracking inventory in the video game EverQuest.

Monthly Routine

Now with this setup in place, I have a monthly recurring project in my OmniFocus to download the CSV for my account and add them to my ledger data. Once I’ve downloaded the file, I run reckon over it to have it properly format the data and suggest accounts to add them to. Since reckon - as I mentioned before - uses Bayesian learning to find out what accounts a posting likely belongs to, it makes sense to have a corpus for it to learn from which includes all possible accounts. And because I really like Makefiles, I have one with a simple task in there to generate a big file which contains all of my postings:

SOURCES := $(shell find . -iname "*.dat*" -mindepth 2)

corpus.dat: $(SOURCES)
	cat $(SOURCES) > $@

Now I run reckon with something like

reckon -f raw_data/2015/checking05.csv --contains-header -o 201505_checking.dat -l corpus.dat

in order to parse the CSV file. Usually when I run this, reckon detects almost all of my recurrent transactions like rent, gas, electricity, internet, subway and ferry fare. I just have to confirm the account by hitting enter. With this it takes me about 5-10 minutes to get through a CSV file. After I’m done I might go through the file in vim to make some adaptions. For example I have a very generic account named “Expenses:Amazon” which reckon detects to use for everything coming from Amazon. However since I buy a variety of things (household items, clothes, etc) on Amazon I open my “past orders” page on amazon.com and check my transactions in ledger against it and file them into more specific accounts. When I’m happy with it, I commit my changes to git and have an up to date version of my accounting data. I then can run all the queries on it to give me some overview of what I was spending money on in the last month. Ledger is versatile enough that I could spend a lot of time on explaining all the possibilities. But the simplest way to get started is just showing the top level balances which will give you an overview about Income, Expenses and Assets (if you have named your accounts like that):

ledger -f 2015.dat --period-sort "(amount)" balance -M --begin 2015/04/01 --end 2015/05/01 --depth=1

Verdict

I’m really happy with this setup so far. It’s comparably low tech and low investment and I can (for the most part) use the tools I know and I don’t have to store my financial data on someone elses server. Still if I want to do accounting on a different machine, it’s just a matter of cloning the git repo to it and installing ledger. The import process is not too cumbersome, although I have to remember when I pay off my credit card from my checking account for example that I only enter the transaction once as it will show up in both CSV downloads (once as debit and once as credit). This has caused some confusion for me in the past when I forgot but generally isn’t too bad.

I’m also not doing any super advanced things with it so far. I’ve played with it’s Gnuplot suppport and ran different queries in different situations to track down where I actually spent more money than the month before. I’m sure there are more use cases that will arise over time and while I’m no accountant (and probably used some words wrong on this post) it has been super interesting to get some more structure and insight into my personal finances.

You had the project management all along

Every time you start to write code, even for some fun side project, you start to think about the different components that will make up the whole thing. You start to form a mental model of all the high level parts that comprise the finished application. You plan out some rough course of action for yourself. Which parts you want to tackle first. Which things to stub out and which things to punt on for later. You just (most of the time) don’t write them down. But as you go on, you think about what the first workable version will be able to do. Then you think about the next one, maybe refactor some things to accomodate new features. Maybe you write down some notes for you or add some //TODO: lines in the code so you know what to do when you come back to it. But the important part here is, that you’re planning the application. You’re basically already doing 85% of what project management for a small to medium software project is all about.

So what’s the difference? Well really mostly thinking about the structure of your project on a higher level and writing things down. To be clear: It’s ok if you only want to write code. And you can be or get really good at it. However only wanting to write code is like saying you only want to hammer wood together. Sure there is beauty in how you do it. There are good carpenters out there who are able to do woodwork like nobody else. And everybody wants to have such a craftsperson on their team when it comes down to doing the work. However this will only be useful up to a certain level. And that’s absolutely ok. But if you want to level up as a carpenter you will need to understand what it is that you’re actually building.

And that is very much the same for a software engineer. In my mind being a senior engineer also means that you understand the problems you are solving past the perfect implementation of a binary sort. That you understand that writing code is a means to an end and not the ultimate purpose. That you understand what problem you are solving and the environment it will exist in so you plan for it accordingly. This means a good chunk of “non-coding time”. It means that you understand how to break the problem apart and how it would speed up the implementation if you suddenly got 2 or 3 more engineers on the project. Or if it wouldn’t help at all. It means that you understand how the project could continue if you suddenly decided to go on vacation because you know you don’t want to be a spof. It means that you have things planned out so someone else could take it over or even make the whole project happen without you. It means understanding what existing or future work would be great for a more junior engineer on the team to level up on and plan work so it’s possible for them to do it. And it means writing things down and communicating them.

This can be as easy as creating a project in your JIRA instance and adding a bunch of subtasks. It can be Gantt charts if you are so inclined and want to show dependencies better. It can be a markdown document laying out all the bits and pieces you have thought about so far.

You are not your project

All of these things might feel weird at the beginning. All you want to do is write code, find the perfect abstraction, make it beautiful. You will suck at this at first because you are not used to it. But at the same time you will suddenly see others implementing things you want to exist, doing work for you and learning while they do it. And maybe even take a whole project over from you and finish it. And it will feel weird again. You will have this feeling of not having finished something. Of only going 80% there. Of only having done the “soft” parts. But in reality you just transferred a ton of knowledge. You made it possible for someone else to work on something that previously only existed in your head. You are not your projects. I’ve said before that you need to capture ideas for others to work on. It’s the only way to scale yourself. And it frees up your time to work on other things. And even if you end up working on the project all by yourself (which is less and less likely as your organization grows), there will be a plan for others who are interested to follow along with what’s going on. There is clear communication of what’s in progress and what the current state of things is. And other engineers can learn from your example. Because suddenly you’re doing project management. And it’s not even that weird. You have been doing it on some level all along. And you should, it’s part of your job. As an engineer you understand best how work gets done. So you are in the perfect position to plan out the structure of your projects. And there is no such thing as “no project management” anyways. You can only decide to do it badly or try to do it well. And seeing all the benefits of doing it well come to life is so much more fun.

How I use it

The arrival of the notebooks coincided with the first Monday of February for me. So that Monday morning I sat down to start to fill out my yearly, monthly and weekly goals. The notebooks had actually arrived on the weekend already. However other than looking at them quickly I didn’t touch them but spent the weekend researching the intended use of the notebooks on the popforms site. I read about how to set the yearly theme, create goals for the month and how to plan your week. This first review probably took a whole hour or so for me because I had to do the whole thing. But it was actually a pretty good exercise and a great way to reflect a bit more about the things I have on my todo list. Since I’ve written my original blog post about OmniFocus, I’ve added a couple of useful perspectives and one of them is for planning my weeks and days. It’s basically modeled after this blog post and shows all not on-hold or blocked projects from which I then pick things to work on every day. This perspective was the basis for my monthly and weekly planning with the spark notebooks. It reflects all my duties and obligations. For this first review after I set my yearly goals, planned my month to be in line with those goals (in an ideal world your work projects align with your yearly goals but in reality it’s a healthy mix of things you have to do and things that bring you closer to your long term goals) and then planned my weekly tasks according to my montly plan I went on to time blocking. I grabbed my calendar and put all the meetings in the time blocking view of the spark notebook. I then went ahead and filled all slots (or most of them) where I didn’t have meetings with blocks for things I wanted to get done from my weekly goals. And after this hour or two of planning I went into my week. The following weekly and monthly reviews were much quicker as I didn’t have to do everything from scratch again. To the point where my weekly reviews take me between 15 and 30 minutes right now.

A somewhat big surprise was however that it turns out I can’t plan my whole week with time blocks in advance. I did that for the first two and every time interruptions and unplanned work and meetings destroyed my carefully layed out plan. So now I timeblock the first 2 or 3 days of the week on Monday and then do it again on Wednesday for the rest of the week. And while doing that I literally put those blocks into my calendar as well so people see that I’m working on something during that time and that no meetings should go there if possible. I also stopped putting meetings in the spark notebook outline as it was somewhat a double maintenance of the same thing. So the calendar for me holds the daily outline of meetings and blocks of time where I wanna do work. And the notebook tells me how I have planned to spend those time blocks. This has worked pretty well for me so far and I definitely have more structured time than I used to.

Another great tool in the notebook are the meeting notes templates. They give you a structured form in which to write your notes down. Although I have some problems with this feature (see below) I’ve really come to like them for facilitating PostMortems. They are great for taking notes during the reconstruction of the timeline and the “Follow Up/Nest Steps” box is perfect for jotting down remediation items before transferring them to Jira. I don’t use the “Main points” box as it doesn’t make a ton of sense for PostMortems and I still rarely take notes in other meetings (something I definitely want to get better at).

Balancing it with OmniFocus

A point I was curious about when I ordered the notebook was how this could be balanced with my OmniFocus setup. I rely heavily on OmniFocus and everything that needs to get done has a place in there. And I really didn’t want to end doing duplicate work. So for the first week or two it felt kinda weird. I was a bit confused about where to look for what to work on next and which things to track where and in which granularity. I started out with a pretty detailed time blocking setup in the notebook and followed that one closely. But that turned out to be too much. I found my natural balance there to put the higher level tasks/projects in the time blocks and then have the break down of those in OmniFocus. This means I can have the overview of what I am supposed to work on in a given time slot in the notebook and then just open OmniFocus to see what the next important part is for that project. Thus I still do my weekly review in OmniFocus and then go on to plan the week in the spark notebook picking things from the OmniFocus “Plan” perspective.

The natural balance there is that the spark notebook gives me the higher level overview of my plans that I was always missing in OmniFocus. It’s a really great software for structuring lists, todos and break projects down into smaller things. But I have never figured out a great way to plan the higher level in there. And that’s where the spark notebook fits in perfectly for me. So when I start to work in a time blocked slot, I check the notebook what I have noted down to work on during that time and then check OmniFocus for the next action on that project. This sounds a bit more tedious than it really is, often enough I remember what I allocated the time block for and don’t need to open the notebook and even if I do, it’s not a ton of overhead.

Short comings and things that don’t work for me (so far)

After using the spark notebook for a couple of weeks now I have found a couple of things that unfortunately didn’t work that well for me or that I was missing. None of them are a dealbreaker and I wasn’t expecting everything to make perfect sense, since it’s probably impossibly to make something work for so many people with different work and life styles.

The first thing that I was missing was more bookmarks. The spark notebook comes with 2 bookmarks to quickly find pages. I’m using one to mark my weekly time block table and the other one to mark the position of where I left off with meeting notes. However I would love to have at least two more to be able to quickly find the monthly overview and the project pages. Maybe even three to also quickly find the scribble pages at the end. And I would love it if the spark notebook also had a Moleskine style pocket in the back to put smaller cards and notes in.

Speaking of the meeting notes, I’m really enjoying the layout and it’s extremely good for taking structured meeting notes. However the fact that there is only a limited number of them means that I’m constantly trying to decide if a meeting is important enough to “waste” a meeting page if there is a chance that I’m not taking notes at all (did I mention that I’m a poor notetaker?). Especially with the anxiety to run out of meetings notes in my spark notebook and then having to bring a second notebook to meetings. I don’t think this is something that can be fixed within the spark notebook but I have to be ok with.

I have also yet to find a use case for the project planning pages and the scribble/free use pages at the end. My problem with the project planning notes is that it’s easier for me to keep that part in OmniFocus (at least the task breakdown, I could definitely benefit from doing a more formal write up of the goals and notes write up) and easier to find as I don’t have a bookmark left over to use for the project pages. Kind of the same goes for the free use pages at the end for me. I haven’t really used them yet as I can’t bookmark them and I’m afraid of running out.

The 30 day challenge is something I’m really torn on. I love the idea but it’s been really hard for me to follow. I’m not sure if it’s because it’s a bit out of sight in the weekly plan (maybe I should account time for it in the time blocking more) or something else. But something I want to try there is to always mention the challenge as another weekly goal in the plan and see if that helps.

As I mentioned above, I also had to adapt my time blocking routine to do it twice a week as planning all time blocks for the week in advance doesn’t work for me. I have so far also written down 6-7 goals per week as this seems to be the number of things I can actually find the time to work on. I don’t always finish all of them and I might try to break them down more in the future.

Verdict

While I haven’t yet fully come to use all the features the spark notebook provides, it’s really been a great addition to my planning tools. Especially since I’ve always missed a higher level planning overview with OmniFocus, this is where the notebook fits in great for me. It gives me a good sense of the higher level things I need to do and whether or not I have allocated time for it appropriately on a weekly level. I use both tools to plan work and non-work related things and the combination of the spark notebook and OmniFocus has definitely become crucial to my productivity.

Of course the overall application has a lot of features. And they keep growing and changing as we figure out how a growing engineering team is using it. As we have remediation items coming from PostMortems. And as more teams need and add more deployment stacks for slightly different applications. The original version of deployinator had an execution model where all commands where executed in a streaming manner within an HTTP request. That meant we had to configure the correct output buffering, had a long running request doing the work and generally a somewhat confusing scenario where we often weren’t sure what would happen when you close your laptop lid in the middle of a deploy. We also started to run into problems where the deploy would often break with SSH broken pipe errors (all commands are run over ssh) in the middle of a run. We tracked it down to an oddity in TCP behaviour between modern versions of OSX and Linux. And we decided that it was time to move the deploy run out of the HTTP request. We thought about different ways of doing that and prototyped a couple of things. And then one day while working on one of the prototypes of the new deployment model I took a step back and realized that I was basically trying to reimplement OS process management in Ruby. And this was not what I wanted Deployinator to be. Deployinator is UNIX. So a deployment is now done by forking into a separate process, setting the process title to the stack and stage name and letting it run. ps, kill, nice all still work. If you need to log into the deployment server and figure things out, you can still use the tools you use every day. The rest of Deployinator also always has been very UNIX inspired. The deployment process runs commands over ssh and distributes commands to multiple machines via dsh. All deployment output is written to a log file. The log file is tailed by a websocket server to present it back to the web application. The log in the web app shows all output of what the shell commands are doing. If the commands write to STDERR, Deployinator shows it in red and bubbles it up to a separate error log. This means you can write your deployment commands in the well known UNIX style. Infos go to STDOUT, errors go to STDERR. In addition Deployinator also comes with a command line tool to kick off any deploy without needing a working web server.

And in my opinion this is how it should be. The actual steps of how your software gets deployed will always be a little different. You might run a Rails app instead of a PHP application. You might have a compiled binary that needs to be shipped or you will have to restart services. You might git pull on the servers directly instead of rsyncing files over. But there is always the operating system as the common denominator (or almost always). And by using that foundation in your tooling you already have a common ground when it comes to understanding and debugging what your deployment system does. And there are a lot of existing tools, like rsync, git or ssh which you can reuse and leverage. There is also a great response by Corey about how the GitHub deployment system works. And the final paragraph there is really what I love most about it:

I think people would be underwhelmed by the technology and implementation though. It’s just a bit of ruby, UNIX, and HTTP. It’s not pushing the boundaries of computing, it just chugs along doing its job so we don’t have to.

And it really doesn’t have to be complicated. Deployment can start with a simple shell script. And then you can wrap it in a web frontend. Or an IRC command. Or an iPhone app. But at its core it’s still manipulating files and putting them on a computer. Deployment is still UNIX.

The pattern of devaluing women’s contribution is as old as human civilization

I have so many opinions on that and I hope the following brain dump makes sense/helps.

I should start off with the disclaimer that I think there is something to that quote but it’s been completely repurposed to serve as a backwards justification for a lot of things. I think the variant “If it’s a good idea, go ahead and do it. It is much easier to apologize than it is to get permission.” is a much better version (although still problematic) of this quote. But I highly dislike the often quoted form together with its corollary “move fast and break things” which basically has the vast majority of the same problems.

So why do I dislike it? First and for all because it’s arrogant and disrespectful. It has the implication that rules don’t apply for some (for a certain value of some) people and that it is in their judgement to decide what applies to them. It also implicitly means, because (usually though not always) rules are made to protect/help people, that what you want to do is more important than protecting/helping. I’m probably extremely biased because I work in infrastructure where a large part of work is maintenance. But in computering what following this rule most likely means is hack something together that works and then figure out how to maintain it and who. And in that regard it often comes down to upholding the romantic VC notion of a 10X engineer/lone wolf programmer who is so genius that you have to get everything out of their way because they can change the game in an instant. That they don’t have to communicate, follow rules, or workflows, because their beautiful mind justifies everything. Another thing I highly dislike.

The next problem I have with this statement is that it is so ambivalent that it doesn’t really mean anything. And as so often, can only be verified in hindsight. The article you linked really pin pointed one of the major problems there. In order to be granted forgiveness, you are betting on “the authority” (this could be your manager, execs or tech leads, or even the police) to turn a blind eye on something you did or even praise you for breaking the “law” for making things better. This usually is only the case if you are a member of the same race, culture, class, group as those you will have to ask for forgiveness. Which doesn’t work well for everybody. It’s also important here, that - if I’m not mistaken - the quote comes from a time where Grace Hopper was almost retired and already an accomplished Rear Admiral. The power and influence that comes with such a rank shouldn’t be neglected. And I highly doubt that the same thing worked when she was a Sea(wo)man or Petty Officer.

That being said there is something to that quote. But as I said in the beginning, I see it more in the context of the variant quote. And more importantly in the context of efficiency thoroughness trade-off. A lot of times you can’t ask everyone for permission to do something because it takes too much time and doesn’t make sense. Especially when it comes to computering there is often a lot of merit in trying to get a prototype in place so there is a concrete thing to talk about. It’s also often worth it to only bounce ideas off of a handful of people before trying it out instead of getting a formal review and the exec’s agreement to do it. But with all of this the impact if it’s a bad idea has to be taken into account. If everybody runs off doing their weird ideas, we likely would have chaos. At the same time if everybody spends their day with getting permissions about work, there won’t be any work getting done. Ideally we trust our colleagues that they know what is needed to bring things forward. That is why there are always tendencies to reduce bureaucracy and empower individuals. But I don’t think this means you should do things where you have to ask for forgiveness. Because if you have to, you likely made someone else’s day pretty miserable.

I hope this was a somewhat coherent write up and answers at least some of the questions you had. Also I’m super happy to be proven wrong here since this is very likely a pretty narrow view on things.

Building a plant

The general setup of a plant is most often pretty structured on a high level. You have a process that someone invented at some point. You burn sand or combine acid to create a new product that has properties which are desirable. Then you probably spent a lot of time improving and refining the process and making the actual plant work better (this is where the vast amount of IP and patents come from). And at some point you end up with a pretty good and structured plan how to repeat it all over the world. Of course the process refinements never stop but they get a little smaller over time.

Then at some point the time comes to build a new one, maybe at a new location maybe at an old one. But you as the engineer (or project manager at that point) get told that a new plant is needed. You likely have done that many times before. So you’re grabbing the documentation, maybe a project plan template for it and you start planning. This is really where the first important parts come together. A plant is build by tens or hundreds of people all in all. But at this stage you have to make sure you’re getting the right people on the project for each component of the plant. Obviously there will have to be some buildings. So you go to Facility Management and ask for an engineer to work with you on the project. You don’t have to know how to construct buildings yourself, but it’s an important part of the plant so you get an engineer you can delegate the work to. Of course a dark, empty building isn’t that helpful. We are gonna put a ton of machinery in there. And all of it needs power. So you’re gonna get an electrical engineer on the project to plan out the electrical infrastructure and hook the plant up to power. Someone also has to actually install all that machinery, make sure you’re choosing the right parts and devices, work on the actual construction. This means, you gotta get an engineer to supervise that. Once the plant is running there are a lot of things that constantly need regulation. Water flow, temperature, air flow, all those things need to be controlled and automated. You’ll get an automation, measurement and control engineer to take care of this of course. And in the end this is also about a chemical process, so a chemical engineer will also be on the project. With all these engineers on board, it’s your job to keep them all on the same page. You have to set up meetings, establish communication and make sure everyone is included on updates since changes might influence their area of work.

This is only an exemplary run down of what goes into building a plant. Depending on what kind of engineer you are and the size and complexity of the plant you might be doing one those things yourself. But for a big project you might also just be the technical project lead and delegate all of those things to others.

Building infrastructure

So let’s talk about the other kind of infrastructure projects. Where computers are involved. Because they aren’t that different. Sure they are often much cheaper, you rarely get to work on a giant multi million dollar project. It is also much easier and cheaper to experiment if you’re writing software. You can experiment with things before the whole project is done, change directions much more quickly and generally see intermittent results with less hassle. And you usually don’t have to manage tens of people and contractor companies on the project. On the other side you are usually trying something new, so you don’t have the security of having done this before multiple times. You have to come up with a lot of things for the first time as you are trying to solve problems with this new piece of infrastructure (this is why it might also be a good idea to write down a spec for your project, to give you a better picture of what parts are actually involved).

But the big thing both types of projects have in common is the collaboration and delegation part. You most likely won’t have to deal with facility management, construction and maybe not even electrical power in your project. But there are still a lot of parts. The software you plan to write or the service you want to introduce is probably gonna run on some form of computer. And although it has become way easier to spin up a virtual machine or commission some hardware, you probably want to at least talk to an ops engineer about it. There could be different classes of machines to choose from that have proven to work better for different workloads. The requirements for hardware (virtualized or real) could change if the scope or implementation details of the project change. It could be that new machines are automatically added to the monitoring system unless you tell it not to. And speaking of monitoring: there should be some. And generally there is a lot of knowledge in Ops about monitoring things. You also want to automate the setup for your new service, write some Chef or Puppet to make it smooth to create new instances. Which is probably another thing your ops team can help with. So ask for an ops engineer to be on your project. Some infrastructure services also need to persist data. Maybe you need a cool new database, although likely there is already an existing place where you can put data. In any way this is something where an Ops engineer (or DBA) can help with as well. As soon as you access and write data, call to other services, maybe need some form of authentication or have a naive implementation where you shell out in your PHP code you should have a security engineer on board to make sure it doesn’t end badly. Even if you don’t think you are, having someone from security on board for the project or maybe even just review code changes a lot. Suddenly it’s not that weird service anymore that someone put into production. You get code review from a very different angle and have the good feeling of everything being alright. Of course your new thing also has a ton of tests. Unit tests are the simple thing here, as you can probably hook them up to your existing CI infrastructure. However there might be integration and end-to-end tests that you want to create to increase confidence in changes when someone else (and also yourself) works on the project. So when you start of the project, include someone from your testing/QA/CI infrastructure team. You want to increase confidence in changes with your new piece of infrastructure and engineers working close to testing can definitely help. And speaking of confidence, there should be a way of making things work in development as close as possible to how they are in production. There should be a way to run it on a VM, on your laptop or some other way where you can be sure changes you make in development work in production. If you have a team that takes care of those things, get an engineer from that team involved. Make sure they know that you care about things working in development.

Recently, with the increased focus on cross team collaboration, there is a big chance that you can do a lot of those things yourself. And you should, it’s a great way to learn (although that doesn’t mean you shouldn’t talk to the engineers that work in those areas every day). But you can’t do everything yourself, so depending on the scale of the project and your familiarity with all those areas, there will have to be delegation. And not all projects touch all of those things. You might not have data to persist. Or you already have an internal auth solution that you can plug in. However when in doubt it’s always better to err on the side of caution and include more people or ask them if they think the project would benefit from it. The important part is to take the time to talk to people and let them know you want their input on the project or you want them to also work on the project. And then create a mailing list, or a forum group or an IRC channel or however you handle communication in your organization and invite all those engineers to join and make it clear that project communication will happen there. You want them to always know where they can get information about the project, its progress and an up-to-date status in case they want to check if the project touches things they work on. And not have to be afraid they might miss things.

Because you’re building a plant. Kind of.

Thanks to Ben Hughes for reading drafts of this, giving me feedback and being generally rad.

Don’t be. While being on-call can be challenging, it is also very rewarding.

This is why I wanted to share my personal development and experience about learning to be on-call. Because fundamentally I am convinced that it’s not only a necessary but also a rewarding thing to do. When I started at Etsy a bit over 3 years ago I had never worked anywhere where I had to be on-call. I was either too junior (engineers in training didn’t have to be on-call) or I was working on research and actual shipped software where there were no production systems to look after. And when I applied for jobs I even actively sought out job positions where I didn’t have to be on-call a lot as I saw it as a tedious, annoying and scary thing that I didn’t want to do.

How I got into this

Then when I joined Etsy every engineer had to sign up for a week of general developer on-call per year (almost all of our on-call rotations are a week long). This is the general on-call rotation for anything relating to the web app of etsy.com where the knowledge of how things work an ops engineer usually has doesn’t suffice. We have amazingly good ops engineers who can fix a ton of things themselves and if you have a really bad on-call week you get paged once as a developer. Back then we were small (or big) enough so that we could cover the year with every one being on-call at most once. That wasn’t too bad and so I put my week in about 6 months after I started so I could learn some things about the web application before actually being on-call. Since my team works on developer tooling and not the website itself I chose this quite long period of time before my first on-call just because I wouldn’t have that much exposure to the app in my regular workday. Lucky me, I happened to choose the week that included the weekend where we wanted to upgrade our main SPOF database. So needless to say, I was really scared of the on-call. I didn’t have enough knowledge about the architecture to know where things could break or debug it and I had never been on-call before so even having to pay attention to whether or not I had cell reception all the time was already making me nervous. I also hadn’t been at Etsy long enough to know how often I would get paged, how serious/time-sensitive a page would usually be, whether or not I could be underground for 30 minutes to take the subway home, which action should I use to tell PagerDuty that I’m working on the incident, and what even would happen when I get paged and what to do next. Most of this was just me being nervous. We had a lot of documenation about being on-call and a lot of people to talk to. And I stayed up all night during the database upgrade and didn’t even get paged a single time. The whole thing was planned really well and all the people who knew how the site worked were already online and it turned out to be an invaluable learning experience for me.

Getting used to the game

Then after my first year or so we changed the on-call schedule a bit. We had more engineers than we needed to fill the dev on-call rotation. We started to have more specialized teams who would take on their own on-call and were thus excluded from the generic one. So we switched to a system where you could volunteer to be on-call every 4 weeks or so for a week. We set it up to be two tiered so if you’ve never been on-call, you would be the L1 contact who gets paged first. If you really can’t get to a computer or have no clue what to do, you can escalate to the L2 who is more senior and has experience with being on-call. And then after 6 months you were released from the rotation and a new round started where engineers could volunteer. I signed up immediately when this got introduced. I wanted to learn more about what it means to be on-call. The rotation was still super quiet and I almost never got paged. But just the fact that I would now have to bring a charged phone, a Mifi and a laptop with me wherever I went every 4 weeks made on-call less of a scary exception but a scheduled routine. This took away a lot of my fears. I learned a lot about how noisy of a rotation to expect, how to plan my day so that I would have cell reception all the time, and how to react when I got paged. I still felt awkward being on-call because I still almost never really knew the parts involved when I got paged. I would work as a communication broker to call in the right people but I could hardly ever fix things myself. Again I learned a ton from watching the people I called debug and fix problems, especially in parts of the app I had never touched before. But after a while it was also somewhat unsatisfying to almost never get paged and then if a page happened, not being able to actually do something about it.

Level up: The Ops Rotation

In addition to that, by then I had worked on mostly infrastructure things. I worked on a lot of Chef recipes, upgraded all of etsy.com to a new release of PHP and reworked core parts of how software deployment worked. I touched all the things that would wake up an ops engineer (but thankfully never did) and I wanted to own up to the responsibility. But you only ever hear horror stories about ops on-call from almost anyone. Because once you have an automated system being able to wake you up at night (95% of Nagios alerts used to go to the Etsy ops rotation back then) it changes on-call a lot. So to own up to me having changed one of the biggest parts of our stack, I decided to sign up for shadowing the ops on-call engineer for a week. I got my phone hooked into Nagios. And it got real. I had added my email a couple of months earlier already to get a feel for the alert volume and to know whether I broke things I was working on but adding your phone is a different kind of real. I chose a week where one of our senior ops engineers would be on-call and I got woken up for everything for a week. It was pretty brutal. I got woken up in the middle of the night and had no idea what Nagios wanted from me. I would log onto IRC and get some information from Marcus what the alert was about and how to fix it. I mostly felt helpless and super slow with debugging what might be causing the production issues at hand. The week ended with a couple of sleepless nights and a site outage on Friday. And I can’t even begin to explain how much I learned during that week. About systems I’ve never seen before, about database replication, about hating disk space alerts, and how to work after being deprived of sleep because of a busy on-call night. And the learning was what got me hooked. A couple of months later I signed up for the ops on-call rotation for good and have been part of it for over a year now.

Long Story Short

I can honestly say leaning into on-call has been one of the best things I did for growing as an engineer. It put me way out of my comfort zone and I had to overcome a good chunk of impostor syndrome and fear for it. I write software and design systems with a different view now. I have way more experience with all the tools we use to manage our infrastructure - having to find something in Chef at 3am really helps learning your way around it - and a better intuition when it comes to adding things to it. When we added an on-call rotation to my team I was super relaxed because I knew it wouldn’t be anything that I hadn’t dealt with before in the ops rotation. It’s a great feeling to know that you are doing your part to keep things running and you can bond with a lot of other people over sharing on-call pain and how to make things better. It’s not always fun. In fact it’s fitting that I’m writing this blog post slightly sleep deprived, while I’m on-call, have been woken up almost every night since Friday, had to manage to find someone to cover for me while I’m traveling for a whole day and had to hook up notifications and mobile internet in another country for the first 2 days of on-call. However I’m always happy when I’m done with another on-call week and can look back on the things I learned. After all it’s the challenges through which we grow.

So without further ado, here is my list:

Germaine Greer - The Female Eunuch

I started the year off with finally finishing Germaine Greer’s feminist classic from 1970 about the role of women in modern society. I had known about the book for a couple of years and after having read Bell Hooks’ The Will to Change: Men, Masculinity, and Love last year I decided to finally read it. I definitely enjoyed it. Especially as a man it opens your eyes to a lot of things you never encounter in your daily life. It’s very graphic at times and there are some long-winded parts in the middle but I would definitely recommend it to anyone who’s interested in feminism. I also started reading her newest book “The Whole Woman” this year which is the sequel she never wanted to write. And so far I like it and it’s alarming how few things have changed since “The Female Eunuch”.

Erik Hollnagel - The ETTO Principle: Efficiency-Thoroughness Trade-Off

It’s no secret that I’m interested in human factors and system safety and how to apply lessons learned to our field of creating and managing complex computer systems. So it also shouldn’t be a surprise that this book really hit home for me. It’s well written and touches on a myriad of different aspects about how we trade off thoroughness for efficiency and how production pressure changes our way of making decisions. It’s a pretty fast read and I really enjoyed it. It also has a huge references and related literature section following each chapter which makes it great to start diving deeper into the topic.

The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win

I originally started reading it in 2013 when it came out and I enjoyed it back then. But I somehow still dropped the ball and stopped reading it. I finally went back and finished it this summer. It’s a good and interesting read and having worked in traditional plant production companies I liked a lot of the parallels in there. It gets a little weird at the end and the last quarter feels like the authors really had to wrap up the book. And no matter how you look at it, it’s definitely business romanticism. But if you don’t mind that, it’s definitely entertaining.

Susan Cain - Quiet: The Power of Introverts in a World That Can’t Stop Talking

This is another one I started in 2013 and then dropped for no real reason. I finished it this year and thoroughly enjoyed it. I knew before I started reading that I fall on the introvert side of the scale but the book really helped recognizing some more patterns and making me feel better about it. This is also the only book I finished as a Kindle audiobook and while I likely won’t do it again, it was an interesting experience. It’s a great read and definitely recommended for anyone who works with other humans in their daily life.

Kourosh Dini - Create Flow with OmniFocus 2

This one was kind of a surprise read for me. I’ve written before about how much I have OmniFocus is integrated in my life. And when this book popped up in one of my RSS feeds I decided to give it a read. It’s definitely not a cheap book and I jumped over the first half as it’s basically an introduction into OmniFocus which I already know how to use. The book isn’t a total game changer, but the latter half gives some good food for thought on how to make the most of OmniFocus’ Perspectives and some unusual use cases for it.

Bonus: Jon Cowie - Customizing Chef

I added this as a bonus round, because while I definitely read it, I had the privilege to do so as a reviewer. I’m really happy that Jon asked me to review his book and while I had done a lot of Chef before, I learned tons about its internals from this book. If you work with Chef and want to get more out of it or even just understand some of the internals a little better, definitely read this book.

I really enjoyed reading all those books this year. And one of my New Year’s resolutions is definitely to read more next year. I’ve planned to set some time aside every day to read and hope to have a longer list of things I read next year. If not, it sure isn’t because of a small Kindle backlog.

1. There’s always more work that you could do
2. There are always more things to read, watch or catch up on
3. Most notifications don’t really need to interrupt you

The Workday

And I started to make some changes from there. The very first one was strict working hours. At the beginning of the year I decided that I will go home at 6.30pm every day. Unless something is really on fire. It really helped that I take the ferry to and from work everyday. So I settled for a ferry schedule that I wanted to take and stuck with it. There are no excuses. All the work I’m doing in the evening will still be there the next day. And there is no work, work email or any such things after I got home. In order to further this more I also deleted any work related email and calendar accounts from my personal laptop a couple of weeks ago. If I want to get more work done I have to get up and start work earlier. In reality this usually plays out to me being in the office around 10am or 10:30 with 30 to 45 minutes of working from home before that. Usually I check email and try to flag things I want to get done that day in my OmniFocus at home as it’s more quiet, earlier in the day, and less busy. Obviously there are exceptions to this, as I said already when things are on fire, when I’m on-call of course, or when I’m really in the zone and don’t want to stop (although this is really really rare after 5:30pm to be honest). On regular days I stick to my working hours.

Reading things

The next step was embracing the fact that there is always more to read. I’m pretty happy with my e-mail setup but it took some time to be ok with heavy filtering and only checking it occasionally. And in addition to not having any sort of notifications for my work email I have also turned off icon badges in the iPhone mail client (except for VIP mails from friends and family). The number of unread emails doesn’t really mean anything other than that it causes you to keep checking it because it’s a pattern of “todo” items. And you really want to get this number down. For no particular reason other than that it feels good to cross things off. So it’s easy to get into the habit of keep doing it. I realized that the icon badge doesn’t actually mean anything for me as all the emails I need to answer are in my OmniFocus so there is no need for badges on my email client. I also turned off unread count badges for basically everything else, but most notably RSS feeds and articles I’ve saved for later. Reading things shouldn’t be a chore but something you enjoy when you have the time. I always felt bad that I have so many things pile up in my different accounts. And it ended up being a constant hassle of cleaning up the lists in there, instead of enjoying the things I can read from it. So I stopped feeling bad about having an insane backlog of articles in my queue and now see it more as a big pool of interesting things to read when I have the time (thank you Mike for this).

Notifications and the phone

So that leaves notifications. You have probably realized by now that every app on your phone competes for your attention. And even more so with notifications. Every single app wants to be able to push notifications to your phone. Even if it’s just a game that wants to remind you what you’re missing while you’re not playing. So while realizing that a ton of things actually don’t need my attention, I started to divide the things on my phone into 3 groups. The first one is allowed to push notifications, make sounds and interrupt me and they are basically only the phone and messages app for SMS. The second group is stuff that I care about enough to allow notifications but isn’t urgent. Twitter clients, Foursquare and Pushover (which I use to tell me about IRC mentions when I’m idle) notifications fall into that. Whenever I have time I’ll skim through the notifications on the lock screen on my phone but nothing in that list is allowed to make a sound or vibrate the phone. And all the other apps on my phone don’t get to push notifications at all. My notification setup has also much improved since I got a Pebble. While it’s not crucial, it makes checking notifications a matter of a handful of seconds by flicking my wrist versus checking my phone. The downside is that when it comes to vibration it’s all or nothing on the Pebble. Right now it doesn’t bother me much but is definitely something I’m watching out for in case it gets annoying. And in addition to cleaning up notifications I also cleaned up my iPhone’s home screen while I was at it. I only have things on there that I actually (want to) use every day. That way my phone looks and feels way less cluttered (and it’s way more fun to have wallpapers).

What else?

Changing those simple habits has made a big impact in my life. I’m definitely more exhausted when I get home because I’m trying to get as much done as possible during the day. But at the same time I’m super excited to get back to work the next morning. My phone doesn’t make a noise except for really important things, so although I have work email and calendars on there, when I just have it in my pocket or on a table it doesn’t remind me of work things after work (I disconnect the Pebble when I get home as I don’t have a need for it there). I actually feel less stressed out about my phone and interruptions all day and get more out of the things I actually want to do on my phone.

There are still some things I want to improve though. I definitely don’t spend as much time reading as I want to. Something I want to try there is take a page from my friend Mathias’ book and get 30 minutes of reading in every morning before I head into the office. At the beginning of the year I have also started to keep a work journal to jot down things that happened during the day and that I worked on. And I would love to expand that to also include non-work related things.

I’m also notoriously bad about taking vacation days. I usually end up with a 2 to 3 week long vacation at the end of the year because I need to use up all my days. I really want to get back to taking a longer time off in the middle of the year to enjoy the summer and spread more days off over the year.

What did you do to improve your work-life balance this year? Email me, tweet me or better take some time and write a blog post about it!

But I think there is a very undervalued part of that whole situation. A lot of operations engineers (at least in my part of the internet) will happily claim they can’t write code or write shitty code (by the way if you still think you don’t write code as an ops engineer go read Christopher Webber’s great blog post about it). When in reality ops engineers write a ton of code and I have seen and reviewed amazing apps and tools being knocked out by people who will happily make excuses for their code whenever you talk about the awesome thing they wrote. The big advantage of working together is exchanging knowledge that was previously in siloed domains. It’s great that developers carry pagers and know about deployment and how to write Chef recipes or Puppet modules. But that also means as an operations engineer you can tap into a giant stream of software engineering knowledge that developers have learned, improved and cared about for years. And one of those is code review.

Code review is a great way to get free learning and feedback about the things you are working on. It’s a way to get someone with a different context to think about whether your solution to a problem (and that’s what basically all programming, scripting and automation is) makes sense to someone else. It’s also a way to learn about paradigms, conventions and unknown techniques to make things better. I’m a software engineer by trade and whenever I start with something new, work on a new project or try to solve a new problem I try to seek out code review. I recently worked on our Android app. And while I have worked with Java and written Android code before, it’s been years and was in a completely different code base. So while I was writing down the code that would solve the problem I was having, it was very non-idiomatic when it came to our Android coding style. Once I was done I asked my coworker Hannah if she could take a look at my code. And she was super excited about it and immediately jumped on it. And she gave me tons of feedback. She showed me the app loader structure that would make my code much more DRY and I learned a lot about how we approach Android development. And even though she initially wasn’t completely familiar with the problem I was solving and it definitely took a bit longer for me to actually deploy the code I learned a ton.

One of the keys here was definitely that she was so excited to review code for me. And this is where the reviewer comes in. Being asked for a code review means the person values your opinion and would love to get your feedback on something that is arguably important to them and trusts you to be able to improve it. It also means you now have to balance the fine line between not actually giving useful feedback and overdoing the code review and effectively blocking their progress. E.g. while it likely doesn’t make sense to suggest an abstract factory pattern for a procedural script of python code, showing where it could greatly benefit from using functions is a simple way to improve things. Fundamentally how to give good code reviews is a complicated topic which I don’t want to get into here. But the important part is that this is something you should be excited about and should let the person asking for the review know that you are.

Often code reviews are still seen as a slow down, annoying and nothing that really adds value as you already know your code works. However seeing code reviews as an opportunity to learn, get another perspective and ultimatively a way of sharing knowledge is in my opinion way more accurate when done right (right being used lax here as everyone has to define for themselves what that means). It is a great way to learn about different parts of your infrastructure’s code, new programming paradigms and methods, how to communicate more effectively and in the end leads to a more resilient organization and enjoyable work and programming environment.

Setting the stage

Before I start I want to make very clear that I’m likely not a power email user and what works for me might not work for you. This is also mostly about how I manage my work email, as my personal email is low volume enough so that it’s probably not interesting. I also use a somewhat esoteric e-mail client and while most of the things I’ll talk about are generic, using mutt makes it a lot easier for me. And to give you a ballpark number for e-mail volume: I receive about 370 e-mails per day - your mileage may vary.

So how do I use e-mail?

The first two very important factors for me are filtering and Inbox Zero. I am subscribed to a ton of mailing lists at work. Everything I deem interesting (and I’m a super nosy person) I subscribe to, but I have strict rules about what goes into my inbox:

• E-mails addressed directly to me
• My team’s mailing list
• Mailing lists of teams I work closely with
• Low volume mailing lists (less than 5 messages a day)
• Important automated e-mails (e.g. from Nagios)

Everything else gets filtered into separate mailboxes. No exceptions. If a low volume list gets more busy it gets filtered.

With this setup I check my inbox a couple of times a day. The frequency depends on how busy I am obviously, but I check e-mail at most every 30 minutes and at least 5 times a day. And everything that is filtered I check once a day up until once a week, depending on how important the mailing list is. I also clear out all mailboxes at least once a week and archive all e-mail in there. This usually takes about 5 - 20 minutes and I do it before doing my weekly GTD review.

In mutt I also use the solarized light theme (as I do almost everywhere else) which helps a lot as e-mails are color coded differently. Read e-mails are gray, e-mails addressed to me directly or via cc are green and messages from mailing lists are blue. That way I can open up mutt, take a quick glance to see if I have new important email or if I can postpone going through my email. This is sometimes so fast that my terminal emulator warns me about the shell being closed too fast again and there might be something wrong, which I still find hilarious. When I actually go through my e-mail, I file them into mailboxes depending on whether I want to read them later or already know that they need an answer and archive everything else. Following the GTD principle if I can answer the e-mail in 2 minutes I do it right away. Otherwise I move it to the corresponding mailbox from which it gets pulled into my Omnifocus inbox. All of this is done with simple keyboard shortcuts that work on single or multiple messages.

I also have e-mail set up on my iPhone with the iOS built-in mail client via IMAP. However I only ever skim e-mail on there and at most answer if it takes me less than 2 minutes. I also have the labeling enabled that tells me whether a messages was sent to me directly or via cc or just because I’m part of a mailing list. That way I can also check very quickly if there is something in there that potentially needs my attention.

The important part

The most important part to take away from this is that you don’t need to read all e-mail you get. Especially in a big company there are enough things going on that you can’t possibly keep up with everything. And that’s ok. In addition to that what really works for me is using trusted clients that have been around for a while. I have been using mutt for years and I have most of its shortcuts in my muscle memory. When I came to work after a week of vacation and had 6000 emails in my inbox it took me 10 seconds to clear out all the automated emails based on their from addresses and cut the number of messages by 95%. The learning curve for mutt was pretty steep at the beginning but it has payed off over the years and since it’s open source I know it will be around and not suddenly disappear (or at least it’s unlikely). I also love being able to write my e-mail in vim and am a big fan of plain text e-mails. On my phone I also use the built-in e-mail client as it’s likely to stay and not completely disappear or get shut down either. My usage on the phone is also light enough so I don’t care about small changes in UX or functionality with OS upgrades. I have yet to encounter a bad surprise after upgrading my phone.

When it comes to dealing with stress and the feeling of being overwhelmed with e-mail, the biggest change for me besides filtering was to turn off all notifications. No e-mail that I receive will ever make a sound or make my phone vibrate. There are no lock screen notifications on my phone besides e-mail from people in my iPhone VIP list which is mostly family and even then it just shows up. No sound, no vibration. I decide when I have time to read e-mail.

As I said, most of these things are applicable no matter what e-mail client you use. I happen to use mutt (set up similar to how it’s explained here and these are my config files in case you’re interested), but there are a ton of good and proven clients out there (I used OSX Mail.app for years and always liked it). And plain old IMAP is honestly pretty cool. But most of all - in my opinion - the biggest problems with e-mail are social or rather psychological problems (trying to keep up with everything, wanting to get notified all the time) and not technological ones, and they can be solved.

So something needed to change. I had tried running several times, but since I find it utterly boring I never really got into the habit of doing it regularly. Plus there is no real instant feedback from running, so even when I managed to go, I never had the feeling of actually doing sports. Clearly I needed a way to track progress. So the first step was to get a scale. That way I would be able to at least track my weight. I decided to get the Fitbit Aria scale as I liked the idea of syncing it to an account where I can get pretty graphs. I like graphs. The next step then was to track how active I am and what I eat so that I could get a rough overview of general activity and calorie burn. As I had already created an account with Fitbit for the scale I decided to get the Fitbit flex wristband (I later replaced it with the now discontinued Force) to track steps and calories.

Now I had feedback and graphs for what I was doing all day, how active I was, how long/good I was sleeping and I kept track of what I was eating. Having this incentive meant that I would be going running or shooting some hoops for at least 20 minutes everyday before or after work. I also stopped eating everything I found (which seems to be big part of the secret of losing weight) as it would go into my food journal in the Fitbit application. And this worked really well. Over the course of a couple of months I lost about 11 kilos (about 24 lbs) and even had to put new holes in my belt.

But soon after all of this my habit of how I used the Fitbit changed. In early 2014 I decided that I didn’t want to work long hours anymore and improve my work-life balance. This meant I took the conscious decision that I would leave work at around 6pm everyday to catch the ferry home. No more staying late unless things are on fire and no more working from home after I left the office. If I wanted to get more stuff done I’d have to get up early. While the getting up early part didn’t work very well at the beginning, this now meant that I would go to bed at a regular time (usually between 10-11pm) and get about 9-10h of sleep most nights. This however made my Fitbit sleep tracking basically obsolete for me. I knew that I slept well most nights and if I didn’t it was mostly because I violated the rule and went to bed late (or I was on-call and got woken up at night). In addition to that I got a Pebble and was now rocking dual wearables. Which really wasn’t a pleasant feeling. Both the Fitbit and the Pebble aren’t super big but having stuff hanging on your wrist all day meant that I would come home and take both of because it felt much better not to have anything on the wrists. And that feeling amplified when I went to bed. I felt much more relaxed and less restricted when I wouldn’t wear anything on my wrists when I was sleeping. So most days I would take off the Fitbit (and Pebble) when I got home and not put it back on until the next morning (with the exception of being on-call where we use sleep data to improve the on-call rotation). I still liked having graphs about steps, but mostly for the sake of having graphs. I didn’t act on them in any way other than occasionally bragging on twitter about how much I walked.

So when I came home from a week of vacation in early July I decided to not put the Fitbit back on at all and see how it feels. And it was great! I felt much more free and less restricted. I hadn’t been using the data it collected for anything really in almost 6 months. Plus I never really felt comfortable with the fact that details about my activity and calorie intake live on a server in the cloud. Thanks (partly) to the Fitbit I got back to a good intuition about how much I should eat and how much sport I should do every week. I now go to the gym regularly, have a way better sleep schedule and eat more consciously and more importantly less than 1.5 years ago (although there are definitely improvements to make there still). And if anything is off about food, sports or sleep I notice immediately as I start to feel unwell. This doesn’t mean I would never ever use a fitness tracker again. If they eventually end up being less intrusive in daily life and maybe even come with a collection application I can install on my own servers I would happily try it again. But for now it’s “Bye Bye Fitbit”.

So I started to look into how people mirror their repositories for backups, speed, availability and other things. There exist quite a lot of solutions out there which are mostly written in Ruby or Python. While this is fine and I would encourage you to look into those, I didn’t want to deal with installing pip to install some Python script or installing yet another gem just for something that can be accomplished with a couple of lines of shell. So I wrote my own set of scripts in Bourne shell (one of the default installed shells in FreeBSD) so I could just cron them up on my backup box.

First I needed a way to get a list of all my repositories. Thankfully GitHub has a pretty great API so I can just get a list of all my repositories and their git clone URLs:

#!/bin/sh

# Usage:
# github_repo_list.sh mrtazz [34345k34j3k4b2jk3]
#
# get a list of all public repos for a user
if [ -z $1 ]; then
  echo "Usage:"
  echo "github_repo_list.sh USERNAME [TOKEN]"
  exit 1
fi

if [ ! -z $2 ]; then
  TOKEN="&access_token=${2}"
fi

CURL=$(which curl)
if [ -z ${CURL} ]; then
  # fall back to /usr/local/bin/curl
  CURL="/usr/local/bin/curl"
fi

BASEURL="https://api.github.com/users/${1}/repos?type=owner${TOKEN}"
count=1

while [ ${count} -gt 0 ]; do

  lines=$(${CURL} "${BASEURL}&page=${count}" -s | grep git_url | cut -d" " -f6 | sed -e "s/[\",]//g")

  # stop if we don't get any more content. A bit hacky but I don't want to
  # parse HTTP header data to figure out the last page
  if [ "${lines}" == "" ]; then
    count=0
  else
    for line in ${lines}; do echo ${line} ; done
    count=`expr $count + 1`
  fi

done

This script takes a username and an optional access token and retrieves the public list of repositories for that user. It then outputs the git clone URLs one per line so it’s easily stored in a text file or fed into other scripts. There are some minor inefficiencies and missing features in there as it curls one more time than needed to the GitHub API to figure out if there are more results and it also only supports public repositories as I don’t have private ones at the moment. However changing the URL to call if I ever want to mirror private repositories is relatively easy and I don’t care that much about the extra curl as this script is not gonna be run very frequently.

This now gives me a list of all repositories on my account I want to mirror. The next step is actually mirroring them. For that I wrote a script that looks like this:

#!/bin/sh

# take a list of git clone urls on STDIN and clone them if they don't exist.

if [ -z $1 ]; then
  echo "Usage:"
  echo "github_repo_sync.sh directory"
  exit 1
fi

GIT=$(which git)

if [ -z ${GIT} ]; then
  # if git is not in path fall back to /usr/local
  if [ -f /usr/local/bin/git ]; then
    GIT="/usr/local/bin/git"
  else
    echo "You need to have git installed."
    exit 1
  fi
fi

# switch to archive directory
cd $1

while read line; do
  directory=$(echo "${line}" | cut -d "/" -f 5)

  if [ ! -d ${directory} ]; then
    ${GIT} clone --mirror ${line}
  else
    cd ${directory}
    ${GIT} fetch -p origin
    cd ..
  fi

done

This script checks for each entry in a list of git clone URLs passed in via STDIN and if the directory already exists it fetches changes and if not clones it into the given directory. The mirroring commands reflect the instructions in this GitHub guide.

Now to tie those two together I just set up two cron entries to run those two commands:

0 20 * * * ~/bin/github_repo_list.sh mrtazz 0f6 > /backup/github/github_repo_list.txt
0 21 * * * ~/bin/github_repo_sync.sh /backup/github < /backup/github/github_repo_list.txt

The first cron entry fetches the list of repositories and sticks them into a text file. The second one runs an hour later and actually syncs all the changes. I set it up to sync into the zfs pool that gets snapshotted every night anyways (as described here) so I get that for free. I’m not super happy with running this on a cron as there could be a smarter solution that checks for changes via the API and marks repositories as dirty, but this is the simplest thing that could work and way less work than interacting more with the API. In addition I would love to exclude forks from the backup since I don’t really care about backing those up. But I’ll leave this for iteration 2.

I track changes to the script in my bin folder repository on GitHub, so if you’re interested in tracking changes to this setup, follow it there.

But isn’t this just a cheap way of getting off the hook?

This is why I decided to write down my thoughts about why this isn’t the case and what the New View is about relating to responsibility and trying to prevent the same incident from happening again.

First let’s look into what we are working with every day. Be it in airtraffic control and flying airplanes, operating modern trains, working in a hospital and taking care of patients or keeping a website running. All of those are complex socio-technical systems. That means the systems as a whole consist of many many technological parts and humans as operators interact with them. And they are big and complex enough as so they are in itself intractable for any person. This means at no point is there a simple and clear plan to follow and at no point is it possible for anyone to fully describe the system end to end with all of its interactions. This means anyone working within the system has to choose carefully between checking every single step for any risk imaginable and actually getting work done (something that Erik Hollnagel calls ETTO or Efficiency-Thoroughness-Trade-Off). For example an airplane pilot might speed up going through the pre take-off checklisting because being extremely thorough almost certainly means introducing delays or maybe even missing the plane’s flight slot. A doctor maybe only goes through the part of the patient report that is relevant to the immediate action or surgery they are about to do because of the huge number of patients they have to take care of. A software engineer wants to make something faster to provide a better experience for the user and subsequently brings down the site by exhausting available resources too fast.

Now these very specific examples might seem like people slacking off and if they just did all those things according to the rules and regulations, everything would be fine and nothing can go wrong. And conversely if we fire the person that caused that deviation from the rules we have a perfectly simple reason why our complex system broke. This is a very natural approach to accident investigation. Even Nietzsche talked about it before.

In the search for a cause of an accident we do tend to stop, in the words of Nietzsche, by ‘the first interpretation that explains the unknown in familiar terms’ and ‘to use the feeling of pleasure … as our criterion for truth.’

— Erik Hollnagel, The ETTO Principle: Efficiency-Thoroughness Trade-Off (p. 10)

The reality is however that in complex, intractable systems it’s impossible to follow all the rules and attend to work with 100% thoroughness. There is even a behaviour called “work-to-rule” that describes the action of working exactly what the rules describe and thus causing a slowdown that can even come close to a stop.

So now that we have established that people take shortcuts and thoroughness tradeoffs all the time, we can also safely assume - as those actions are likely what makes sense at the time - that other operators (would) do the same. Bringing us to a point where certain actions that just before seemed like the cause of trouble are now considered to be a natural behaviour of people working within the system. And as Sidney Dekker puts it so aptly:

Indeed, as soon as you have reason to believe that any other practitioner would have done the same thing as the one whose assessments and actions are now controversial, you should start looking at the system.

— Sidney Dekker, The Field Guide to Understanding Human Error (p. 195)

But is that really true? Maybe all the others who would have done this very thing a dozen times before just had better judgement? Maybe they just were more aware of the situation and noticed that it would be an appropriate response. Whereas in the failure case the operator just failed to recognize that now was not the time to do this. It turns out smart people have thought about this very thing before. The Austrian physicist and philosopher Ernst Mach came to this conclusion in 1905:

Erkenntnis und Irrtum fließen aus denselben psychischen Quellen; nur der Erfolg vermag beide zu scheiden. Der klar erkannte Irrtum ist als Korrektiv ebenso erkenntnisfördend wie die positive Erkenntnis.

— Ernst Mach, Erkenntnis und Irrtum (p. 116)

This translates to something like “knowledge and error flow from the same mental sources; only success can tell one from the other. A clearly recognized error as a corrective is fostering knowledge as much as positive realization.”. Which makes it clear that the decision of whether or not something was “the right thing to do” or an error is defined by post-hoc analysis of the situation. An advantage the operator didn’t have in the situation.

So now what? Our precious theory about the bad apple is gone. Where do we go from there? Are we not allowed to talk about human actions at all anymore?

Quite the opposite.

Humans are a crucial part of socio-technical systems. But they are what makes systems safe. As we have said before, our complex systems are in large parts intractable and thus there is no way we could design a ruleset of things that we could have a machine execute and everything would be safe. The thousands of little adjustments, human operators carry out every minute are the pillars of our system safety. With the little crux that they sometimes also lead to adversarial outcomes. And this is the part we are interested in. How does something that is done over and over again seemingly suddenly lead to an incident? Why does it make sense for a person in that situation to act in the way they did? After all the basic assumption is that people don’t come to work to do a bad job.

There is a difference between explaining and excusing human performance.

— Sidney Dekker, The Field Guide to Understanding Human Error (p. 196)

So does the human operator in this New View get off the hook? The answer here is no. Because thinking about failure and outages in this way means the practitioner was never on the hook for explaining their behaviour in the first place. However being part of an incident on the very sharp end of the situation brings some new responsibilities with it. It means the human is now the specialist with most of the knowledge about how the system surprised us and broke down. They know best how they expected the system to react and what it actually did. They are the foremost authority on what detections they utilized and what to put in place to realize faster that something is going wrong. They know which tools they reached for, which they had to improvise, and which tools they were missing. This means they are very much on the hook. But on the hook for helping to find ways to make the system safer going forward.

If this has sparked your interest in the field, my coworker Ian has also recently published a set of resources on the Etsy Engineering blog to get started with the topic of System Safety, Human Error and Just Culture.

In the coding world VIM provides a great built-in functionality for that which is called “skeleton files’. This is a great way to always have a good to go version of C source or header files, Makefiles or RPM spec files. However this is all based on filetypes (or rather file endings) and since I write most of my notes and all my blog posts in Markdown for example and they all have the same file ending this doesn’t help me much for having different templates. So I started to look around for VIM functionality or plugins that would just let me load templates from a specific location and maybe expand some variables (as I for example like to have the date auto inserted into meeting notes). I didn’t want a full fledged templating engine, although I could certainly have installed and wrapped the Mustache implementation written in VimL to do that for me. But I wanted to keep it simple and apparently that solution didn’t exist yet.

This is why I wrote a VIM plugin called vim-stencil. It’s a handful of lines of VimL and it does exactly 2 things:

• Load a template from a specified location
• Expand some variables (currently even only one: the date)

So now with a simple call to :Stencil in VIM I can choose a template for the type of file I’m editing (yes it supports tab completion) and load that into my buffer. I even get the current date for free in templates where I choose to have it. No fuzz, no complicated setup. But a small thing that increases my productivity a lot.

However last week I decided that I wanted to have them all be in a layout similar to my website so the whole page doesn’t change just because you click on a link on my projects page. But I also wanted to keep the code for the pages in the respective repo so it’s all in one place while at the same time I didn’t want to copy the layout into each repository.

Thankfully there is trick you can use with GitHub Pages. If you add git submodules to your repository they are gettiing pulled in automatically on page build. So I created a shared repository to hold the template I wanted for my projects. And now all I have to do to get a project page with the correct layout is:

• git checkout gh-pages
• git submodule add https://github.com/mrtazz/jekyll-layouts.git _layouts
• copy the README.md of my project to index.md and add the jekyll frontmatter:

---
layout: project
title: project name
---

• add a _config.yml and fill out the following values:

gaugesid: tracking code for the gaug.es gauge
projecturl: github url for the ribbon in the upper right corner
basesite: base URL to get the CSS from
markdown: kramdown

• git push

The only dependency now is that the CSS comes from my main website. Which I’m fine with and is actually a feature because if I ever change something there I want the project pages to reflect that change also. The other downside is that if I change the project layout repository I will have to update the reference in all the project repositories. Which should be fairly straightforward with some automation and is at least better than copying files around and committing them to each repository.

Computers are terrible. Nothing works.

We all love to share our opinion and give talks about all the things that are wrong with computing. JavaScript is a popular topic, so is PHP, Redis, nodejs, Scala, Rails, Nagios, MongoDB and almost any other technology on different occasions. You can not spend a week (or sometimes even a day) without a new blog post about how some technology or piece of software (or better yet: everything) is broken lacking any positive take away.

I was at Monitorama at the beginning of May which is always amazing and great to meet and talk with friends and other people who work in infrastructure and operations in the vastest sense. And it’s easily one of my favorite conferences. I got to give a talk there and close to the end before introducing how our Nagios setup works I asked the audience to raise their hand if they have strong feelings about Nagios. From what I could see almost everybody raised their hand. Then I asked them to raise the other hand if those feelings were love. Almost nobody did.

We have arrived at a point where a piece of software (granted with some fixable inconveniences) monitors a complex system of computers, lets us know when something is broken and arguably enables a ton of businesses. And yet almost nobody has to say anything good about it. Everybody will tell you how terrible it is.

This isn’t about Nagios though. This is about general attitude. We work in a field where you can (compared to other professions) make a lot of money, usually don’t have to be too concerned about unemployment (it’s even kind of a sport already to complain and make fun of all the recruiter requests we get and be upset they are not perfectly tailored to what we want to work on) and generally get paid to solve problems. And then we start to complain about the fact that we can’t always choose the problems we have to solve. We build complex systems with a myriad of interactions and components and then have the hubris to say we should understand them in their entirety and whoever doesn’t is plain stupid.

Let’s be clear here. I don’t think you can never complain about things. I don’t think everything is unicorns and rainbows. And I don’t think you are not allowed to say something when things are horrible. But as a profession I get the feeling we have started to basically take negativity for granted. And with that we set an awful example for peers and especially (young) people coming to our industry. We say “computers are terrible” when it really just means “the computer does a thing that is really inconvenient for me right now”. We have a machine in front of us that let’s us boot other computers all around the world, talk to our families face-to-face wherever they might be and access any information known to mankind in an instant. Yet what we say is “look at all the crap we have to put up with everyday, recognize our prowess”. We could invent teleportation and enable everybody to be anywhere they want in an instant. As long as it sometimes puts us inconveniently a couple of meters away from our desired destination we would be tweeting that “this is horrible and everything is broken”.

Technology has made a lot of things possible for me. I would have never thought that I would live in what might be the best city on earth, love every day I go to work and am able to work on things I’m interested in, and have the opportunity to travel to conferences to have people listen to what I say about computers. Am I stressed, annoyed or even mad sometimes about yum resolving dependencies wrong, having to use a different way of comparing variables in PHP or the fact that my laptop goes to a blank screen again 20 seconds after I unlocked it? Yup. Does that make everything terrible? Nope.

I’m sure as hell happy and thankful I can work with computers every day, do interesting things and contribute to a platform where people create businesses and make a living on. Because computers are pretty amazing and I get to learn something new every day. And sometimes even feel almost like a wizard.

The Basics

I have OmniFocus running on my personal and work laptop as well as on the iPhone and iPad (although I hardly ever use it on there). They are all synced through webdav and owncloud on my personal servers. I mainly use it on the laptop and the iPhone client serves mostly for quickly inputting data or pulling up a list when I’m on the go.

Collecting all the things

As every GTD guide ever will tell you, the system only works if it is your one and only system that contains everything. And thus you have to add all your todos and ideas in there. I try to heavily follow this approach as I found it to be very true for me that I lose confidence in the tool as soon as it doesn’t contain my whole world. Paramount to this is the ability to enter new items from basically everywhere and support every way that could generate things for you to do. Luckily for me this means only a handful of things:

• Random things that I come up with
• Email
• GitHub issues
• Jira

This basically covers all variations of how I have new things landing on my plate. And thus I have made sure all those things find an easy way into my inbox.

Random things

I use Alfred 2 heavily on the desktop to quickly switch to or open apps, convert units, lookup people, and a myriad of other things. Naturally that means this is also the place where I should be inputting all new todos as they come to mind. For that I’m using an awesome workflow that I found somewhere on the internet. It allows me to fire up the Alfred prompt and simply enter todo do awesome thing @context and on hitting enter the new item is in my inbox with the correct context. This allows me literally add new things in a matter of seconds and bother later with filtering, remembering and doing them.

Email

Email is a little bit trickier. There are awesome plugins for Mail.app to work with Omnifocus and I hear they make it a breeze to get things done. However my email client of choice is mutt. Which means there is a bit more hacking to do (as usual). However I found a great Python script that parses emails and adds them to Omnifocus. I also added this keybinding to my mutt configuration:

macro index,pager \Ca "<enter-command>unset
wait_key<enter><pipe-message>mutt-to-omnifocus.py
<enter><save-message>=gtd-needs-reply/<enter><sync-mailbox>"

Now all I have to do when reading an Email or browsing through the list is hit Ctrl-a and mutt automatically creates a task in my Omnifocus inbox and moves the email out of my inbox into a folder I creatively called gtd-needs-reply (I also have one called gtd-to-read which I use for emails that I still have to read). This keeps my Email inbox clean and has the benefit since it adds an Omnifocus entry with the context “Email” that I can easily find all the emails I have to write with a custom perspective (more on that later).

GitHub Issues

A decent amount of things to do for me are also generated via GitHub Issues. This can either be issues on one of the public GitHub projects I maintain or more often a code review at work. We use Pull Requests on GitHub Enterprise for code reviews at Etsy and if someone wants you to review code, they assign the pull request to you. Since there is no need for me to go through my Email for notifications about code I have to review, I wrote a simple script that runs every 10 minutes and checks whether I have issues assigned that are not yet in my Omnifcous. This script reads a configuration file which can have an arbitrary number of GitHub (Enterprise) instances and asks for all issues assigned to a user owning the OAuth token. It then generates a ticket title based on the repo URL and issue number and adds a configurable context (Github or EtsyGithub for me) to it. It then creates an Omnifocus inbox tasks based on that data, again easily findable by context in Omnifocus.

Jira

We use Jira at Etsy to manage tickets and workload and thus the majority of my work is captured in there. Since I don’t want to have two places to look for things, I’m also pulling all my Jira tickets into Omnifocus. This is done with basically the same script as the GitHub sync but uses jira4r as the input source. It then drops a todo item with the Jira project key and ticket number into my inbox with the context Etsy:Jira. This makes it super easy to organize all the work I am assigned in Omnifocus. The only downside to that is that it’s not a 2-way sync. Right now I clean up and close tasks in Omnifocus (and Jira) when I actually finish them or during the weekly review. I also only create tickets for myself in Jira and don’t add tickets from Omnifocus when I create new todos with the Etsy:Jira context. This wouldn’t be very hard to do but I haven’t found it to be super painful to do it manually.

Basic Structure

So now that I have an easy way to enter all the incoming work into Omnifcous, the next step is organizing all the things. For that I use folders heavily for the basic structure and something close to the Areas of Responsibility in the GTD book. I have top level folders for Etsy, Personal, Talks and Open Source. And under those another layer of folders which reflect finer grained areas of responsibility. You see the structure here:

Within those areas a have the actual projects I work on and usually a single action list project for miscellaneous things. I organize active and someday projects in there by putting someday projects in On Hold status. This makes it easy to find projects I’m working on by filtering for active ones in a perspective. The project view is the most important one for finding and organizing my work. I never fully got into using contexts for things other than automated tools that pull in data. I rarely find myself in actually different contexts where it makes sense to pull up a specific list and all my tries to get that working ended in confusion for me (ymmv).

Perspectives all the way down

Based on that structure I have created a handful of custom perspectives to quickly find things I need. You can see the overview of my perspectives in the screenshot below.

The most important one is the Today perspective. It holds all items that are due, overdue or flagged. This is my daily todo list with things I wanna get done today. The next ones are Etsy active projects, next actions and weekly summary. Those I pull up for planning daily tasks and writing my weekly summary. I also have a perspective for Personal active projects which I don’t use that much but still pull up often enough to be valuable. The only crux with those perspectives is that they are mostly project and not context based. That means most of the perspectives don’t sync to the iPhone. For now that is ok for me because I mostly use the iPhone to add stuff to the inbox and to check my daily todo list which I made a context based perspective. And I’ve also heard that project based perspectives will be syncing to the iPhone in the future. So that will help a lot.

Review, Review, Review

As every person that is trying to do their version of GTD will tell you, consistent reviews are the heart of a working system. And that is no different for me. I try to really be disciplined about my weekly reviews and try to do daily reviews but often only end up actually doing them 3 times a week or so. Which is not too bad as long as the weekly review is consistent.

Daily

My daily review routine is pretty straight forward. I pull up the today list and mark everything as done I completed but haven’t checked off yet. Then I pull up my active perspectives and flag stuff I wanna work on today. That’s it, simple and easy.

Weekly

My weekly review is a bit more complex. I actually have a recurring project that becomes available every Friday and is due on Sunday and looks like this:

This is my checklist to do my weekly review. So every weekend I will clear out and archive all email and filter unprocessed email into gtd-to-read or gtd-needs-reply. This is mostly mailing list stuff since I try to stay on Inbox Zero during the week. I then put every thing I can think of that has to be done into the inbox. I check my calendar from last week if there is anything left to be done from meetings and check next weeks calendar for stuff I have to prepare. I then hit the Review button in Omnifocus and start reviewing all my projects. That usually starts with sorting all the inbox items into the folder structure and then going through all the other projects to mark things as completed and add new actions. This usually takes a bit longer for active projects, whereas on hold projects I can go over quickly because they don’t usually have a lot of activity. I have set my default review cycle to 5 days in general. That means projects become available for review again after 5 days, so when I don’t get to it on the Weekend and do my review on Monday morning, I still have the projects ready for review on Friday.

Verdict

For now I think I have found a good balance of using a lot of features of Omnifocus while still keeping it somewhat simple and not going overboard with the setup. Automating a lot of things - especially for inputting data - has made a big difference in trusting the system to be my only source of truth for work that needs to be done. My biggest problems are still making sure to take enough time for the reviews, keep adding todos and paying attention to my daily list even if I’m stressed and some days also … you know … actually getting things done.

Enter Twitter, twsh and GitHub

I had signed up for Twitter in mid 2007 while I was writing my Bachelor thesis and literally had no idea what I was supposed to do with it. In my circle of coworkers and friends who worked with computers I was most of the time one of the first to explore new things and thus I didn’t know a single person with a twitter account. I don’t even remember how I heard about Twitter in a world that doesn’t have Twitter. But I had extremely interesting tweets back then already of course. Fast forward to 2009 my new found interest in programming from my new job and the programming I had to do for class assignments in university meant that I was constantly trying out new things and experimenting with the concepts I learned. So at some point I decided I wanted to have a twitter command line client and started to write the twitter shell. It was painful and slow and I had no idea what I was doing. It was living there in a subversion repository on my Mac mini at home and I wanted to open source it eventually. I had no idea what that really meant either. But I had used a lot of open source software and was always fascinated by the idea that I could just look into how things work.

I first looked into hosting it on Google code but I found it to be ugly and weird. And a lot of people in my twitter stream were talking about git and this new thing called GitHub. Since there was nothing really tying me into subversion, I moved the code over to git and signed up for a GitHub account on March 28th 2009.

Brave New World

And it literally changed my world. I suddenly found a lot of people who were doing so many interesting things. And it encouraged me to work on all the ideas I had floating in my mind but thought were useless or boring. I found out about the programming communities behind languages like ruby or python. How to package software to be installed from PyPi. I started thinking about how to split software into projects as libraries and how to design APIs. I wrote API wrappers for things like Instapaper, a Notifo library or a tool to import YAML based groceries lists into the iPhone Groceries app and I also heard about continuous integration for the first time.

And I was fascinated by the idea of having a build system do all those things I usually ran commands for automatically. I read up on it and tried to get Integrity up and running, which seemed to be the most accessible solution to me at the time. And having worked on the notifo library I wanted it to push to my phone whenever a build would break or work (yes in general it was quiet enough for me back then that I wanted all of the notifications). So the time had come to contribute to an Open Source project and figure out how to ruby and write unit tests and all of those things. I wrote the notifier, submitted a pull request and after some feedback and improvements, Simon merged my notifier into master and I was super excited. I was finally able to have it running on my own CI instance and know the status of my builds with just a quick glance at my phone:

After that I was less terrified of contributing to Open Source and just trying out things. I kept perusing the GitHub explore page and found all those interesting projects. I went into something like an Open Source rampage and tried to contribute to and open source as much as possible. I even signed up for Calendar About Nothing and maintained something like 120 days with consecutive contributions at some point. And whenever I decided to push a new project I would meet and engage with more people and learn new things. For example I wrote the C++ implementation of Mustache templating for fun and met Jan because of that. Which then led to me meeting a lot of other awesome people in Berlin and around the internet.

And even though twsh never actually got finished and I lost interest in it, I can definitely say that GitHub has changed my computing life to the amazing. And I probably wouldn’t be where I am now without it.

For me a backup has two important parts:

• Have data in a different location
• Be able to restore data from the past

The time sensitivity of those two properties are pretty different for me. For example I have chosen for myself that I’m happy with only being able to restore deleted data from the last day. So if I create something and delete it 5 hours later, I’m ok with not being able to recover it. On the other hand I’m very aware of the fact that my mailserver can disappear at any given time:

that moment when you want to make dinner and your mailserver disappears

— Daniel Schauenberg (@mrtazz) December 14, 2013

This is why I want to copy data to a remote location as often as possible (which for me means about every 15 minutes). And my setup is heavily based around those ideas. The core of the backup system is ZFS and a separate file system for each machine I want to backup. In order to have the ability to go back in time I use zfs snapshots. Every night the following script runs on my backup server and creates a snapshot for the day:

#!/bin/sh
# simple script to snapshot locations on a ZFS backup pool

timestamp=`date +%Y-%m-%d-%H:%M:%S`
for volume in $(ls /backup); do
  echo "Creating snapshot for ${volume} at date ${timestamp}"
  /sbin/zfs snapshot backup/${volume}@${timestamp}
done

And to make sure that I really do have snapshots I have this simple nagios script to tell me if the snapshotting worked last night.

#!/bin/sh

# nagios script to check age of backup snapshots

YESTERDAY=`date -v-1d +%Y-%m-%d`
EXITCODE=0

for backup in $(ls /backup); do
  zfs list -t snapshot | grep ${backup} | grep -q ${YESTERDAY}
  if [ $? != 0 ]; then
    echo "Snapshot of ${backup} missing for ${YESTERDAY}."
    EXITCODE=2
  fi
done

if [ ${EXITCODE} == 0 ]; then
  echo "All backup volumes were snapshotted on ${YESTERDAY}."
fi

exit ${EXITCODE}

And this check (which runs on all my servers because I have zpools everywhere) to tell me about the disk health of the backup zpool:

#!/bin/sh

# check for zpool health
ZPOOL=`which zpool`
EXITSTATUS=0
IFS=$'\n'

for line in $(${ZPOOL} list -o name,health | grep -v NAME | grep -v ONLINE)
do
  echo $line
  EXITSTATUS=2
done

if [ $EXITSTATUS == 0 ]; then
  echo "All pools are healthy."
fi

exit $EXITSTATUS

With this setup in place I can simply copy files into the file system that belongs to that machine and it will get snapshotted every night. And what’s an awesome tool to copy data? That’s right, rsync.

My backup script runs once every 15 minutes and looks like this:

#!/bin/sh
#
# Backup script to pull in changes from remote hosts
#
for backup in $(ls /backup); do
  grep -q ${backup} ~/.backupexcludes
  if [ $? != 0 ]; then
    /usr/local/bin/rsync -e 'ssh -o BatchMode=yes -o ConnectTimeout=10' \
--archive --delete --timeout=5 ${backup}:. /backup/${backup}/
  fi
done

This allows me to have machines that I used to backup but are no longer online in an excludes list. That way rsync (and ssh) doesn’t hang or error for something that doesn’t need to be backed up anymore anyways. And in case a machine is unavailable or disappears the timeout settings in that script make sure it just gets skipped and retried on the next run.

I’m pretty happy with the setup, my backup server pulls in data from all my servers on the internet and stores it (forever?). It is chef’d for the most part (though there is always more to automate) and is pretty simple in my opinion. The backup situation for my laptop is not ideal yet, as I manually back it up by running rsync. I want to set the backup server up to also serve some of the backup filesystems as Timemachine targets, so I can just use Timemachine on my laptop and have it automatically run the backups.

But in the meantime I can add a new backup with this one weird trick:

zfs create /backup/newhost && chown -R mrtazz:mrtazz /backup/newhost

See this is not what you should have to remember to commit the code you just wrote to disk. I’ve seen such recommendations and the following complaints when the workflow breaks down a couple of times now. It even led me to compose this fascinating tweet at one point:

Before complaining about git please consider answering these 2 questions: 1. Were you trying to be overly clever? 2. Were you expecting svn?

— Daniel Schauenberg (@mrtazz) June 4, 2013

The Example

But in all seriousness in the end the most important thing about git is that it lets you commit changes to disk in a way so that your future self can figure out what changed later. And I want to highlight why I think it makes more sense to start simple and then figure out if additional features really make sense for you.

Let’s take an example workflow you can find on the internet and take a look:

git pull –rebase instead of git pull

git rebase -i @{u} before git push

(on “feature”)

git merge master

to make feature compatible with latest master

(on “master”)

git merge –no-ff feature to ship a feature

However if “feature” contains only 1 commit, avoid the merge commit:

(on “master”)

git cherry-pick feature

It doesn’t matter where this is from because I don’t want to disrespect the author or their ability to choose a workflow that works for them. It just is a good example (of which you can find hundreds that are similar on the web) of what a lot of people think a git workflow has to be like and cargo cult it into their setup. After all if you’re not using all features of git you must be missing out on precious productivity right?

Back to the example. Have you kept track and can remember without looking when to merge, rebase or cherry-pick? You are basically doing the same thing - bringing new commits to your branch - and you have to do 3 different things depending on some circumstances. But this isn’t even the most error prone part. Let’s consider you work on a fairly active project with a lot of people. You are maybe even doing continuous integration and continuous deployment and trunk/master gets commited to all the time. You are working on a feature in a branch (btw while we’re at it read Camille Fourniers’ excellent blog post to find out why you actually shouldn’t do this in a continuous deployment setup) and want to pull in changes from master. So you merge master as per the instructions. You notice someone else has pushed to the feature branch in the meantime. You update the local branch with pull --rebase as per the instructions again and push up your changes. If you ever had to do this you won’t be surprised that your working tree now looks something like this:

You now have the same commit with two different SHA IDs on different branches. This might not look like a big problem right now. But I for one don’t like to have multiple IDs for the same thing and also think that having two commits doing the same thing is not an ideal situation.

But even if you work alone on your feature branch and don’t need to rebase before pushing you will eventually run into a similar situation on your master branch. You have worked on your feature branch for a while. You opened a pull request, incorporated feedback and pushed it back up to the branch. People have linked to your commits in tickets to note that it fixes stuff they’ve been waiting on. Now the big moment has come and you want to integrate with master. You pull in changes from origin, you run git merge --no-ff feature-branch to bring in the changes with a clean merge commit. You run git push but there have been upstream changes. So you bring them in again as always with git pull --rebase. Now you can push and everything’s fine right? Well almost, except that last rebase has rewritten all your commits from your feature branch (if you additionally ran git rebase -i before pushing you are probably very well aware of this). You might think that this is ok or even intended since you wanted to clean up your commits anyways. And that this makes it much cleaner and easier to read. However what you effectively just did was rewrite (git) history and remove public references of changes. Everybody who linked to your commits now has links pointing to a non existing resource (they will still be there until you eventually clean up the remote feature branch but this is just details in my opinion). Pull requests don’t get automatically closed and you sure don’t remember to do it by hand and all references in there are useless anyways. And all just for a little bit of beautification of how you actually did your work.

So what are you saying?

Does that mean you should never use rebase? Am I saying that only the basic git commands are supposed to be part of a workflow? Do I call all these developers using advanced techniques stupid?

N-O-P-E

I do use rebase in some occasions, I don’t think it’s the devil’s work. There are legitimate reasons to use it and it sure is helpful sometimes. However when you’re starting to get into git or migrating to it and building a workflow for your team it often makes the most sense to start with the simplest thing that could possibly work. git add, commit, push, pull, merge. All these are safe operations and don’t break anything for others and most likely don’t bring you into weird situations where you’re totally stuck and lost. There might come a time where you bring rebase and its siblings into the mix. But then there should be a reason and it hopefully is because it helps you be more productive and removes confusion in your team. And I sure hope you know how it works and what you’re getting yourself into. Because it can be what you end up with but it shouldn’t be where you start. Keep it stupid simple.

I added a git repo ~/.dotoverrides to all the machines I work on (or at least most of them) which contains a vimrc, a zshrc and so on. On my work machines this is pushed to a repo on our internal GitHub Enterprise instance so I can easily share it between machines. And all my regular dotfiles now source those override files at the very end.

So in my regular .vimrc I have something like this:

" source overrides configs
if filereadable($HOME."/.dotoverrides/vimrc")
  exec ":source ". $HOME . "/.dotoverrides/vimrc"
endif

In my .zshrc I have this:

[ -f  ${HOME}/.dotoverrides/zshrc ] && source ${HOME}/.dotoverrides/zshrc

And in git (only works if you have at least v1.7.10) I’ve added this stanza:

[include]
  path = ~/.dotoverrides/gitconfig

Now I can easily share and push/pull my regular dotfiles in public GitHub and don’t have to pay attention whether or not I’m on the correct branch and if I’m not accidentally pushing to the wrong remote. Whenever I need to use different settings on a work machine I just make sure to add it to the overrides file and have it ready as soon as I open a new shell, run a git command or open vim again.

So much easier!

#!/bin/sh

USERHOME=$1
SIZE=$2

zfs create -omountpoint=/encrypted tank/encrypted
zfs create tank/encrypted/keys
zfs create -omountpoint=none tank/encrypted/zvols
zfs create -ocompression=on tank/encrypted/zvols/${USERHOME}
zfs create -V ${SIZE}G tank/encrypted/zvols/${USERHOME}/disk0
zfs create -V ${SIZE}G tank/encrypted/zvols/${USERHOME}/disk1

zfs create tank/encrypted/keys/${USERHOME}
dd if=/dev/random of=/encrypted/keys/${USERHOME}/disk0 bs=64 count=1
dd if=/dev/random of=/encrypted/keys/${USERHOME}/disk1 bs=64 count=1
geli init -s 4096 -K /encrypted/keys/${USERHOME}/disk0 \
/dev/zvol/tank/encrypted/zvols/${USERHOME}/disk0
geli init -s 4096 -K /encrypted/keys/${USERHOME}/disk1 \
/dev/zvol/tank/encrypted/zvols/${USERHOME}/disk1

geli attach -k /encrypted/keys/${USERHOME}/disk0 \
/dev/zvol/tank/encrypted/zvols/${USERHOME}/disk0
geli attach -k /encrypted/keys/${USERHOME}/disk1 \
/dev/zvol/tank/encrypted/zvols/${USERHOME}/disk1

zpool create ${USERHOME}-home raidz \
/dev/zvol/tank/encrypted/zvols/${USERHOME}/disk0.eli \
/dev/zvol/tank/encrypted/zvols/${USERHOME}/disk1.eli

I try to keep the script updated on GitHub.

And the book was absolutely great. It does a great job of running through many areas where sexism and the patriarchy does a lot of harm to men as well. And it made me think a lot about my upbringing, childhood, and what kind of biases I carry because of it. But more importantly it made me think about the things that I value and was at times longing for. E.g. the fact that relationships between men are often very competitive and don’t provide any space for emotional depth and vulnerability. Limiting any chance of properly dealing with emotional situations in that setting.

I can wholeheartedly recommend this book to anyone (especially men) interested in learning more about the topic.

The Basics

Let’s start with the basics. By default tmux uses ctrl-b as its prefix key for commands and escaping. But the years of using screen have ingrained in my muscle memory to use ctrl-a, so I switched with this simple setting:

unbind C-b
set -g prefix C-a

I also added a couple of important baseline settings to make tmux in general look nice in colored terminals and work with unicode in the display window as well as in the status bar. I also wanted to have the window numbering start at 1, since it doesn’t make sense to me for accessing succesive windows to start on the right side of the keyboard and then continue on the left side. And I also wanted to have a simple shortcut (ctrl-a r) to reload configuration in a live tmux session whenever I change something.

# force a reload of the config file
unbind r
bind r source-file ~/.tmux.conf

# start window numbering at 1 for easier switching
set -g base-index 1

# colors
set -g default-terminal "screen-256color"

# unicode
setw -g utf8 on
set -g status-utf8 on

The next important change was modifying the status bar. There are a lot of crazy things you can do and overload your tmux status bar with more information than you could ever need. I try to balance the contents of my status bar to only have information in there I actually care about. This means I have the local hostname and the name of the current session on the left side and then all the windows. The right side contains the current battery status (when I’m on a laptop), the status of my mail (inbox, to read, to answer) and the time and date. Although I see less and less benefit of having the mail check in there and will probably remove it soon (currently it’s only showing the inbox mail count). I also have the status bar configured to show terminal bells in red so I always know when there is something that needs attention in a window (I have weechat and mutt set to alert via terminal bells). For the colorscheme I use a solarized light theme as you can see in the screenshot:

And the configuration for my status bar looks like this:

# status bar config
set -g status-left "#h:[#S]"
set -g status-left-length 50
set -g status-right-length 50
set -g status-right "⚡ #(~/bin/tmux-battery) [✉#(~/bin/imap_check.py)] %H:%M %d-%h-%Y"
setw -g window-status-current-format "|#I:#W|"
set-window-option -g automatic-rename off

# listen to alerts from all windows
set -g bell-action any

This is the base configuration I use for basic project sessions with tmux. I have two simple shell aliases to make it easier to re-attach to a session and create new ones based on the current directory I’m in:

alias tma='tmux attach -d -t'
alias git-tmux='tmux new -s $(basename $(pwd))'

With those I can ran tma <tab> in any shell and get a tab completion list for all the current sessions running. Which is handy when logging into a machine or generally working in a new shell. The second one I usually use when I checked something in a local project (which are usually in git hence the name of the alias) and then decided that I want a proper workspace but don’t have an existing session already. The alias will just create a new session on the spot and name it after the current directory name. This also has the big advantage that all new shells spawned inside of tmux (e.g. opening a new window with ctrl-a c) will be started in that directory. Within those open sessions I have some more important shortcuts I use often. They allow me to cycle through panes (vertical or horizontal splits in a window created with ctrl-a V and ctrl-a H) with ctrl-a a and to switch between windows with ctrl-a <tab>.

# rebind pane tiling
bind V split-window -h
bind H split-window

# quick pane cycling
unbind ^A
bind ^A select-pane -t :.+

# screen like window toggling
bind Tab last-window
bind Escape copy-mode

And last but not least in every basic setup - as an avid vim user - movement commands live on the home row of course. And different panes can be selected with ctrl-a and the corresponding movement command.

# vim movement bindings
set-window-option -g mode-keys vi
bind h select-pane -L
bind j select-pane -D
bind k select-pane -U
bind l select-pane -R

Next Level

I used to use tmux sessions in multiple tabs in iTerm for a long time. However these sessions became very long living and whenever I needed to update iTerm (which wasn’t that often to be honest) I had to recreate the tabs again as I wanted. Additionally I felt it was unnecessary to have multiple ways of doing basically the same thing (iTerm tabs and tmux windows/sessions) when I can just decide to use one. So I decided to switch to tmux as the main working environment on the laptop for everything. This means I have a tmux session on my laptop dedicated to communication which has a window that runs mosh with an attached tmux session from the server where I run weechat. And another window that does the same to my work VM which runs my work IRC client. And another window that just runs mutt for email reading. This means at any given time I have two nested tmux sessions as you can see in the screenshot below:

This lets me reattach the communications session even when I accidentally close my terminal and have it be exactly how I left it off. Even when I don’t connect to a remote host, I often have nested tmux sessions locally since I use it basically like terminal tabs. This is very useful but needs one more setting in the configuration to work. Since both nested tmux sessions expect the same meta command, I have this stanza in my configuration:

bind-key a  send-prefix

This sends the command prefix to the inner tmux session when I hit ctrl-a a thus enabling me to execute commands in nested tmux sessions.

In order to easily switch between sessions I mainly use to important commands. The first one is the tmux built-in ctrl-a s which gives me a list of all current sessions on the system (the same list the tma tab completion gives me) and I can easily switch sessions from within a tmux session. However this means finding the session I want in a list that might contain 20 or more sessions. And all I really want is to switch to the session named “chef”. This is why I added another extremely useful shortcut:

# bind fast session switching
unbind S
bind S command-prompt "switch -t %1"

Now when I hit ctrl-a S I get a (switch) prompt where I can enter the name of the session I want (or just the prefix as long as it is unique) and switch to that session when hitting Return. This is super helpful since I have most of my sessions named after the directory/project name anyways. So I usually know which session to switch to.

We have to go deeper

But this is not the end yet. I have one more very useful bit of configuration I use everyday which is related to how I remote login into servers. For this purpose I have a tmux session on a server called “jumpsessions” in which I open a new tmux window whenever I ssh into a server. However this got very confusing after a while and I had no idea what all those windows were. So I added this little bit into my ~/.ssh/config:

Host *
PermitLocalCommand yes
LocalCommand if [[ $TERM == screen* ]]; then printf "\033k%h\033\\"; fi

This runs a local command on each ssh login on the server I login in. With the effect that it prints the local hostname with an escape sequence that triggers tmux to set the window title to that hostname. This means if I now open the list of windows (ctrl-a w) I can see to which server each window is connected. And this is also the reason why I have automatic window renaming turned off.

But of course I don’t want to browse through all of those windows to get to a server, so I just use the “find-window” command in tmux (ctrl-a f) and enter the server name (which is also the window name) and will automatically switch to the correct session on hitting enter.

And as the final stage of inception, I often run a screen session on those servers to execute long running commands. Which means I’m now three levels deep into terminal multiplexers and it still works like a charm.

The setup I’m describing has grown pretty organically and is heavily based on what I use and how I work everyday. This is also probably a bit too technical to be considered a general purpose manual. But that will have to do for now. I am also a heavy FreeBSD user, as it makes a lot of things easier and more enjoyable for me. So my setup is also very biased towards that.

Email

Maybe one of the most important parts is email. I switched from hosted email providers to self hosted (first on a friend’s server) in 2005, when the 12MB Inbox I had wasn’t big enough anymore and before GMail was widely available in Germany (at least I only knew one person with a GMail account back then). So nothing has changed for me there. I also think email is considered to be one of the more painful things to self host although I don’t think this is true. I run a setup based on FreeBSD, sendmail and the Dovecot IMAP server which is not very complicated to set up. Especially the FreeBSD/sendmail part literally takes 10 minutes. I don’t really run spam filtering since it hasn’t been a problem (I do filter some known spam addresses in my procmail rules though). I read my email in mutt on the laptop where it is synced with offlineimap and also run mutt in a tmux session on my mailserver to access it from anywhere. On iOS devices I use the built-in Mail application and have come to love Triage for quickly going through email when I have a minute.

Calendars and Contacts

Another very important aspect of my daily synced data are calendars and contacts. Especially with the iPhone and iPad being in constant use, I want that data to be synced everywhere. I used to use iCloud for that and it works beautifully and I wanted something which works equally flawless. After some trial and error I found ownCloud which provides CalDav and CardDav services as well as general WebDAV. The setup guides are really good and include most of the common clients. I nevertheless ran into some problems with the initial setup on iOS and OSX clients because of when and where they expect slashes or protocol headers. However this is a configuration/documentation issue, which is annoying but can be solved.

File sync

I used to use Dropbox a lot. I loved the simplicity and being able to have files in sync everywhere. I even put my git repos in there at some point so I could continue working from every computer I used. With time I used it less and less as I simplified my workflows a lot but it was still important to have a proper file sync solution, mostly for convenince options like syncing Alfred preferences. But I wanted to get all my documents out of a location that somebody else had under control. Thankfully ownCloud also comes with a client to sync the WebDAV directory between computers. So I basically set that up and copied everything over from Dropbox. It has been working really well so far, though I don’t have heavy requirements for syncing and files in there don’t change that often.

GTD/Todo tracking

I track everything in OmniFocus. Literally. Work stuff, personal stuff, movies I want to watch, books I want to read, it pulls in GitHub issues and Jira tickets that are assigned to me, I plan blog posts I want to write and talks I want to give in there. I extensively use custom perspectives to get data out. It’s safe to say that it’s an important piece of software for me. Luckily Omni products support syncing via WebDAV and have been for a while. Thus it was very easy to switch from the hosted Omni Sync Server, which works flawlessly, to just use WebDAV endpoint of ownCloud. I have since also looked around to find out if there are alternatives to OmniFocus if I ever wanted to switch away from OSX. Sadly it seems to be that self hosting is rarely an option for any app and I have only found a handful that even provided a synchronisation mechanism that does not involve DropBox, Apple or their own cloud sync solution.

Note taking

Note taking was also an important part that had to continue to work for me. I don’t take a lot of notes all the time. But when I need to jot something down, it must not matter whether I’m on my phone or in VIM on my laptop. I was a very happy Simplenote customer and still think it’s the best cloud based note taking platform there is. I even wrote a VIM plugin for it so I’d never have to leave my trusty editor. This also meant a solution that would replace it needed a decent iOS client, notes I can access from VIM, support for Markdown and a syncing engine that is ideally based on WebDAV, since I was already running that. And after some searching I actually found this unicorn of note taking solutions. It’s simply called Notebooks and it’s a simple app that displays the folders and files in a WebDav directory, let’s you edit text files and view them in Markdown mode. And even take and attach pictures. It comes as a Universal App for iPhone and iPad and has an OSX client in a beta version, which I don’t use because I can just edit all the files in VIM. Which makes me very happy.

Password syncing

The only application I haven’t found a satisfying self hosted solution yet is password syncing. I use 1Password and am very happy with it. However the only non-LAN solutions for syncing that it provides are Dropbox and iCloud. So I switched to Wi-fi sync for my passwords. It’s not ideal and there will come a point where I am on my iPad and don’t have a password there and am too lazy to open the laptop to sync. However since all passwords for my crucial services are already synced this won’t be the end of the world and can very likely wait until I am on a nother device or have both the laptop and the iPad open. So I’m not 100% happy with it but it is one of those “good enough” solutions.

IRC and Instant Messaging

Being able to idle on IRC and have a proper chat client at hand everywhere has always been important to me and for that I have run terminal based clients in a screen or tmux session for years now. Since I (similarly to email) never used any of the cloud based solutions, I was already running ZNC and Bitlbee. And since the changes in GTalk earlier this year which broke a lot of stuff for me, I also already had a Jabber account which I was using for chat and OTR.

Backup

How to handle backups was one of the bigger concerns I had. Now that I would be hosting all my data I needed a proper plan so when one of my servers dies I’m not losing everything that was on there. Like probably almost every Mac user, I used to use Arq to backup my laptop to an encrypted S3 bucket. However that was only ever the client side. And I was happy with it because it included my mail folder and thus I had a backup of my email. And when I stopped using that to not push all my date to S3 I also didn’t backup my email anymore. After some thought it was clear to me that I wanted to have a backup in a location with as much control as possible. I decided to buy an HP Microserver and put it in my apartment. It runs FreeBSD (surprise!) and has a 2x2TB encrypted ZFS RAID. The backup location for each of my machines on that RAID is an independent filesystem so I can snapshot it regularly and go back in time if I have to. The server pulls in data from my servers via rsync and that’s how I do backups. It’s less automated than I want it to be right now and I still have to configure it to server as a TimeMachine destination for my laptop. But this is already a pretty good solution for me.

Where I still use the cloud

I’ve extensively talked about how I moved my data into self hosted applications and what I use for those use cases. However that doesn’t mean that I’m completely free of cloud based applications. Obviously there are a variety of applications that don’t support this yet or where it’s not even something that would work without changing the product a lot. That means I still use Dropbox to sync Papers or automatically pull in pictures from Instagram. Since Google Reader died I switched to Feedbin and have no intention to stop using it, I have my Kindle books at Amazon, my music in the iTunes Cloud, I use a variety of infrastructure software as a service to monitor my servers and I run my public image sharing and custom URL shortener on S3 and Heroku and this blog on GitHub Pages. The difference for me is, that most of this data I don’t necessarily regard as private as the ones I pulled into my own hosting. I will probably experiment with how I can do some of this on my own in the future, but it is less important to me right now.

Why are you telling me all this?

As I said in the first section, this is not considered a manual of how to host your own data. While I try to keep my Chef cookbooks for this stuff up to date, they are very custom tailored and probably not of great use for everybody. If you want to get started and host your own data, I highly recommend checking out Alex Payne’s Sovereign Project. It’s an Ansible project which installs a lot of the things I’ve been talking about here and is definitely much easier to get started with. I do hope though I was able to share some ideas and make hosting your own data sound a little less scary.

I also realize that even with an easy to get started guide and a collection of Chef recipes this is not something every person can run and you need some understanding of (and tolerance for) running your own services. There has been some work going on for some time to make it easier to host your own services and even have decentralized applications. The newest one I am aware of is called Grand Decentral Station and looks very promising. I would love to see some of these ideas flourish and be pushed forward. And maybe have a future in which we can not only pay people to run services for us, but also to develop services we can run ourselves as easy as it is to set up a TV or a Roomba today.

The Riak handbook is a great overview over one of the most popular and sophisticated NoSQL databases during the early 2010s. The book guides you through a number of easy to follow examples to learn all the basic properties of Riak. It uses JavaScript as an easy accessible language to understand the examples. It shows that Mathias has done deep research into the database and has put a lot of effort into creating an accessible resource to learn Riak.

The IRC setup

My basic setup revolves around using the ZNC bouncer which keeps me always connected. I still use weechat in a remote tmux session most of the time, but like to have the option to switch clients without losing my connection or backlog. I also use Growl pretty heavily in combination with OSX notification center to alert me of special keywords or all messages in certain channels. Past solutions included running the IRC client locally with a growl plugin or remote tail-ing a notification logfile. Those solutions were close to what I wanted but tied too much to the client, when I really wanted to have notifications directly from my bouncer. And since znc has a module to log all messages to various logfiles, I decided to get my notifications from there.

Enter logstash

I had read about logstash before and decided to give it a try for this. I won’t go into detail about installing and running it here, but check out the getting started for a good introduction.

For the first important step, we need logstash to listen to changes in the bouncer’s logfiles. This is pretty easy and can be accomplished with the following logstash configuration bits:

input {
  file {
    path => "/home/username/.znc/users/zncuser/moddata/log/*"
    type => "znclog"
  }
}

Per default the log module puts all log files under users/youruser/moddata/log/ and creates a logfile per day which is named after the channel name and date. The logstash input just reads all files that are in there and adds a type to the captured logs to be able to better identify them in subsequent filters. The pattern is not really ideal since older logfiles are not interesting for notifications but are also kept open. So at the moment I work around that by moving my logfiles to a backup partition every night, but there might be a better way to do it.

The next step is to remove lines which I’m never interested in for notifications, like my own messages and JOIN/QUIT messages for example. For this the logstash grep filter definitions are very useful:

filter {
  grep {
    type => "znclog"
    match => ["@message", "\[[0-9:]{8}\](.+?)<USERNAME>"]
    negate => true
  }
  grep {
    type => "znclog"
    match => ["@message", "\*\*\* (Quits|Joins|Parts|.+ sets mode: |.+ is now known as)"]
    negate => true
  }
}

The grep filter is also very useful for another criterion on which I want notifications, namely for all of my private messages. Since all channel names per IRC convention have a # in the name, we can just assume that logfiles without that sign are for private messages. It is important to set drop => false here since we don’t want grep to drop the log line (which is default behaviour).

grep {
  type => "znclog"
  match => ["@source", "#"]
  add_tag => ["pmnotification"]
  negate => true
  drop => false
}

This also needs to be added to the filter section and tags all messages coming from logfiles without a # in the name with "pmnotifcation". Now let’s go to the actual parsing of log events. Since there are going to be some repeated patterns and I wanted to have an easy way to add new ones, I have a ‘pattern library file’ which is included in the configuration.

NOTIFYME (pizza|cupcakes|fire)
IRCNOTIFY %{DATA}%{NOTIFYME}%{GREEDYDATA}
IRCTIME [0-9:]{8}
IRCCHANNELS (nunagios|chef|food)

The terms in capital letters can be used as regex placeholders. The interesting ones are NOTIFYME/IRCNOTIFY which are used as a collection of regexes on which I want to show a notification and IRCCHANNELS which are basically the channel names for which I want notifications for all messages. In order to get those notifications I set up a set of grok filters.

grok {
  match => ["@source", "%{IRCCHANNELS}"]
  add_tag => ["channelnotification"]
  exclude_tags => ["pmnotification"]
  patterns_dir => '/home/username/logstash-patterns'
}

This grok ruleset grabs all events from the channels based on the IRCCHANNELS match and tags them with the "channelnotification" tag. PMs are excluded from that match because they have already matched.

grok {
  pattern => "\[%{IRCTIME:irctime}\](.+?)<%{DATA:ircsender}>%{GREEDYDATA:ircmessage}"
  tags => ["channelnotification"]
  patterns_dir => '/home/username/logstash-patterns'
}
grok {
  pattern => "\[%{IRCTIME:irctime}\](.+?)<%{DATA:ircsender}>%{GREEDYDATA:ircmessage}"
  tags => ["pmnotification"]
  patterns_dir => '/home/username/logstash-patterns'
}

These rulesets extract the timestamp, sender and message data for the notifications into separate fields so they are easily accessible later on. I have the same ruleset for channel notifications and private messages, because I didn’t find a way to match any tag (the tags setting requires an event to match all given tags) so I couldn’t combine them into one rule. Though this seems like something that should be fixable.

grok {
  pattern => "\[%{IRCTIME:irctime}\](.+?)<%{DATA:ircsender}>%{IRCNOTIFY:ircmessage}"
  add_tag => ["notification"]
  exclude_tags => ["pmnotification"]
  patterns_dir => '/home/username/logstash-patterns'
}

And finally the last pattern ruleset matches the regexes that are defined for all events and parses them into the fields mentioned before. Notice that all rulesets include a patterns_dir section which points to the folder with the regex defintions file described above.

The last part of the logstash ruleset is defining an output for the notifications. For a while I just appended them to a logfile and tail-ed that from my laptop over ssh. This worked ok, but I had problems with duplicate notifications when restarting the polling script and wasn’t really happy with this solution. And since I already had Redis running on that host, I thought I’d give that a try.

output {
  redis {
    host => 'localhost'
    data_type => 'list'
    key => 'notifications'
    tags => ["pmnotification"]
    password => 'secret'
  }
  redis {
    host => 'localhost'
    data_type => 'list'
    key => 'notifications'
    tags => ["channelnotification"]
    password => 'secret'
  }
  redis {
    host => 'localhost'
    data_type => 'list'
    key => 'notifications'
    tags => ["notification"]
    password => 'secret'
  }
}

The output config basically just says the for every type of notification log event, append it to a Redis list with the name 'notifications' on the instance running on localhost.

The client side

The last part now is actually getting the notifications into growl on the OSX side of things. For this I have Growl setup to forward everything to notification center and run the following script on my Mac:

import sys
import gntp
import json
import redis
import gntp.notifier

r = redis.StrictRedis(host='ircserver',
                      port=6379, db=0,
                      password="secret"
)
app = "irc-growl"

while 1:
    key, logline = r.blpop("notifications")
    try:
        log = json.loads(logline)
    except Exception as e:
        title = "Failure loading logline: " + str(logline)
        message = "error({0})".format(e)
        gntp.notifier.mini(message, applicationName=app, title=title)
        continue

    try:
        channel = "-".join(log["@source"].split("/")[-1].split("_")[1:-1])
    except Exception as e:
        title = "Failure parsing channel name in: " + str(log["@source"])
        message = "error({0})".format(e)
        gntp.notifier.mini(message, applicationName=app, title=title)
        continue

    try:
        title = ("%s in %s" % (log["@fields"]["ircsender"][0],
                  channel.encode("utf-8")))
    except Exception as e:
        title = "Failure parsing ircsender in: " + str(log)
        message = "error({0})".format(e)
        print title
        print message
        gntp.notifier.mini(message, applicationName=app, title=title)
        continue

    message = (log["@fields"]["ircmessage"][0]).encode("utf-8")
    gntp.notifier.mini(message, applicationName=app, title=title)

This uses the python gntp library to talk to Growl and the redis client to talk to Redis. Specifically for the Redis connection I use blpop, which pops an element (in our case a notification) from the list and if there is none waits for the next one to come in. For every notification it parses out the timestamp, channel, sender and message from the fields I set in the logstash grok rules, formats it nicely, sends it to growl and then gets the next one or waits for new notifications to come in.

Verdict

There are still some improvements I want to make. Mostly around moving the old log files or only reading the newest one. And improving the script so it survives network disconnects and possibly run it under launchd. Also if I’m not running the script to pull notifications, they are piling up in Redis at the moment. So next time I connect, I get an abundance of new notifications. Notification center batches them nicely to not litter the whole screen and only the last 20 are in the sidebar. So it’s not really a problem, but I thought about running a cron to prune the list to a maximum of 20 notifications or so.

I now have a setup where I get my notifications directly from the bouncer logs and can display them on any (OSX) host which has the script set up. It should also be fairly simple to adapt this to other notification display systems. The setup is no longer bound to which IRC client I use or whether or not I constantly have it running on a server. Plus the alerting keywords and channels are easily extended because I only have to add patterns to the library file and not touch the config itself.

My setup

I have a small set of servers which I’m using for basic services. These include mail server, IMAP, backup MX, IRC bouncer and general remote shell for running mutt, weechat, newsbeuter and other terminal based applications. I recently got around to more or less properly create cookbooks for this as I am running chef for configuration management. This also prompted me to finally set up monitoring and alerting for the services I care about.

External service monitoring

Servers are not very useful when their services are not accessible from the outside world. So you want to monitor this from an external source which usually tries to establish a connection to specified TCP ports. The general first service to use is pingdom. They provide a great service with great statistics. However since I want to monitor more than the free plan offers (and possibly more than the cheapest paid plan also), I was looking into an alternative. Since I already have an account at zerigo for some DNS services, I decided to give their Watchdog service a try. It’s $15 per 3 months and allows 50 service checks for 10 hosts with checking time down to every 5 minutes. This is more than enough for my needs and comes down to $5 a month. The only drawback is that they only provide email notifications (which can be somewhat mitigated with ifttt or the mail to text gateway of your mobile provider) to one user and a not really great statistics overview. Otherwise it works pretty great.

Process monitoring

The next step is to monitor the processes which are actually providing those services. For this I’m running a Sensu instance on Heroku in the setup I described before. Sensu is an awesome monitoring framework which provides a lot of flexibility, so it’s definitely worth checking out. Since it runs on two small Heroku instances I can host the server and API for free which works pretty well. As basic checks I test for running sendmail, cron and dovecot processes. If the checks fail the given threshold, an alert is pushed to an IRC channel on my grove.io organization. Admittingly this is a little bit overkill since the basic plans for grove.io start at $10, but I like to play and experiment with chat based interfaces to infrastructure automation and monitoring. An alternative would be to use Campfire which is free for a small amount of users. I am also playing with the idea of having a Boxcar handler either for Sensu itself or alerting to Boxcar from IRC. Boxcar is a pretty sweet service which handles push notifications to mobile phones and I’m already using it for notifications from my IRC bouncer and ifttt.com. And since I’m also running an instance of Hubot (also on a free Heroku instance) it should be rather trivial to have the bot listen for patterns and send Boxcar notifications upon match.

Log processing

Since I don’t want to log into several servers to quickly check different logfiles, I’m sending all of my log data to Papertrail. They provide an easy endpoint to send log lines from various systems such as syslog, rsyslog or directly from an application with an rsyslog handler. Their basic free plan allows for 100MB of log data per month with a searchable archive of 1 week. This amount should be enough for a small set of systems with average log data. After that you get 1GB of log lines in the first stage of paid plans for $7, which is still a decent trade. The big advantage is that I can now log into a web interface and see specific log information (for example about chef runs) across all of my servers.

Machine level metrics

Additionally I also gather machine level metrics for all of my servers. These include basic information about CPU and memory usage, disk space and uptime. All of these metrics are gathered by collectd and its various plugins and are sent to Librato Metrics for graphing. This is a lot easier and less hassle than managing your own Graphite instance. And you only pay for the metrics you actually send. The data I currently send there are basic metrics from 2 servers and the number of Sensu check occurrences and it adds up to something around $5 a month.

Verdict

This setup gives me (in my opinion) a pretty good monitoring solution for my personal infrastructure. Since I don’t consume a lot of resources for the services I depend on, I can usually use the free or cheapest plan available. With the cheapest options it’s around $10 a month and even adding grove.io and paid Papretrail into the mix only brings you to a bit more than $25 a month. Of course depending heavily on 3rd party services opens a whole new discussion about availability which you should be aware of.

For configuration examples for the services mentioned above, you can check out my chef cookbooks. They are mostly run on FreeBSD but should be somewhat easy to adapt to a different environment.

Some months ago I wanted to set up monitoring for a handful of servers I use for personal stuff. As a first solution Nagios came to mind. However for several reasons I didn’t want to set it up and configure it. And I really didn’t want to dedicate an existing server to do monitoring or get a new one just for that purpose. Around that time I also read about Sensu, a new approach to monitoring, which is a result of Nagios not being a good fit for the monitoring needs at Sonian. Its technology stack is Ruby, Redis and AMQP. I immediately thought it should be possible to put this on the Heroku Cedar stack and run it on an instance there, which would make a nice solution for monitoring a small number of systems. So I hacked away and with a lot of help (and patience) from Sean Porter, the adaptions to make the server and API part of Sensu deployable on Heroku are in the new 0.9.6 release.

Setting up the Sensu repository

In order to get started and configure your Sensu instance, clone the example repository from Github.

    git clone https://github.com/mrtazz/sensu-heroku-example

The example includes a basic folder layout for running a server or API instance on Heroku. All configuration files can be dropped in the config/ folder. They will be picked up by the process when Sensu starts. The example repo also includes a basic handler (bin/showme.rb), which prints event data to STDOUT. There are a lot more handlers in the Sensu community plugins repository on Github. Since handlers are just ruby scripts, you can download the handlers you want and also put it in the bin/ directory. Don’t forget to add the correct configuration file for the handler in the config/ directory also. A great overview how to configure Sensu can be found on Joe Miller’s blog and there is also an official install guide.

Deployment

In order to deploy Sensu to Heroku, you need to create two apps. One will be the Sensu API instance and the other one the Sensu server. It doesn’t really matter, which one you start with. The important thing is, that you only need to add the RabbitMQ and Redis plugins once and can then reuse the settings on the second instance.

So create the first instance on the cedar stack from within the example repo and add the plugins:

    heroku create --stack cedar awesome-sensu-server
    heroku plugins:install redistogo
    heroku plugins:install rabbitmq
    heroku config:add API_PORT=80

You have to add the API_PORT environment variable to the server instance, since otherwise it will assume it’s running the API itself and assign the instance locale port from the PORT environment variable to use as the API port. After that is done, push the code to Heroku and scale up a worker process:

    git push heroku master
    heroku ps:scale app=1

For the API instance create a new branch in the repo or clone the example repo into a new location. Then initialize the API:

    heroku create --stack cedar awesome-sensu-api
    heroku config:add REDISTOGO_URL="value from server instance"
    heroku config:add RABBITMQ_URL="value from server instance"

Now change the Procfile to start up the API instead of the Sensu server like this:

    app: sensu-api -v -c config/config.json -d config/

Commit the changes and push it to the Heroku app:

    git push heroku-api master
    heroku ps:scale app=1

Now all you have to do is set up clients and voila, you have Heroku hosted monitoring. If you’re not yet familiar with setting up clients, I highly recommend Joe Miller’s blog again. He’s a strong contributor to Sensu and has written an abundance of blog posts and tutorials about it. And of course there is also the sensu wiki.

Further improvements

A definite improvement for plugins and handlers would be to be able to also read configuration from environment variables. At the moment the way to go is to add a configuration JSON file in the config folder. This is fine except for the fact that you’d also have API keys commited to the repo.

And obviously more bugs will probably come up, once more people run Sensu on Heroku. I’ve been running a low volume instance for a couple of weeks now and it works pretty great so far.

The setup I am going to describe is based heavily on Joshua Timberman’s post about managing Mac OS X workstations with Chef. If you already know Chef, go read it, it’s great. As all my workstations are running OS X, the steps described are only actually tested on this OS, but should hopefully apply for any other supported OS as well. And of course the setup should be installable to the environment of a normal user (no need to wake up root just because you want to add a plugin to your shell).

However as I am very new to Chef and configuration management, some things may not be described 100% accurately, so read this post with two big hands of salt (or two cups of coffee).

Configu-what?

If you are not familiar with configuration management, you can go read it on the Wikipedias. But in a nutshell it is the possibility to have an automated build with one build target which is ‘set up the machine production ready’. As in a classical automated software build, the system knows what needs to be done to complete the build target and can track what has already been done. Therefore all steps are idempotent, which means executing a step multiple times always leads to the same result (and no duplicated resources). Therefore it is important that you treat your configuration in the same way you would treat your automated build: There are no steps executed outside the system. If you force yourself to use your configuration management system for every install and configuration you will see how it simplifies your life, at least when you set up a new machine again. Chef is one implementation for such a management system (other popular choices are Puppet and cfengine). Chef is (mainly) written in Ruby and supports cookbooks written in Ruby itself or the Chef DSL which we will see in a later example.

To the Cloud!!

Chef comes in two flavours: Chef Server and Chef Solo. The main difference here is that with Chef server everything related to your configuration is managed on a server and machines register on it to get their configuration and then perform all actions locally with chef-client. Chef Solo on the other hand is basically a client run where you have to download your configuration manually beforehand. The downloaded configuration is then used by the executable to set up your machine. So in a Solo run there is no external resource involved, but there are also some features which are only available in the server edition. For managing my own configuration I decided if I am going to learn Chef I might as well do it with the full stack. However setting up Chef server is a real hassle as many different technologies are involved and is not really recommended for someone new to Chef. Fortunately Opscode (the company behind Chef) provides a so-called ‘Hosted Chef’ service, which really just means a Chef server in the cloud. And as it is free up until 5 nodes, it is a great way to get started with Chef.

Clients, nodes, knife, cookbook, recipe?

The basic terminology can be a bit confusing (especially as half of the search results usually link to gourmet sites). So let’s try to clear some terminology right upfront:

• Cookbooks: Basic Chef configuration/distribution unit
• Recipe: Subunit of cookbooks. All basic steps are taken in recipes
• Client: A client which connects to the Chef server, level at which certificates are issued
• Node: An actual machine which asks the server for its configuration
• Roles: Collection of cookbooks which can be assigned to nodes
• Knife: Command line client to interact with the Chef server
• Data bags: JSON encoded information which doesn’t fit anywhere else to store

This might still be a bit confusing, but let’s just start with our configuration to see how these parts all play together. The big benefit of Chef (I’m sure it’s the same with most of the other systems), which is also a point which is often discussed as a weakness, is the fact that everything really is Ruby or json. This means it is source code, which again means we can easily manage it with an SCM (I will use git in the examples, but it really applies to your favourite SCM, too). So let’s start with creating our configuration repository:

    $ mkdir chef-repo ; cd chef-repo
    $ git init .

Now that we have our repository set up, we can start to add cookbooks. There are in general two ways to get cookbooks into your repository.

• create the files and folder yourself
• knife (the command line client, remember?)

Knife is definitely the better way as you can create cookbook scaffolds, add cookbooks directly from the community site or use one of the great plugins (like pulling cookbooks directly from Github). But to get a better understanding of the cookbook basics, we’ll create everything by hand now.

The first cookbook

As an example cookbook we’ll want to install oh-my-zsh with our own custom .zshrc. Although this is probably not such a common install as git for example, it is a reasonably easy one and a good example for how to automate steps which would normally be done manually. The steps we want to automate are:

• download and install oh-my-zsh
• install our custom .zshrc

So first of all let’s create the basic folder structure:

    $ mkdir -p cookbooks/oh-my-zsh/recipes
    $ mkdir -p cookbooks/oh-my-zsh/templates/default
    $ touch cookbooks/oh-my-zsh/recipes/default.rb
    $ touch cookbooks/oh-my-zsh/templates/default/dot.zshrc.erb
    $ touch cookbooks/oh-my-zsh/README.rdoc
    $ touch cookbooks/oh-my-zsh/metadata.rb

The rough knife equivalent (which creates all the possible folders for the cookbook) would be knife cookbook create oh-my-zsh -o./cookbooks. However in order to get our oh-my-zsh cookbook working, we only need the files and folders shown above. The README.rdoc and metadata.rb files are just for metadata about the cookbook and only the Ruby file is directly parsed by the Chef server for information. But every cookbook should also contain a README which explains its purpose in a spoken language (you create README files for all of your projects, don’t you?).

In order to setup the cookbook, first insert your current .zshrc into oh-my-zsh/templates/default/dot.zshrc.erb. This makes it available to our recipes as a template file. Now we want to configure the actual recipe. Therefore enter the following into oh-my-zsh/recipes/default.rb:

    script "oh-my-zsh install from github" do
      interpreter "bash"
      url = https://github.com/robbyrussell/oh-my-zsh/raw/master/tools/install.sh
      code <<-EOS
        curl -sLf #{url} -o - | sh
        rm #{ENV['HOME']}/.zshrc
      EOS
      not_if { File.directory? "#{ENV['HOME']}/.oh-my-zsh" }
    end

This just executes the shell script passed to the code directive. The used interpreter is bash and the not_if directive secures the idempotency of this step. The script is only executed if the directory ~/.oh-my-zsh does not exist. The shell script just contains the usual oh-my-zsh installer and removes the generic .zshrc which is important for the next step. As we want to install our own config file but don’t want to do it everytime, we use the following Chef block (written to oh-my-zsh/recipes/default.rb directly after the install script):

    template "#{ENV['HOME']}/.zshrc" do
      mode   0700
      owner  ENV['USER']
      group  Etc.getgrgid(Process.gid).name
      source "dot.zshrc.erb"
      variables({ :home => ENV['HOME'] })
      not_if { File.exist? "#{ENV['HOME']}/.zshrc" }
    end

This creates the file given as the template parameter (our zsh config file) with the given properties. It makes sure the file is owned and only readable by us, takes the content from the dot.zshrc.erb template and passes variables to the renderer. As you might have already seen, templates are just ERB. This means we can use the ERB syntax (<%= var %>) within a template to insert dynamic content passed from the recipe.

One additional step, we might want to take, is source .profile in our config file. This is especially useful if you use environment management like rvm, virtualenv or kerl. These usually need to be activated in the shell config. In order to make sure that they are present in every shell the activation step is written into .profile. Therefore we also want to source it in our zsh config. The not_if method here also conserves the idempotency of the step.

    script "source .profile in .zshrc" do
      interpreter "bash"
      code <<-EOS
      echo "source #{ENV['HOME']}/.profile" >> #{ENV['HOME']}/.zshrc
      EOS
      not_if "grep \"source #{ENV['HOME']}/.profile\" #{ENV['HOME']}/.zshrc"
    end

The server comes into play

After finishing these steps, we can upload the cookbook to our server. In order to be able to do this, the server needs to be set up, so if you haven’t already sign up for a free hosted chef. After creating your organization, put your client and validation certificates in ~/.chef. I find this to be a convenient place for all your Chef related configuration, but you can of course choose another directory (just make sure that you also adapt subsequent steps in this post accordingly). Now we can upload our cookbook with:

    knife cookbook upload oh-my-zsh

We have a cookbook on the server now, but no node uses it, yet (we also don’t have nodes set up at the moment but bear with me here). In order to match nodes to cookbooks Chef employs the concept of ‘run lists’. These are basically lists of recipes which can be added to a node so that it knows what to install. As run lists are mostly very similar between nodes of the same category, we can set up a role for it in Chef. A role is just a specific set of attributes and a run list which is mapped to a name. As there may be multiple machines we use as workstations we create a role ‘workstation’ in the roles directory of our Chef repository:

    $ mkdir -p roles
    $ touch roles/workstation.rb

Again this is just Ruby so we add the following information to workstation.rb:

    name "workstation"
    description "development workstations"
    run_list(
      "recipe[oh-my-zsh]"
    )

Now every node which is assigned the ‘workstation’ role will know that it has to install chef oh-my-zsh recipe. Let’s upload the role to our server:

    $ knife upload role from file workstation.rb

In the management web interface (or via knife) we can now assign the role ‘workstation’ to specific nodes. However we first need a client which is allowed to connect to the server API. Clients and nodes are somewhat the same in Chef. Theoretically it is possible that a client manages a number of nodes, but normally every node corresponds to one client. Therefore we create a new client for our workstation. You can also run chef-client on your node and provide the validator certificate for your organization. If the node does not yet exist on the server it is created. However this means that you have to have the validator certificate (which is the ultimate key to your server) on the node. This might not be a problem for setting up your development machine, but is bad security in general. So the better way is to create the client and node on the server and provide the correct credentials (at least read and update) for the client on the node. One more advantage is that we can now already assign roles to our nodes (via the ‘Roles’ menu) and add the ‘workstation’ role to the newly created node. All these steps can of course also be accomplished with knife, but I find the web management console easier to start with. When all this is done, download the client’s certificate and also put it in ~/.chef. Theoretically your node is correctly set up already. However Chef makes the assumption that it is run with privileges. Therefore the default data directory is in /etc/chef. As we want to setup our development machine and not a server, it makes sense to run chef-client as your normal user. In order to do this, you would now have to make the default directories accessible for your user. But we can also override the paths used in the client config. I also keep my paths in ~/.chef (everything in one place, remember?) so a good adaption of your client.rb might be:

    base_dir = "#{ENV['HOME']}/.chef"
    run_path "#{base_dir}/run"
    checksum_path "#{base_dir}/checksum"
    file_cache_path "#{base_dir}/cache"
    file_backup_path "#{base_dir}/backup"
    cache_options({:path => "#{base_dir}/cache/checksums", :skip_expires => true})

This will make sure only subdirectories of ~/.chef will be used for caching, checksums, etc. After these steps there is only one thing to do.

Sit back and watch

    $ chef-client -c ~/.chef/client.rb -k ~/.chef/client.pem

The above command will run the Chef client with the specified config and client certificate. It will then fetch the cookbooks from the server, determine which to execute via the nodes run list and run them. If everything went well you now have oh-my-zsh installed and can go on and add additional cookbooks to your repository.

Further reading

You should now be equipped with a basic working setup to create your configuration with Chef. Play around with new cookbooks and try to force yourself to do everything system configuration related in terms of cookbooks and data bags. You’ll only learn it by doing it. If you feel comfortable enough with this basic setup, see the following links for some more sophisticated possibilities.

• Managing workstations with Chef
• Encrypted data bags
• Light Weight Resource Providers(LWRP)

[Couchapps][2] are a great way to build web apps hosted directly on a CouchDB. This is due to the integrated HTTP server, so if you can fit your application into the constraints of HTML/CSS/JavaScript, you get the storage (almost) for free. The heart of couchapps is [evently][3], a JavaScript framework simplifying the development of event based web applications. The development process is accompanied by the [couchapp python script][4], which maps a certain directory structure to the evently application layout. This makes it easy to develop the source code, which is normally stored as attachments in the design document, in your favourite editor. However this difference in how the application is developed and how it is deployed, makes it a bit more difficult to automatically test the application.

Enter cucumber

Fortunately the ruby world has provided us with a great tool for acceptance testing web applications: [Cucumber][5]. Cucumber is a testing framework, which encourages [BDD][6] style development. It features different drivers for (headless) browser testing and supports an easy, natural language like, syntax for creating tests (scenarios as they are called in BDD world). If you don’t use it already, give it a try, it is really great. However to fully embrace an automated testing approach we need some helpers to do additional work, for example create and destroy the testing environments.

We have the technology, we can make him stronger

The first problem was the CouchDB native authentication db used by couchapps to profit from the already existing user management. Fortunately there is [a way][7] to change the db used for authentication to an arbitrary one. The next nice-to-have is an easy setup for choosing databases for different environments like tests. Fortunately rails already provides a clean setup for this, which we can copy. Now we only have to bundle a [simple CouchDB library][8] and some helper methods and we are ready to go. This is what [couchapp-cucumber][9] is about. I bundled all these steps as a simple cucumber drop-in. I’m sure it can be improved in a lot of ways.

So fork it. Hack away. And happy testing.

[1]: {{ page.url }} [2]: http://couchapp.org [3]: http://couchapp.org/page/evently [4]: https://github.com/couchapp/couchapp [5]: http://cukes.info [6]: http://en.wikipedia.org/wiki/Behavior_Driven_Development [7]: http://lenaherrmann.net/2010/04/29/security-in-couchdb-changing-the-authentication-db [8]: https://gist.github.com/738128 [9]: https://github.com/mrtazz/couchapp-cucumber

The resulting (localization) software has to run on different platforms such as Windows, Unix and the iPhone. This is why I wanted a build system with continuous integration, where the software is built and tested on these platforms. For continuous integration I normally use [integrity][4] or [cijoe][5], which are simple but therefore don’t support notifying builds for several platforms. [Hudson][6] has support for agents, but as far as I found out, they have to be hudson agents. For my Master’s project I wanted to be able to use integrity as well as cijoe as agents and have a simple central command instance, which notifies them.

Ramrod

This is why I built [ramrod][7]. It is a small sinatra application, which acts as a sort of CI control center. Ramrod can be notified to build a project via simple HTTP POST. All registered agents are then notified subsequently. The agents have to be configured to notify the result back to ramrod, where all the results are then displayed. This can be easily done with cijoe and integrity. Ramrod itself has a simple structure and can be deployed to any ruby hosting platform.

The project is still in a very early stage and there are a lot of things which have to be improved (or even implemented), such as a notification system, authentication and of course a much better design. But I think the initial release v0.1.0 is already quite helpful and usable.

[1]: {{ page.url }} [2]: http://twitter.com/mrtazz/status/24040433982 [3]: http://cone.informatik.uni-freiburg.de [4]: http://integrityapp.com [5]: http://github.com/defunkt/cijoe [6]: http://hudson-ci.org [7]: http://github.com/mrtazz/ramrod

Magic does not come easy

I think there is one big mistake a lot of people do when switching. They expect to start vim and instantly have their fingers dance over the keyboard and coworkers being stunned by the awesome magic. But as with everything you learn, this is not the case. It is a long (think years long and not weeks long) way to even come close to this dance. I started fighting with vim in 2004 and think I have mastered the first 10% now. In my opinion Yehuda Katz was right when he [said][4] that you should really start in insert mode. It turns out, that you can really use Vim (especially [MacVim][5]) like any other editor. From there on, if you force yourself to learn and use a new command everyday, you will see significant speed (and magic) improvements quite fast.

Bundle configuration with pathogen

Another topic which is very important (and powerful) is bundle support. Textmate’s bundles are split into different type in Vim. There are plugins, compiler, syntax and some more folders which can be used for configuration. If you do it the old way, it is really cumbersome. You have to copy new scripts into the according folders in your ~/.vim folder. You have to check that nothing gets overwritten, and after some time you will lose track of what plugins you have installed.

Fortunately, Tim Pope wrote the great [pathogen plugin][6]. You only have to put it into a folder called autoload and enter 3 lines into your ~/.vimrc and your done. Now you can create a new folder under ~/.vim/bundle for each plugin you want to install. Pathogen will automatically load the plugin for you. You can even go further and put your configuration into [git][7]. If you add all of your plugins as git submodule, you can easily update them and have your configuration in sync on all your machines. It’s that easy.

You mentioned Textmate?

Right. I mentioned that I tried several times to switch to Textmate, as all the smart OSX users seem to use it. And if all the smart people use it, it must be awesome right?

After using Textmate for some time, I suffered the same symptoms all the Vim switchers were talking about. Coding was slow, I had to think how I do this and that much to often, shortcuts are way to complicated and I tried to find a Vim mode for Textmate, that worked for me. I was trying to instantly unleash the magic. And that just doesn’t work. While I think Textmate is (one of) the best pure OSX editors and I hope that there will be a version 2 someday, I am still incredibly more productive with Vim. So Textmate is more of a fun to use tool, which I use when I am in the mood.

But there is still this emacs thing out there, I hear. And it also wants to be learned.

[1]: {{ page.url }} [2]: http://www.vim.org [3]: http://macromates.com [4]: http://yehudakatz.com/2010/07/29/everyone-who-tried-to-convince-me-to-use-vim-was-wrong/ [5]: http://code.google.com/p/macvim/ [6]: http://github.com/tpope/vim-pathogen [7]: http://git-scm.com

On the same evening I decided to build a [python library][github_notifo-py], to be able to easily notify users from python applications. I also thought it would be cool to get push notifications from your CI server about your builds. So I implemented an [integrity][integrity_url] notifier which is now available in the main line.

At the moment I personally use notifo to get notified about twitter mentions via [push.ly][pushly_url], website changes via [femtoo][femtoo_url] and commits to some of my github projects via a [service hook][notifo_hook]. Once I have found suitable hosting for integrity for my projects, I will also use the integrity notifier. And I am very excited to see what else will be build upon this service.

[notifo_post]: {{ page.url }} [notifo]: http://notifo.com [notifo_announce]: http://paulstamatiou.com/notifo-yc-w2010-gets-a-co-founder-me [prowl_url]: http://prowl.weks.net/ [growl_url]: http://growl.info/ [github_notifo-py]: http://github.com/mrtazz/notifo.py [integrity_url]: http://integrityapp.com/ [pushly_url]: http://push.ly [femtoo_url]: http://femtoo.com [notifo_hook]: http://github.com/github/github-services

Then, while reading some blog posts about continuous integration and build notification, I stumbled upon [thunk.us][thunkus]. It is a simple status management service. You register your thunk there, poke it to set the status and everyone who needs to (i.e. knows the ID) can check it. It can easily be used to notify about succeeded/failed builds, status of queues or coffee supply or anything else for that matter. Allthough I had no immediate usage for such a system, I read the API definitions and found them to be interesting enough to build something upon it.

Enter [thunkapi.py][thunkus_github]. A python library with command line client to interact with the thunk.us service. I built the library with ease of use in mind (and I think I somewhat succeeded). One can easily provide states, payload and a UID (or list of UIDs where applicable) to the exposed methods and get a clean python dict object in return.

I will still have to see if I have a productive use for the thunk.us service. But the fun of building the library was motivation enough. So have fun with it, use it if you like it and if you dislike something about it: [Fork me!][thunkus_github]

[thunkus_post]: {{ page.url }} [instapaper]: http://instapaper.com [instapaperlib]: http://pypi.python.org/pypi/instapaperlib [thunkus]: http://thunk.us [thunkus_github]: http://github.com/mrtazz/thunkapi.py

The problem is once we are in the supermarket, usually nobody has checked if we still have ketchup or not. And so the guessing starts. This is why I decided to build an easy web application where we can set and check the current status of our ketchup supply. So last weekend I sat down and since I wanted to get into ruby anyway, I built a sinatra app. It is quite simple and works with a token based API, but it does the job. It is hosted on [heroku][ketchup_heroku] if you want to check it out. But you can also fork it on [the githubs][ketchup_github], if you want to improve it or adapt it to something else.

[ketchupstatus_post]: {{ page.url }} [ketchup_heroku]: http://ketchupstatus.heroku.com [ketchup_github]: http://github.com/mrtazz/ketchupstatus

However I also wanted to be able to add links quickly which come from other sources. That is why I wrote a library for instapaper in python and a command line client using this library. For source code and examples see the github page and to install use PyPi.

The results can be seen on [github][plustache] and at the moment I am working on getting to a point where I am satisfied enough with the code to tag it v0.1.0. I am excited to port mustache to a more static language and challenge the difficulties of still keeping it simple. And I am even more excited about how this mustache thing will evolve.

[plustache_post]: {{ page.url }} [@defunkt]: http://twitter.com/defunkt [mustache]: http://mustache.github.com [pystache]: http://github.com/defunkt/pystache [mustache.js]: http://github.com/janl/mustache.js [mustache.erl]: http://github.com/mojombo/mustache.erl [mustache.node]: http://github.com/raycmorgan/Mu [macvim]: http://code.google.com/p/macvim/ [plustache]: http://github.com/mrtazz/plustache

Some months ago I realized that my Mac mini, used as a media server in my livingroom, has been getting a bit long in the tooth. The 1,8 GHz Core (no 2) Duo with 1GB RAM and 80GB hard disk was top notch when I bought it around 3 years ago. But in the meantime I have aquired around 300GB of music, TV shows and movies (thank you iTunes store and handbrake) and the startup time of iTunes seemed to increase every week. It was also getting more and more time consuming to play flash videos, not to speak of HD TV shows.

Another thing that always somehow bugged me, was the fact that I had a whole machine only for serving media content, which just seems a bit too much. Since I don’t watch TV shows all day and also don’t listen to music all the time, the Mac mini was idle around 75% of the day. Sure it doesn’t consume that much power when idling, but nevertheless it always felt like a waste. Additionally there are some Firewire hard disks connected to the Mac mini, since I ran out of internal storage a long time ago (and didn’t feel like replacing the hard disk). Just more negative contribution to my carbon footprint.

My main machine for work and university is a black MacBook from late 2007/early 2008. For coding and longer studying, I hooked it up to an external 22-inch Full HD display and Apple wireless keyboard/mouse. Since I got tired of cabling/uncabling the MacBook, it happened all too often, that I worked on the small screen too long to be comfortable or had the MacBook connected to the display all weekend and missed its mobility. I also more and more often had the feeling, that the external display was a bit too small and that the combination of 22-inch and 1920x1080 resolution wasn’t right for me.

So around September this year I started thinking about how to improve my situation.

Choosing new hardware

The first solution I thought about, was the new Mac mini, which has two display outputs. I could hook it up to the display and the TV and have a machine for working and a multimedia machine at the same time. However, I am not satisfied with the hard disk size of the Mac mini. It is only a matter of time until my media data exceeds 500GB and I am not ready to give up the optical drive for more storage and the Mac mini server.

Then the new 27-inch iMac came out and I was instantly stunned by the display. Also the low-end 27-inch has a 1TB hard disk, which should be enough for some time, a rather strong CPU and a fairly large amount of RAM (did I mention the display?). So it was almost decided that the new setup would include a 27-inch iMac. The following weeks consisted of discussing the solution (thanks @0ktan for pointing out the size of the display several times a day), calculating how to finance it, and making lists to justify not going with the Mac mini variant. After this (for me fairly normal) decision process, I was ready to order and when I was somewhat surprised by a christmas bonus, I knew it was time for a christmas present for myself. So I ordered the low end 27-inch iMac with Apple Care and the new Apple Remote. Only to get notified that it will be delivered 24th to 31st of December (not in time for christmas). The following days I read about broken displays, supply shortage, graphics card failures, but I was convinced that this couldn’t happen to me and kept thinking about a name for it (really the hardest part of all). On the following Saturday I received a shipping confirmation with the 21st of December as the estimated delivery date. It goes without saying, that I was totally excited that the new machine would arrive before christmas, allthough the Apple status site still said 23rd to 29th. And then on the 18th the UPS truck finally arrived.

The screen real estate paradise

After the exciting unboxing, the setup was straight forward as usual. Entering my MobileMe credentials synchronized contacts, calendars, iDisk, and such to the new iMac. I copied my data and applications from my MacBook and the media files from the mac mini and have used the iMac since then almost exclusively (except for some trips to the couch with the MacBook).

Working with the iMac is just awesome. I can now have a browser window, IDE and a terminal conveniently beneath each other without overlapping. iTunes with my whole media library loads in seconds, even when I have a virtual machine running and the whole setup isn’t even that big. Since I have a rather small desktop, I was worried a bit that the iMac would look too big on it and that there wasn’t enough space for books and other stuff. But despite its large screen it consumes hardly more space than the 22-inch display with the VESA mount.

The mini Displayport to HDMI adapter still has to be delivered, so I don’t know yet if it is irritating, if the TV is permanently connected as a secondary display. If this is the case, I will have to manually disconnect the TV when I am working on the iMac. Also because my desktop isn’t that big I have to turn my head a little bit to see the edges of the screen, which can be a bit uncomfortable.

Since these two things can be easily filed under first world problems, I have to say that I am totally happy with my decision and that it is (again) the best Mac I have worked on so far.

For some years now I am using Google Reader as my reader of choice for news feeds. I still think it’s the best way to get daily news (for me), although I do recognize that I don’t read every feed as thoroughly as I used to anymore. I always use the starred items section as a sort of “save for later” storage, so I can open all the unread stories in my browser tabs and read them when I have time. This worked out ok, but in the last months I got more and more used to the instapaper service, which provides a simple web frontend to save websites for later and a great iPhone app. This led to the situation, that I still starred items in Google Reader, but then had to open the pages and manually save them to instapaper.com and unstar them in Google Reader. An alternative is the builtin Google Reader option to share to instapaper, which also needs some manual actions and you still have to unstar the item afterwards.

The solution

So I decided to build a script, which would run nightly on a server and pull all starred items out of Google reader and into instapaper. Since I am trying to improve my python knowledge whenever I can, I decided to build a python implementation. Since I also wanted to dig into the (unofficial) Google Reader API, I decided not to use the very good existing python framework, but to utilize the API myself. I had already written a small instapaper library before, which I mainly used to save articles for later on the command line.

So I sat down and started to write the bridge. All in all it took me a bit more than a month (due to university life, work and exams) to finally get a basic version working, which is able to retrieve starred items, save them to instapaper and then remove the star from instapaper. The script is now run every hour on a server, which is enough for my needs of instapaper being up-to-date. Also I have more choices on iPhone RSS readers with Google Reader syncing now, since Instapaper support is not a requirement for the app anymore. There are still (as always) some things to do, but at least the manual open,save,unstar actions are no more now.