Julius Seporaitis

On Large Language Models

2023-04-02T00:00:00+00:00

I’ve been asked about my thoughts on Large Language Models (LLMs), so I’ve decided to put together a blog post to share my reflections. These opinions are not unique; they are based on social media commentary, newsletters, blog posts, videos, and my own experiences with LLMs like GPT v3.5 and v4 for work and personal projects.

Hype Cycle

We are currently experiencing an unprecedented hype cycle, and with few signs of the Trough of Disillusionment, it appears we are still approaching the Peak of Inflated Expectations. This is largely due to OpenAI’s increasingly impressive GPT models, which are truly remarkable and should not be dismissed or underestimated.

By Jeremykemp at English Wikipedia, CC BY-SA 3.0.

The current hype has led to concerns about AI-based tools replacing jobs, particularly in software engineering. My own anxiety subsided after using the models, realizing that while they are great tools to assist with my work, they cannot replace a skilled software engineer. Here’s a relevant video:

The author’s conclusion lacks self-reflection, as ChatGPT’s output shouldn’t be considered senior-level. A junior engineer could produce similar results with proper documentation and guidance from an experienced engineer. While it’s impressive that the machine learning model generated the output from a natural language request, it was directed by a professional - who the show host clearly is.

Real Life Trends Flatten

In the book Factfulness, Hans Rosling discusses the “straight line instinct,” where people assume upward trends will continue indefinitely, which is often not the case. ChatGPT’s performance will eventually plateau, as there are limits to the improvements a machine learning model can achieve from a given dataset. The focus will likely shift towards model specialization or expanding its capabilities to include visual, audio, or even tactile/sensory input.

Specialization is already evident in products like Copilot X, a ChatGPT v4-driven programming assistant, and in the integration with Microsoft Office for transcribing online meetings and drafting presentations. Expanding modes of operation can be seen in ChatGPT’s latest version, which supports image interpretation.

Training Data

We can't all use AI. Someone has to generate the training data.
— Paul Graham (@paulg) March 14, 2023

The performance plateau is influenced by training data. Commercial datasets are often licensed rather than sold, effectively as a subscription for dataset upkeep. OpenAI likely relies on such datasets, which carries a risk of license revocation. A similar situation occurred when movie rights holders, like Disney, recognized the potential for their own streaming platforms and revoked licenses from platforms like Netflix.

I don’t suggest everyone will create their own LLMs, but data providers may seek a share of OpenAI’s profits through licensing fees. Alternatively, a competitor could secure exclusive licensing, limiting OpenAI’s data access. While hypothetical, such scenarios could have lasting effects on the product.

Context Window

I conducted an experiment with ChatGPT on my pet project, inspired by Simon Willison’s ReAct pattern reimplementation. I provided commands for the model to list files, read files, and apply a diff within my project directory, and instructed it to follow a Thought, Action, PAUSE, Observation loop. I asked ChatGPT to add a simple feature involving installing a dependency, updating a model, and modifying a form template. The following results were consistently observed after multiple attempts:

ChatGPT inferred the need to read the file, but once provided, it forgot the original request, regardless of file size optimization. The file was a Django models file, initially 12KB and ~250 lines, which I reduced to ~2KB. ChatGPT has a limitation of the size of input it can process, called the “context window.” This “context window” affects its ability to remember experiences beyond a certain input size, restricting the model’s capacity to recall prior instructions.

These limitations may not be fundamental, and could be overcome eventually. However, if the context size depends on finite resources (e.g., server memory), increasing the context window to handle larger inputs could raise the service cost, potentially making human labor more economical. Of course, I am speculating.

Experience And Tribal Knowledge

In software engineering and other fields there is a concept called “tribal knowledge,” something that is known but not explicitly documented. For example, tribal knowledge represents hard-earned lessons accumulated over a career, encompassing individual projects and skillsets. By definition it is difficult to train models on it. Some information may be implicitly encoded in code, but without sufficient documentation explaining ‘why’ certain choices were made, the model can only guess. Humans have a significant competitive advantage in this area over machines.

ChatGPT’s current context window is 32K tokens (about 750 words per 1000 tokens). In contrast, my context window spans 23+ years as a hobbyist and 16 as a professional. While my recall is inferior to ChatGPT’s, my precision in distinguishing good from bad answers is higher. Together, we form a more effective combination than either of us alone.

Groundhog Day

Observing the startup scene over the past 15-20 years, I’ve learned that disruptors often become the companies they disrupted. Examples include London cabs now being more accessible than Uber, Netflix contemplating ads in shows just like cable TV, Airbnb resembling Booking.com with added cleaning fees and faceless corporate hosts, and Facebook evolving into an oversized bulletin board. In time, OpenAI will transform into a mishmash of the companies it will replace.

Natural Language Is Not A Programming Language

Early in my career, writing specifications for software engineers taught me that natural language is not the ultimate programming language. Asserting otherwise disregards decades of computer science research. Natural language is inherently ambiguous; while you could request “a TikTok for restaurant chefs,” the initial result likely won’t meet your expectations. Adjusting the app may resemble a product manager’s role, but there’s a point where using a context-free language, a programming language, is more efficient. A human programmer will be needed to translate requirements into code accurately the first time, even if not necessarily faster.

Prompt Engineering

Initially, I found prompt engineering to be a dubious idea, then intriguing, and now I’m uncertain. If language models remain black boxes, prompt engineering could resemble astrology or alchemy, a pseudo-scientific approach to coaxing desired outputs from ML models. However, if OpenAI offers integration points centered around prompt engineering, like rapidly testing significant words or conversations, a burgeoning new industry could emerge. Until then, sustaining investment in prompt engineering roles, which currently involve guesswork, may be challenging once we pass the Peak of Inflated Expectations.

Translators & Dictionaries

Google Translate, like ChatGPT, was a remarkable accomplishment. While it didn’t replace human translators, it replaced dictionaries and made information more accessible globally. Similarly, ChatGPT won’t replace humans but will enable them to do more. However, now may be the worst time to be “the dictionary.”

Career In Software Engineering

In my opinion, the software engineering career path is secure, and it’s now easier to begin. Previously it was a rite of passage. Finding answers required attending university or extensive online searches and reading, but now, answers are a prompt away. However, it’s crucial not to take LLM-generated information at face value and to cross-check. LLMs can confidently provide misleading information, and even ChatGPT 4 veers off-course in 30-50% of my own requests, necessitating further instruction or rephrasing.

For example, I am comfortable delegating simple tasks to LLMs when I can easily verify the output. As Andy Grove said, “delegation without supervision is abdication.” For instance, I asked ChatGPT for a regular expression to create a complex input validator in a Django project. I tested the provided regex and asked for edge cases, validating them. ChatGPT generated a validator class that required tweaking. After adjusting and testing, the task was complete in 15 minutes, significantly faster than if I were to do it from scratch. This allowed me to move onto more engaging tasks, making it a win!

Fetishisation of AI

There’s a recurring theme I call the “fetishisation of AI,” which assumes AI is or will be better than humans in every situation. This belief is evident when people think decisions are correct simply because AI made them or when they assume AI will always be the preferred choice. For example, a recent article suggested the demand for human software engineers would collapse as AI could fulfill everyone’s needs. This notion is false, as previously discussed regarding the limitations of natural language for building software.

Banning LLM Development

As I was writing this, a group of notable people signed a petition for OpenAI to stop developing AI. I categorically disagree with the basic idea, regardless of who is calling for it. Banning is a reactive response. It is impossible to “ban and investigate the impact.” The only way to investigate the impact is to go through the development process, carefully assessing the results at each step. That is a proactive approach.

Threshold For Concern

I’ll become concerned when an LLM can create a product without active human direction, seeking feedback along the way. At that point, it will be a fully autonomous agent, and it’s unclear what will happen. We may end up building tools for AI to access more information rather than building tools to help us be more productive.

Someone is already experimenting with this, but the results are limited:

Huh, well that's starting to work... 🤔

Kind of just goes on... and on... building and reprioritizing a task list for itself... pic.twitter.com/whcsSNdBHz
— Yohei (@yoheinakajima) March 26, 2023

The Future Is Bright

LLMs will democratise programming as a craft, and will lower the entry bar for a new generation of entrepreneurs. If you find programming hard, but have enough patience to describe what you want and the skills to glue the bits together - you can build any prototype faster than ever.

LLMs will augment software engineering to a new level. There will undoubtedly be bumps along the road, but in the end we will use ChatGPT more than we will be abused by it.

LLMs will be the world’s most patient but unreliable teachers. As long as you are willing to verify what it says is true, you’ll learn.

I also must make a confession. I wrote this article and a good friend reviewed it and provided feedback. At the same time I am an ESL (Eglish as a Second Language) speaker and I am very conscious that my writing can be excessively verbose, repetitive or non-idiomatic. So you can feel where this is going - I asked ChatGPT to make each paragraph shorter and more concise, also explaining to me the changes it made and why. I still had to tweak the result, but I think overall it did a great job.

Therefore, maybe just like Google Translate made the world’s information more accessible to everyone, one of the best use-cases for LLMs will be, to make everyone’s ideas more accessible to the world.

2022 Year In Review

2022-12-23T00:00:00+00:00

A friend asked me if I was planning to write a review of my year and, whilst I hadn’t considered it myself, I thought it would be great to do and revisit it in a few years time.

Family

Families meeting. Originally planned for 2020, but delayed, mine and H’s parents finally met for the first time this summer. I had a bit of anxiety about it, because personality wise they couldn’t be more different. My parents are Lithuanian, quiet and reserved. H’s - English, loud and social. However, everything turned out better than I expected. There were a lot of conversations, food sampling, and travelling. I managed to learn something new about a brewery in my hometown, and the weather was balmy 30+ degrees throughout.

Museum Hits Hard. One of the places we visited with H’s family in Vilnius was what the locals call the KGB museum, because it is located in the former KGB headquarters. You have to see the torture and killing chambers to understand the pains the Soviet occupation has caused, but that’s not the point of this reflection. While walking through the exposition, I saw a piece my, now deceased, grandmother had gifted to the museum. A mosquito net she brought back from her exile in Siberia. I didn’t know it was there. I did not expect to find it. And I cannot even begin to describe how hard it hit emotionally. Suddenly, the whole thing was 100 times more personal and I cried quietly in the corner.

War. Even though I don’t have relatives in Ukraine, the war still affected me in many ways. First, I am from the Baltics, and we have a turbulent history with Russia. I grew up reading and hearing up close stories about Soviet torture and cruelty during WWII. Reading about the same things happening in Ukraine in 2022 made it up-close-and-personal, and it was very clear who would be next if Ukraine fell. I donated 500 pounds in the first few weeks. Secondly, there was a lot of anxiety both for me and back home. I had a very hard time concentrating on work, but learned to handle it about a week later. I am still keeping up to date with the news daily, and sometimes find it unfathomable that someone would be willing to tell Ukrainians to suck it up and live under Russian rule. It doesn’t work like that - the choice is between freedom and unimaginable cruelty. Through work I have met people from both Russia and Ukraine. No-one wants that conflict to spillover at work, so it does require compartmentalisation and non-judgement. Not easy.

Pets. 2022 marked my first ever full year of owning pets. H had guinea pigs, I grew up without any pets (well, except a fish tank). In September 2021 we bought a coop and got three pet chickens in our garden. Aside from getting eggs, I also learned a few new things. Hens are easy maintenance, high clucking return pets. Hen hotels are a place to drop off chickens when going for a long holiday. Hen x-rays are not worth the price. And, of course, hens had flockdowns, before it was cool.

Hobbies & Projects

Piano. This year I completed my practical grade 1 exam with distinction. Considering I had zero musical training or talent before that, I am quite happy with the result and will continue to study for grade 2. Discussing with friends, it turns out I managed to avoid a discouraging piano teacher that put off a friend from studying further. I shall expand on this in a separate post.

Beer. After a long break, I brewed beer again with H and friends A and L. It is a delightful, if not a bit messy, activity. It can be quite cost effective too. Unfortunately, I had a bit of a health issue and it meant that we gave away most of the bottles as gifts.

Newsletter. I decided that I want to make it easier for people to find my tech related writing. I launched Code Anthropology at the end of July and I still don’t know how to describe it: it does not have code examples, but I don’t think it’s engineering leadership advice either. It’s a mix of learned truths (Everything is Negotiable, Moments of Invention) that I hope can help others. My most popular post was about noticing layoff signs, which I wrote because I saw layoffs happening in the industry, but also because I saw worrying signs at my own company. Ironically, I cannot claim I “predicted” the time accurately. It happened at my company sooner than I thought - in November.

Career

Full circle. Back in 2020 I joined Lyst with a goal to get feedback on and grow my leadership skills, thinking I’d like to move towards the engineering management track. I learned to lead a team, and also how to give feedback and sometimes have unpleasant but necessary conversations. As a team we solved a challenge other people tried to solve multiple times before, and I was very proud. But probably the most heartwarming piece of feedback was the one below. Knowing that just a few years ago I struggled with and lacked confidence in the leadership aspects of the job, it makes me thrilled to see the improvement.

By the end of this year, my hands started itching to build stuff. A small burnout episode in October allowed me a short break to reevaluate priorities, and I made up my mind that I want to go back to being an individual contributor. Maybe Charity Majors had it right - the manager/IC track is a pendulum.

New Start. The above also led me to find a new job, where I will start in February. It is a much smaller startup, in an area that I hope to learn a lot (mortgages), and with a very friendly team that I have already had a chance to meet briefly. Not to mention working again with G who I enjoyed working with in the past.

Books

Turn The Ship Around and Orbiting A Giant Hairball are two books to which I attribute my recent success at work to. I found them through Sourcegraph’s employee handbook. I love the product and wanted to understand what makes the company different (they’re very open). The gist of the first book is that to allow people to be the leaders you want them to be you have to trust them and allow them to make decisions at their level of responsibility. This is in contrast to leadership directing the teams to execute on their decisions. The second book is a collection of stories about corporate culture dysfunctions and how to deal with them without sacrificing your soul. My one sentence summaries don’t do justice.

Death Notice. An English translation of a chinese detective story, recommended by a friend who lived in Beijing for the last few years. The story is about a serial killer who always sends out a written notice letter to the victim, and manages to outsmart the police traps each time. It is not something I would’ve picked up myself in the book shop, but I think that fact made it better. The plot twists were great, some of the dialogues were a bit cringey, but my friend assured me that it was more due to cultural differences.

The Partisan. A book I noticed by complete chance in a bookshop in Durham, and a last copy they had on window display. It piqued my interest because ‘partisan’ is a word for Lithuanian WWII freedom fighters, people who tried to fight back against Soviet occupation until 1953. Most were caught and killed by the KGB, so it is not a surprise that Lithuanian novels about partisans always end in tragedies (“… and then they were captured and killed.”) However… Author of ‘The Partisan’ is a British journalist. He did not inherit the collective mourning, but took creative liberty and turned the script into “what could the cold war have been if KGB didn’t kill them all” And I can tell you, ‘what could’ve been’ in this book was a fantastic cold war spy thriller, with well paced action, very believable characters (including many Lithuanian idiosyncrasies), and entertaining throughout. No pressure, but after this debut I hope the author will write a sequel.

Travel

Paris. Fun fact, I studied French for 6 months in 2015, but have not been to France. It has been a running joke that I should visit and order a baguette, to get my return on investment. Well I finally visited this year and ordered not only a baguette, but two crepes too. I managed to avoid the Paris syndrome, visited some tourist traps and did aimless wandering through the streets with H stopping for coffees or wine and cheese.

Cumbria Way. Ever since I’ve walked the Great Glen Way, I’ve been a fan of multi-day hikes. The Cumbria Way is in the Lake District. It was another delayed trip from 2020, and the first one together with H. Apart from the stunning views, the highlight was learning what part-baked rolls are and going, like some criminal, to a random pub bar in the middle of nowhere and whispering “Hey, do you think your kitchen could bake these for me?” Luckily, they were friendly enough to do that, and we got our lunch for the next day sorted.

Scotland. Scotland is my favourite place in the UK. I have visited the Highlands a couple of times, but I have never been to Edinburgh. Last year one of our friends moved there so it was a good time to visit. I have to say I loved the city, the views from Arthur’s seat and how close the sea was, but my heart is still in the Highlands, which we visited towards the end of the year, as H’s former colleague has moved to a small place near Oban. We stayed in their parents house, with a bedroom overlooking the mountains. I dream we could live there too. However, dipping into a loch in October and then clawing myself back on a coarse pier asphalt is not the most enjoyable experience.

Sports

Personal Achievements. First, I reached my 25th Parkrun milestone, and did 17 parkruns in total. That’s about one third of Saturdays this year. Second, I also ran in the “world famous” 10k run, which gives you a Christmas pudding at the end of the run. Third, I dipped into the sea in January. It must’ve been freezing cold, and my brain has blocked those memories. Knowing myself, I probably complained a lot about the cold.

Cricket. I saw my first live cricket match at Lord’s in London. Interestingly, for the lack of action that’s happening on the pitch, the match turned into a nailbiter in the last 5 minutes. It was a fun experience, but I did not become a big fan after it.

Gigs

Idlewild. A rock band from H’s teenage years.

MEUTE. A strangely intriguing brass techno band from Germany H heard before.

John Legend. I have always been a fan of his lyrical songs, but now I also know that he is adorably cheesy.

Adele at BST Hyde Park. I can tell you that walking on stage while sipping from a tea mug is a proper flex.

What’s Next

I don’t have concrete goals, they always slip through the cracks anyways, but there are a couple of things I am excited about. Me and H will go on a three week trip to Vancouver next year. I would like to do my piano Grade 2 exam. And I would also like to try and sell something online. I am not yet sure what that will be, but have plenty of notes with ideas.

Let's Subscribe To Code Anthropology

2022-07-31T00:00:00+00:00

My recent posts (and drafts in the pipeline) explore the human side of software engineering, the decision making behind Kubernetes, the impostor syndrome, the self-learning, and the psychology of technical interviews.

I enjoyed writing them. Often I feel like a bursting bubble, wanting to speak out or rant. I write it, tidy it up, sometimes get editing help, and end up with a blog post that, when shared, gets some views and either private and public feedback. I enjoy it.

However, this blog is built on a domain name that is hard to remember. The blog posts are also not knowledge based, but idea based - meaning the return rates are low, unless I keep regularly writing and sharing. And of course, the age of personal blogs disappeared with Google Reader.

This is why I decided to try and do a small personal rebranding exercise, to see if using a modern medium could grow a bigger audience. I will post my new content on a new Substack newsletter - Code Anthropology. It is free and it will cover the same themes I have been covering before - the human side of software engineering and our relationship with code, tools, and processes.

I will start publishing soon, meanwhile - please subscribe, share and read the introductory post there.

The Reason Kubernetes Exist

2022-07-27T00:00:00+00:00

A book I read - Software Engineering at Google (check out my reading list) - had an interesting snippet (emphasis mine):

<…> For example, in 2002, Jeff Dean, one of Google’s most senior engineers, wrote the following about running an automated data-processing task as a part of the release process:

[Running the task] is a logistical, time-consuming nightmare. It currently requires getting a list of 50+ machines, starting up a process on each of these 50+ machines, and monitoring its progress on each of the 50+ machines. There is no support for automatically migrating the computation to another machine if one of the machines dies, and monitoring the progress of the jobs is done in an ad hoc manner […] Furthermore, since processes can interfere with each other, there is a complicated, human-implemented “sign up” file to throttle the use of machines, which results in less-than-optimal scheduling, and increased contention for the scarce machine resources

This was an early trigger in Google’s efforts to tame the compute environment, <…>

This led to Google developing the Borg cluster management system, which influenced the design and development of the open source successor that we all know today - Kubernetes. Why is that interesting?

I enjoy uncovering the reasons for systems existence - the ‘why’ behind it, the original problem. It is a part of the same mindset of trying to deepen understanding I wrote in my fundamentals article. It is not about where I am going to use something, but where it is, or in this case - was, used.

By now there are so many articles on the internet written on why one should use and how to use Kubernetes. From a one person blog to a small startup, to beyond. It is very easy to get an engineer’s FOMO - everyone seems to be using it so I should too. We get onboard, the brakes are off and we ride the hype train like there’s no tomorrow.

Meanwhile, it would always be prudent to consider the original problem some system tried to solve, before committing it as a solution to another. If for no other reason than to confirm the original problem or environment is some permutation of ours. Sometimes potentially other viable solutions might surface too.

Just from the snippet above it can be seen that the original problem was scheduling, resource constraint management, automated recovery and execution of tasks across a large fleet of machines. Which should immediately raise a question - is this the only way these problems can be solved today? Depends on the environment.

For a company that does use a lot of resources, runs many long running batch jobs, has a large engineering team, or many services - it may as well be the best long term choice.

Take for instance, my current workplace - Lyst. It has an engineering team of ~100 people, many different workloads, and services that each team is responsible for managing. By the time I arrived, it had been using a largely obscure container orchestration tool that had been deprecated, and had quite a few limitations. Our platform team executed migration to Kubernetes and it was a massive improvement, from engineering productivity to easier platform management.

On the other hand, for a company that just has a website, runs very few long running jobs, with a two-pizza sized engineering team and one or two services, and not a lot of clients - an early startup - it is objectively not the best choice today.

Before I get written off as the next Kubernetes hater, let me redeem myself, because I’ve been in this situation. What is important in the small company, assuming it is a startup, is the speed of development with very low maintenance overhead, and optimizing for growth means having a plan, not deploying a Google level stack on day 1.

As an example, this was my plan when I worked at Genus AI where we used AWS:

AWS Lambda. Was good early on, when the product was not massively complex. It allowed shipping code fast and limited the infrastructure maintenance cost. The start-up was new, B2B, so while we were working out the product-market fit, the costs were low.
AWS Fargate - when product grew and we needed more complex & long running tasks, or when Lambda would’ve been more expensive and the cost trade-off no longer held true
AWS EKS - I left before this stage, but before I left I was considering this as the next step, when the company gew to multiple engineering teams and I’d needed standardized tools for them, instead of asking the teams to fiddle with AWS directly. As good as it would be, not everyone knows how or enjoys that kind of work.

This shows how trying to find the deeper, fundamental, reason behind certain systems, can yield better results, more creative and adaptable solutions, than following blindly and taking a premade solution with the whole maintenance baggage coming along with it. Beyond Kubernetes, this mindset allows me to shun all mentions of ‘don’t look here - this is legacy system,’ do some code anthropology, and where someone would jump at a chance to rewrite a legacy system from scratch, I see an opportunity to evolve it. Which reminds me of another book and another quote:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

The infamous Gall’s Law - the reason kubernetes exist.

There Is Always Someone Walking The Same Path

2022-07-25T00:00:00+00:00

During our careers we go through multiple evolutions of: I need to do something, I learn, I do it, then move to the next thing.

During the learning phase we meet people who are ahead of us. Peers, friends, people who post online - they always seem to know more than us. The more we learn and become better, the more we realise there is more to learn.

It is easy to become intimidated - there is always someone ahead of us.

In those moments of doubt it would be wise to remember that it also works backwards. Wherever you are in your career - starting out or experienced veteran - there is always someone walking the same path behind you. Looking at you with the same eyes, hungry to learn.

Do it. Share your experience. Level others up.

Teach, speak, mentor, document, or blog about it, and you will always have a reader or a listener.

Copy And Learn, Don't Paste

2022-01-25T00:00:00+00:00

Last year I wrote about what to learn if you already know something about programming (On Fundamentals). But what do you do if you are just starting? How can you tell what good code looks like if there is no mentor around to help you? I recently reflected on my own path (I self-taught programming as a teenager), and here’s a piece of advice that I don’t hear very often.

The idea is embarrassingly simple - whatever you try to learn, be it a programming language, a framework, a library - pick an existing open source project that uses it, and try to rewrite it. Open two editors side by side, one empty and one showing the reference code, and go ahead. Copy it, rewriting line by line. On the surface, this sounds boring, but I assure you the benefits are worth it.

The standard advice when learning programming is to think of a small project to write from scratch. That is perfectly fine, but if this is your first endeavour into programming, you will very likely underestimate how long something will take, and thus lose focus. Not finishing small projects can be demoralising and eventually suck out enthusiasm and motivation (“This is too hard for me.”) Even if you do - there is no good reference point to judge whether the code is of good quality.

Now consider an existing project, written by more established engineers, maybe open-source. Immediately you get the guardrails. Whatever you (re)write will have a clearly defined finish line and will be of a comparable quality. Sure, you may not pick up on details immediately but gradually, as you do this copying and later as you write your own code, things will fall into place. You will find yourself having “Aha, that’s why!” moments over and over again.

Pasting the code would be useless, but the slower rewrite process gives more time and space to ask better questions. Instead of “How do I write it well?”, which for a beginner may be hard to judge, you start with good code and ask, “Why is it written the way it is?” Go on to documentation, search opinion pieces, maybe even politely ask the author. It will help you better internalise why the code works the way it does.

Everyone in their programming career stumbled upon some terrible code and thought. “What is this piece of spaghetti?” or “Who in their right mind has written this?” Starting programming by copying and understanding other people’s code teaches appreciation and humility. If you are a beginner reading an established project and the code does not make sense, the chances are higher that there is something for you to learn.

This becomes very useful later when joining an established project. Old code is often written off as a haunted graveyard with sweeping statements such as, “Oh, that legacy system is a mess, better not touch it.” Copying existing code teaches us how to navigate “other people’s code.” Being able to quickly grasp the structure and navigate through someone else’s codebase is truly a beginner’s superpower.

Copying other people’s code can teach you to connect theoretical concepts to their practical application. Object-oriented programming, functional programming, typed code vs untyped code - all sound good in theory. Applying them without fully understanding the practicalities will not teach much. However, through rewriting the code you get a chance to see them all interacting. And while you may not get the theory at the start, you’ll understand the practical application.

The original code is someone else’s, but the one you rewrite is yours. You completely own the creative licence. If you feel like renaming a variable, class or function, go ahead. This simple deviation will teach you how to trace that part throughout the code, because you will want to keep the naming consistent. Those skills are practical and valuable in real world situations.

With time, your confidence in your own skills will grow. You might think, “I know how to do this part better.” Excellent! Go ahead and do it! One of two things will happen: you will improve the code, or stumble into why the original code was written the way it was. Both are valuable lessons.

There is another way to raise the learning bar. Consider, if you were to start building that reference project from scratch, what would you leave out or keep in the first version? Your learning will be twofold. First, you will learn to strip the codebase to the bare essentials. Second, you will learn how to evolve the code so that with each incremental step it continues to work and is better than the previous version. One of the common denominators in the industry nowadays is “small incremental changes”, as opposed to “large, big bang, once a month changes.” How to be flexible with this change size while still keeping the code working is another superpower that can be learned early.

Finally, having a working reference project guarantees the end result is functional. You do not have to understand everything all at once. You can pick and choose, reimplement components or a small part of the system, and copy the rest as is. You will still learn.

My Personal Path

I got into programming wanting to be a game developer, and back in the late 1990s, there were not many books and enthusiast forums. There weren’t many opportunities to find a mentor for a teenager in Eastern Europe. I spent a lot of my learning time rewriting any code I could get my hands on (open source wasn’t as proliferated as it is now) and talking with other enthusiastic people.

Fond memories: I rewrote something called ‘The Nebula Device’, adding class prefixes. I also got my hands on the Half-Life 2 leaked codebase. It was my first insight into the structure of an actual commercial codebase. Each of these were followed by my own attempts at writing something similar, mimicking aspects of all the systems I copied to learn.

On occasion, when I tried to improve the code, I would soon run into a problem and understand why it wasn’t written the way I thought was better. I also did not understand the reference code fully; for example, I couldn’t grasp the difference between heap and stack memory, or how pointers worked in C/C++. I learned those concepts years later in a computer architecture course at university. University enabled me to connect the theoretical parts to all my practice until that point.

The aspects I understood I could command and use for my own purposes. The aspects I did not understand - I copied. I assumed that if people smarter than me wrote code a certain way, there must have been a good reason. Eventually, I understood most of those reasons and the trade-offs behind them.

One of the most frequent pieces of positive feedback I received early in my career was how quickly I could find my way around a new codebase. I still rely on the skills I learned from copying today. These include:

the ability to break down changes
the ability to build a working system in incremental steps (I wrote on this topic here and here)
lack of fear of legacy systems
a way to know how to ask questions and find answers in documentation
how to adjust my coding style to whatever codebase I am working on

This experience also inspired my love of Chesterton’s fence principle. Put simply, “don’t ever take a fence down until you know the reason why it was put up.” And partially why “boring technology” is so appealing - understand existing systems first, before building new ones.

Final Thoughts

The internet of the 1990s and early 2000s was much more innocent. There was no GitHub, and there was less entitlement; more collaboration and implicitly positive intent (at least in the places I used to frequent). The reason I say this is not nostalgic but more as a word of caution.

If you were to plainly copy a project in public and advertise it as your own without stating your intent, you might get bullied on Twitter or GitHub. At least that’s how I see it if I were to do something similar now. I would keep the code private and just for myself. Maybe this is primarily why no one talks about this way of learning.

This concept works in a commercial setting as well. If you are overwhelmed by legacy systems at work, trying to rewrite one line by line might give you a much deeper understanding of how it works and how to mend it to your will.

Good luck copying and learning!

What Can 75,000 Pull Requests Tell?

2021-07-19T00:00:00+00:00

Disclaimer: This is a cross post of the blog post I wrote on my company’s engineering blog. It is here for my own record. If you like the post and want to share it, you should share the original post.

Every last sprint of the quarter at Lyst is an innovation sprint. It varies between teams and individuals, but it can be anything from paying up some tech debt and trying out a new library or tool to building a prototype for a new idea. We chose to do an exploratory analysis of a large sample during the most recent innovation sprint, about 75K, pull requests (PRs), made between 2015 and 2021. This post covers the results.

Code review practices are a subject of continuous academic and industry research. Lyst has a significant amount of code review metadata available. Putting the two together makes it an interesting project to compare and see if there is anything eye-watering wrong (spoiler alert: there is not). Discussing this project with colleagues, we also came up with other interesting questions to be queried from the data.

Data Acquisition

The data acquisition was a two-step process. First, raw data was scraped from Github API using a small wrapper library ghapi and written out in a directory tree matching the API call paths. Second, a utility script was reading the raw data from the filesystem and inserting it into a single SQLite database file.

The two utility scripts are relatively simple. The main caveat for scraping Github API is to conform to the rate limits. It meant the process took longer, and I had to choose what to scrape first (users, teams, repos, and PRs) and what next (comments, reviews, file changes). Saving responses onto the file system made it easy to restart the process without re-scraping repeatedly.

The SQLite database was an excellent container for the data. Tools like datasette or Pandas read_sql function could be used to run and debug queries and extract exactly the data we needed for various plots.

Comparative Analysis

The comparative analysis builds on top of research about code reviews done by Google and Microsoft Research divisions in collaboration with researchers from universities. We tried to recreate and compare the majority of Rigby and Bird’s¹ metrics about code review practices. Rigby and Bird analysed various projects in diverse industries and observed code review practices. Their primary conjecture is that if peer review practices have naturally settled to some common characteristics across a variety of different projects, such elements may indicate of practices that represent generally successful and efficient review methods. They called them convergent practices (CPs), defined as follows:

CP1 - Contemporary peer review follows a lightweight, flexible process.
CP2 - Reviews happen early (before a change is committed), quickly, and frequently.
CP3 - Change sizes are small.
CP4 - Two reviewers find an optimal number of defects.
CP5 - Review has changed from a defect finding activity to a group problem-solving activity.

Another research we tried to recreate is based on the modern code review study done at Google². This case study expressed the convergent practices through metrics derived from the code review metadata. In our analysis, we did the same. Yet, some metrics for Lyst were difficult to derive due to the insufficient granularity of data. Where possible - a less strict or wider-ranging metric definition was used. For example, the scraped data does not include information about resolved review comments, but “the number of comments per PR” is a looser definition, still indicative of how lightweight the code review process is.

Change Review Frequency (CP2)

One of the emergent practices is that code reviews happen early, quickly, and frequently (CP2). This can be quantified by the number of PRs opened per week per engineer.

At Lyst, the median developer opens about 3 pull requests per week, and 80% of developers make up to 5 PRs per week. In comparison², at Google, the median developer authors about 3 changes a week, and 80 percent of authors make up to 7 changes a week.

Next, the number of code reviews can be quantified. It is a pull request review approval/rejection submitted by a currently active user, except the PR author. A single pull request may contain more than one review from the same person, and they are counted individually.

At Lyst, the median engineer reviews about 3 pull requests per week, with 80% of engineers reviewing up to 6 pull requests per week. In comparison², the median for changes reviewed by developers per week at Google is 4, and 80 percent of reviewers review fewer than 10 changes a week.

Review Speed (CP2)

Bird, Carnaham, and Greiler, based on their experience building code review analytics platform³, define two metrics for code review speed - time to first response and time to completion. Time to first response is a measure of how much time passes between when the author submits a change for review and when the first comment or sign-off from a reviewer occurs. Time to completion is the time between when the author submits a change for review to when it has been marked as completed. At Lyst, similar metrics can be defined too. Time to first response - time between when a PR has been created and when the first review occurs. Review being a comment, request for changes, or approval. Time to completion is the time between when a PR has been created and when it has been merged.

At Lyst, the median time to first response is within an hour, and 80% of PRs get their first review within 24 hours.

The time to completion at Lyst is 12 hours for 50% of pull requests, and 80% of pull requests get merged in 89 hours (or ~4 days).

At Google², the overall (all code sizes) median latency for the entire review process is under 4 hours. This is significantly lower than the median time to approval reported by Rigby and Bird¹ in other projects, which is 17.5 hours for AMD, 15.7 hours for Chrome OS, and 14.7, 19.8, and 18.9 hours for three Microsoft projects.

The Google case study further breaks down the time to first response based on the change size. The median time for small changes is under an hour and about 5 hours for very large changes. The research paper does not define what is a small or very large change. Therefore, we picked reasonably sounding numbers for what is a small or very large change at Lyst and tried to compare time to the first response.

Let’s say a small change at Lyst has the number of lines lower than the median size of 30 lines. The 80% (80th percentile) of all changes at Lyst are smaller than 175. This number can be used as a lower boundary for very large changes. With these definitions - the median to first response for small changes at Lyst is still within an hour, and for large ones - 3 hours.

Change Size (CP3)

The third convergent practice is that the changes submitted for code review are of some definition of small. The two most common^³,¹,² measures are the number of files changed and the number of lines changed. Both can be extracted from the dataset used in this analysis.

NOTE: the research papers deviate from the 50th/80th percentile on this part of the analysis and therefore 10th, 35th, 50th, and 90th percentiles are used for comparison.

At Lyst, about 35% of pull requests modify a single file and about 90% modify fewer than 9 files. At Google², over 35% of the changes under consideration modify only a single file and about 90% modify fewer than 10 files.

At Lyst, about 10% of pull requests modify two lines and the median number of lines modified is 30. This is very close to Google², where 10% of changes modify only a single line of code, and the median number of lines modified is 24.

The median change size in both companies is significantly lower than reported by Rigby and Bird¹ for companies such as AMD (44 lines), Lucent (263 lines), and Bing, Office, and SQL Server at Microsoft (somewhere between those numbers), but in line for change sizes in open source projects.

Number Of Reviewers (CP4)

Rigby and Bird¹ investigated whether the considered projects converged to a similar number of involved reviewers. They found this number to be two. From the same research:

At AMD, the median number of reviewers is 2.
For Google Chrome and Android, there is a median of two reviewers.
At Microsoft, the median number of reviewers invited to each review in Bing, Office, and SQL, respectively are 3, 3, and 4. The median number of people that take part in a review (other than the author) is 2.

Finally, at Google², by contrast, fewer than 25% of changes have more than one reviewer, and over 90% have at most five reviewers with a median reviewer count of 1. Larger changes tend to have more reviewers on average. However, even very large changes on average require fewer than two reviewers.

Lyst also converged to a median number of 1 reviewer, with 80% of pull requests having at most 2 reviewers who participated in a code review.

One of the distinctively more interesting measurements defined by Rigby and Bird¹ was the knowledge spreading effect of peer code review. They measured the number of files a developer has modified (submitted changes to), the number of files a developer has reviewed, and the total number of files they know about (submitted V reviewed). The number of distinct files edited and reviewed by engineers at Google², and the union of those two sets, increase with seniority and the total number of files seen is larger than the number of files edited. The finding is that developers know about more files through the code review process.

Bucketing active developers can derive a similar metric by how long they have been at Lyst (in 3-month increments) and then computing the number of files they have edited, reviewed, or both.

At Lyst, this graph is distinctly different from Google’s in a couple of ways. First, the median number of files is significantly lower. Second, the number of files plateaus for a year after the initial six months. Third, reviewed files never cross the edited files curve. It would be an interesting subject for further analysis to try and understand these differences.

Another aspect of the code review at Google is educational, and the case study² tries to quantify this aspect by measuring the number of comments on their code changes. The study found that as developers build experience working at Google, the average number of comments on their changes decreases. They postulate that this decrease in commenting results from reviewers needing to ask fewer questions as they build familiarity with the codebase and corroborates the hypothesis that the educational aspect of code review may pay off over time.

A similar metric can be derived for Lyst by considering only users who started after 2015.

The resulting graph is similarly declining over time. It could be similarly postulated that as engineers at Lyst build familiarity with the codebase they need less guidance. There is one notable difference between the two, early on engineers at Google are receiving at least 2 more comments more on average.

Interesting Questions

This was probably the most interesting/exciting section to work on by far - to try to ask and answer a couple of questions specifically about Lyst. At the same time it probably is easiest to misinterpret, so a mandatory disclaimer:

The goal here is not necessarily to follow the statistical method in answering the question but instead to show some data. Therefore, it can be challenged that the answers suffer from a confirmation bias.

How long does it take for a new engineer to open their first pull request?

This is an interesting metric to look at from a developer onboarding perspective - is it too long? Lyst focuses the first days on setting up the accounts, meets and greets with the team, and giving new-joiners context before easing them into the first task.

The above graph shows the time to first pull request from the time the Github user was created. One thing to note is that this account is usually created 3-5 days before the actual start date. So it can confidently be said that half of the engineers submit their first PR within the first week, and the majority (80%) do that within two weeks.

What is the median number of files in the first pull request?

From the comparative analysis section, we know that the median number of files changed is two. We could see whether there is something worth discussing about that first change. For example, if the first change most often is bigger than the median - maybe the entry-level tasks are big, maybe the new joiner needs some additional guidance and advice, maybe the projects could be improved, or - something else completely.

The median first pull request changes 2 files, just as the overall median.

What is the median number of lines of the first pull request?

From the comparative analysis section, we know that the median number of lines changed for pull requests at Lyst is 30.

While trying to answer this question it turned out the median number of lines in the first pull request is 51, which is higher by almost 2/3rds than the median for tenured engineers. So we split the data into small changes (less than 51 lines) and large changes (more than 51 lines). As expected, small changes are small. However, the average number of lines in the PRs classified as large was 858 lines.

We did not dig deeper, but it may be interesting to further investigate if this is a mistake in analysis or why about half of the newly joined engineers’ first changes are so big.

How many weeks does it take for an engineer to reach median and peak performance?

From the comparative analysis section, we know that the median number of PRs per week is 3, and the 80th percentile is 5 PRs. Let’s approximate the median performance as the number of weeks until a new engineer reaches 3 PRs per week. Similarly, peak performance, as the number of weeks before an engineer reaches 5 PRs per week.

This is a weak definition because it does not take into account continuity, e.g. someone making 3+ PRs just due to circumstance and then falling below that number. The answer should not be mixed up to productivity, as it also does not take many other aspects of productivity[5] into account.

Nonetheless, the answer could be indicative for planning purposes - how long until a newly joined engineer starts contributing as much as half or as most of the rest of the team.

These graphs do not consider the fact that Github accounts are created about a week in advance. With that in mind, it takes between 2-4 weeks for the majority of engineers to reach median performance and between 3-6 weeks - to reach peak performance.

Which day of the week is the most active?

The days on which we are opening the most pull requests are roughly what could be expected - all days are equally productive, with slight undulations on Monday and Friday.

Similarly, we can look at a heatmap of time of day, per day of the week.

Again, nothing unexpected. Tuesday afternoon seems to have the most PRs opened. It’s worth noting not to make any massive conclusions from this heat map because it does not consider how long it took to make a pull request, just when it was opened. It also does not exclude any pull requests made by automated systems.

What impact to the number of pull requests opened did the first lockdown have?

This is another interesting metric to consider through the lens of the number of pull requests being opened 6 months before and 6 months after March 23rd, 2020.

Drawing conclusions just from this graph is impossible - the contributing factors would require further analysis, maybe by looking at the data by isolating only the users active on March 23rd, to account for the team growth. Regardless of the reason, the difference is certain - ~33% higher rate of new pull requests post-lockdown.

Conclusions

It was an interesting project to compare Lyst code review metrics against available industry research and case studies. Unsurprisingly, code review practices at Lyst are on par with what other, large and prominent, technology companies practices converged to. The results also showed areas for further investigation. One thing is certain - this is just the tip of the iceberg. Analysis of data related to engineering has a lot of potential.

If you are data-driven too and interested in working with a great group of people on an amazing product, check out our careers page!

References

2013, Peter C. Rigby, Christian Bird Convergent Contemporary Software Peer Review Practices; Link ↩︎ ↩︎² ↩︎³ ↩︎⁴ ↩︎⁵ ↩︎⁶
2018, Caitlin Sadowski, Emma Soderberg, Luke Church, Michal Sipko, Alberto Bacchelli, Modern Code Review: A Case Study at Google; Link ↩︎ ↩︎² ↩︎³ ↩︎⁴ ↩︎⁵ ↩︎⁶ ↩︎⁷ ↩︎⁸ ↩︎⁹ ↩︎¹⁰
2015, Christian Bird, Trevor Carnahan, Michaela Greiler, Lessons Learned from Building and Deploying a Code Review Analytics Platform; Link ↩︎ ↩︎²

On The Soft Aspects Of Technical Interviews

2021-07-02T00:00:00+00:00

There are many tools and content available to help prepare and solve the problems used in technical interviews. However, I am yet to stumble upon a good summary of the non-technical aspects of those interviews. I call them “soft” because they relate more to behaviours, including communication, listening, and less on hard technical knowledge. I see candidates often missing the opportunity to showcase them, reducing their chances of progressing to the next stage. I hope this post can help those candidates understand how those mistakes look and how to avoid them.

Before we jump into it, let me say: interviewing is hard for both parties. There is always pressure for the candidate to perform, and sometimes a minor error can throw them off balance and ruin an interview. I have failed to solve problems, and I have had brain freeze, where I could not answer simple questions or even write code(!). It happened not early in my career but just last year, when I was interviewing. I have worked as a professional software engineer for sixteen years and have been programming for 22. If you are starting your career, I hope this paragraph can encourage you to persevere.

The interviewer has their own set of pressures too. They have about an hour to simultaneously lead the interview and evaluate the candidate. The time is short, and a decision has to be made. This requires preparation, experience, and skill, and mistakes still happen.

However, pair programming and system design are entrenched parts of the technical interview process. So it is worthwhile familiarizing yourself with the non-technical aspects too. Here are four common soft mistakes candidates make, including how they look, why they impact the outcome, and how to avoid them.

1. No Questions

The candidate jumps straight into code or diagram drawing almost immediately after hearing the problem statement, without stopping to plan, consider or ask further questions.

The problem statement is usually optimised to give the minimal required information to get started, but not necessarily everything. Part of the exercise is to find out what other important details were not presented, e.g. edge cases. System design interviews are all about scoping functional and non-functional requirements. Also, just like in real software development, the cost of a question not-asked increases throughout the interview. At some point it becomes too late to correct, the candidate ends up solving a different problem, and fails the interview.

There should always be at least a couple of questions, e.g., about the input size, throughput, constraints, expectations, or further requirements. Of course, those questions will be evaluated too, but the mere fact that a candidate stops to consider something beyond the basic problem description is a positive sign for the interviewer.

2. Answering the Wrong Question

This mistake manifests itself when the candidate’s answer unintentionally misses the point of the interviewer’s question. It requires very sharp listening skills on both sides of the table. I find this mistake to be one of the harder ones to keep track of. When a candidate is very eloquent (a good skill otherwise!), it is possible to get lulled by a vaguely relevant narrative.

A typical example, using the probe-dig-differentiate framework, the candidate would be probed for more context about a recent project. They would be asked about aspects such as the team size, their role in the team, who made the decisions, any alternative solutions, etc. This probing stage enables further “digging” into how the candidate thinks about trade-offs, how they work in a team, etc. It adds more signal to help differentiate the best candidates.

But some candidates answer the wrong question. Instead, they retell the project requirements and the solution, emphasizing the technologies in use or the complexity of the problem. An interviewer may stop the candidate and rephrase the question. If this happens repeatedly an interviewer could start doubting whether the candidate has good enough communication skills. Reading, listening, comprehension, and the ability to answer the right questions are crucial day-to-day competencies.

I think I know why this happens at least some of the time. Silence can be intimidating, but it should not be. It should be leveraged to your advantage, by saying “let me think” and giving yourself 5-15 seconds to internalize the question. The last time I was going through the interview processes, I scribbled down the questions while the interviewer spoke and used those “let me think” seconds to re-read the question. This pause shows consideration, helps prepare a more coherent answer, and most importantly, boosts confidence in presenting that answer. Short and concise answers score more points than long rambles.

3. Failure to Admit a Lack of Knowledge

Candidates will deliberately try to slide through an answer, either by shifting the subject, answering indirectly, glossing over essential parts, or outright talking nonsense with confidence. Instead of acknowledging - “I don’t know.”

There are some questions where an improvised answer is good, but if the candidate is given a direct question, and proceeds to evade it - it is obvious to an interviewer. Most interview questions are deliberate, and the interviewer knows how good answers (plural) look. By trying to swindle it like a test in the school, a candidate loses a chance to show humility, self-awareness, and honesty. For an interviewer, it raises a question of trust. If this person is hired and given a task, can they be trusted to know when to seek help?

I think candidates succumb to the pressure of having to present themselves in the best possible light. However, there is a massive upside to being open about knowledge gaps. “I don’t know how to do this, but I think I know how to find all the required information in the documentation” is humble and disarming. The interviewer will provide a hint, and the candidate can proceed to the next step, and then focus on what they know instead of clumsily weaselling out of what they don’t.

If the interview is for a senior role and a candidate requires many hints to get through, maybe they are not ready for that role yet. Ultimately, at work, we never know everything, sometimes not even most of it. When a candidate shows that they can ask for help it is a positive signal. Dishonesty, on the other hand, disqualifies a candidate rapidly.

4. Inability to Accept Help

Sometimes candidates get stuck. It is pretty regular, helps evaluate potential knowledge gaps, and is not a negative sign. What matters is how they deal with getting stuck, because the behavior translates to many daily work situations.

A specific example: a candidate wasn’t sure what data structure to use for storing some values and got stuck and/or missed a subtle hint. When a list was suggested as a valid option to move the process forward, the candidate objected. In their view, it was not an efficient use of a list. Fair, but they also could not unblock themselves using any other data structure.

An outright objection to a hint while stuck is a gambit that can backfire. It is standard for a candidate to propose something better or point out mistakes, thus showcasing their specialized knowledge. However, when done while stuck and without a good alternative, it will raise a red flag.

Such behavior raises a question whether a person, if hired, could gracefully handle suggestions for improvements to their work. Or if every piece of feedback given would turn into bickering about minutiae.

Conclusion

Both interviewer and candidate come to the interview hoping for success. Think of the interview as a rollercoaster of small wins (positive emotions) and losses (negative emotions). Not the same negative emotions one may get from an outrageous tweet, but it is there. We are human, we want to succeed, and we want others to be successful too. A failed interview can feel sad even for interviewer. Now think of this quote by Maya Angelou:

“I’ve learned that people will forget what you said, people will forget what you did, but people will never forget how you made them feel.”

I can bet that no interviewer remembers absolutely everything the candidate said or did, save for a few notes. However, they will remember those small wins (positives) and failures (negatives). Therefore I propose my own quote:

If you can maximize the positives and minimize the negative moments that the interviewer percieves - your chances of success will significantly increase.

How do you do that? You still have to be knowledgeable in the technical aspects, but thoughtful questions, active listening, honesty, and humility will get you past the negative dips so smoothly that the interviewer might not notice them.

Good luck with your interviews!

A special thank you to Adomas for substantial feedback on the original draft, and Helen for proofreading.

On Fundamentals

2021-06-15T00:00:00+00:00

Ultimately, when using high-level tools to make an impact, we are limited not by the breadth (how many tools & languages we know) but by the depth of our knowledge (how well we know them). I briefly touched on this in my essay for aspiring engineers when I wrote it seven years ago. Back then, it felt more like intuition. Now - I am convinced it is valid and valuable.

It is the primary reason why someone, who already made their first steps in programming, should look at computer science fundamentals to increase their depth of knowledge. It is primarily driven by personal experience and observation. For example, I strive to deepen or refresh my knowledge of computer science fundamentals regularly. I do this by reading books or solving algorithmic exercises. Of course, I also get bored and move on to another subject, continuously wandering, and that is okay.

It is okay to look around at the tools you are using today and increase that depth of understanding by going one level deeper, metaphorically peeling an onion. However, at some point, it would be good to touch the fundamentals.

Others disagree. The primary objection is focused on the obsoleteness of such knowledge. “Where am I going to implement virtual memory?” they ask, and add “I’m not writing an operating system.” I think that is not a great question. The more important question is “where is virtual memory used?” (Replace “virtual memory” with any algorithm or data structure). Not understanding where these fundamental pieces of knowledge are used numbs us into thinking they are obsolete and irrelevant. They are anything but that.

It is easy to say “the right tool for the job,” and go to the next popular framework, library, or service to avoid technological limitations. But think about it - if all we know is just how to use some tool, not how it works, we are essentially limited by what is publicly documented, tried, or generally accepted. We might just as well use any tool in that case. At the very least, an opportunity for learning a new way of solving a problem is lost. At worst - a creative or low effort and high-impact problem-solving possibility is dismissed as irrelevant.

My favourite high-impact example of the above point that I have seen in practice is this.

Fifteen years ago, the primary caching proxy for websites was a project called Squid. Then, about eleven years ago, a project by the name Varnish came out. Its author Poul-Henning Kamp wrote about its impact (I recommend reading the full article):

“The first user of Varnish, the large Norwegian newspaper VG, replaced 12 machines running Squid with three machines running Varnish. The Squid machines were flat-out 100 percent busy, while the Varnish machines had 90 percent of their CPU available for twiddling their digital thumbs.”

I used Varnish on multiple projects back in the day, and servers twiddling their digital thumbs summarizes the experience and the impact well. We, too, replaced overutilized servers with just a few that barely did any work in those projects. So how did Poul achieve such an impact?

<…> The really short version of the story is that Varnish knows it is not running on the bare metal but under an operating system that provides a virtual-memory-based abstract machine. For example, Varnish does not ignore the fact that memory is virtual; it actively exploits it.”

Poul had a deep understanding of how memory management in an operating system works. As a result, instead of re-inventing it in the application, he made Varnish utilize the memory management provided by the operating system.

As I was working on the draft of this post, on June 8th, several top-rated websites went down and showed error screens for a few hours due to Fastly, a content delivery service, outage. What is relevant here is that the error messages also exposed the software used to deliver the content (not that it was a massive secret). What started with 12 servers in Norway is now running a significant portion of the internet. It was Varnish.

The critical element here is not that Poul knew how OS managed memory but that he knew the stack deeper than anyone else who worked on the caching problem and had an insight. By my recollection, before Varnish, caching was a solved problem, and if you had to cache something - Squid was a de facto choice for that. So no one was looking for insight.

It is worth saying that knowing stack deeper does not imply building more lower-level things from scratch. I have not made Varnish level of impact, but I have found simple solutions to complex problems on multiple occasions because I understood the stack better. Not buy building new things, but by having a creative insight about existing tools.

There is nothing special about this, and anyone can do it. What is needed is some curiosity and time. If all this sounds reasonable, Julia Evans has an excellent blog post on learning how things work, which covers the practical aspects of doing it continuously.

Requiem for Phabricator

2021-06-08T00:00:00+00:00

Last week Phacility, maintainers of Phabricator, announced that starting June 1st, Phabricator will not be maintained anymore.

Phabricator has been a tool I relied on on multiple occasions, made a couple of small contributions to and occasionally did small extensions in a private capacity when needed. It has genuinely been one of my favourite engineering tool suites.

Strange to say it, but Phabricator and Evan’s writing contributed to my growth as an engineer. I’ve never met Evan or the rest of the contributors, but I am immensely thankful to them for Phabricator.

This post is just a shortlist of things that I learned through using it, reading code, or blog posts.

All You Need Is Trunk

Back in 2012, a few months into the YPlan journey, I got my eyes on Chuck Rossi’s tech talk called “The Push.” One of the central ideas in the talk was this: to have a continuous flow of changes to production, engineers should not care about branches. They should be able to push to trunk/master/main branch. It is a less wild idea now, but back then, the mainstream was convoluted git-flow with feature, development and release branches. Chuck Rossi’s presentation turned the whole thing on the head. I was in awe, and we tried something like that at YPlan with GitHub. It was debilitating to do a sequence of small iterative changes using the pull request model. I almost thought there were no convenient tools to do that in the public domain or as a service. Except, there were a few moments in the talk where I thought I saw a familiar interface.

Cue Phabricator. Open-sourced a few years earlier. I talked about it with the team, and we decided to give it a go. It was effortless to set up an instance. We still used GitHub as the repository hosting but switched to Differential Revisions for code review. Worst comes to worst, we thought, we will go back to pull requests. We never looked back.

For a new engineer in the team, it took about a day to get used to the Arcanist tool. It took about a week to understand how powerful the setup was. Almost all engineers I worked with, and who worked with Phabricator, missed it after switching back to GitHub and the Pull Request based model. Those who never used it continuously - seldom get what’s so good about it.

After working in companies that used Phabricator and GitHub, my observation was that engineering teams using Phabricator were at least an order of magnitude more efficient at pushing material changes to production than with GitHub. Someone could effortlessly churn up to 10 changes a day to production.

Of course, the tool is just one aspect of the story, but there is something powerful in the ability to slice and dice and stack or drop your changes as needed that contributes to that story nonetheless.

Write, Review, Merge, Publish

Ever since I read “The Psychology of Computer Programming”, I was immensely fascinated by how seemingly inconsequential details can have a massive impact on engineers work. My favourite story is how a well-performing company’s HR department moved a watercooler away from a secretary’s desk. The secretary was annoyed by people chatting there all the time. In the aftermath, the engineering teams started missing deadlines and projects ground to a halt. Noone asked what those engineers were talking about and assumed they were wasting time drinking morning coffee. That watercooler was the informal standup for unblocking one another.

Similarly, I think there are assumptions and decisions encoded in the Pull Request based development model. What would happen if they got adjusted? These changes could be almost philosophical or cultural and yet profoundly affect work output. I think Evan Priestley’s article “Write, Review, Merge, Publish” is an article that had such an effect on me. It explains the differences between Phabricator and the Pull Request based - Write, Publish, Review, Merge model. I won’t even try to summarize it here. The difference is so subtle. Yet, it has a definitive psychological effect on how people perceive their changes, for example - “as finished.”

The Serendipity Of Relationships

One of the main things that I relied on Phabricator throughout the projects that used it - is the seamless linking of objects. If a task T123 is relevant to a revision D456 under code-review - you mention it in the code review description, and it is immediately a two-sided link. I can go to T123 and see D456, and all other revisions, blog posts, wiki pages, and so on that mentioned it. What is the point? It gives a fantastic opportunity to find low-effort and high-value improvements. Write down ideas into tasks, add some context in the description, mention the task id anywhere relevant. Later, come back, and you can see all the instances of relationships - tasks, code changes, blog posts, wiki pages - in one place.

It leads to a fantastic result that I called serendipity driven development. It works like this: if there is some bug or a problem that is not prioritized but annoying or impact is not clear, and fixing it would take effort. Leave it to brew, mention it where relevant, do work, read Hacker News. At some point, an idea will pop up, and a way to solve that problem in minimal time may come up. I’ve seen this in practice, and it feels fantastic doing it as well as watching others do it.

The Power Of Integrated Application Suite

Another thing that I loved about Phabricator is that it came as a suite of smaller applications. Task management, workboards, code review, code ownership, wiki, paste, internal blogging platform, if-this-then-that business rules, countdown timers, polling, build orchestrator, API.

All for free. All tightly integrated, allowing those relationships and more. Sure, there are plugins for GitHub, but many times if A and B work with GitHub, they don’t integrate well with one another or charge a fortune for that upper tier of service. If I ever get lucky to see a suite of apps that works as well with GitHub and among themselves, as Phabricator suite did, and do not cost a fortune - I might reconsider. But until then - I genuinely miss the zero effort things like that.

Multi-Repo What?

There was a time when multi-repo became this crazy hype and turned into a mono-vs-multi repository war. I read a lot on both, worked with both, and got to reflect on both, and I conjecture this:

The multi-repo appeal is in some cases driven by the popular continuous integration SaaS-es only allowing a single project configuration file per repository, or otherwise making the process of having multiple projects in one repo a significant nuisance to manage.

Some would say, “that’s the whole point.” Well, cue Phabricator and especially its code ownership tool with Harbormaster build orchestrator. It enabled the setup of multiple build triggers and build jobs, depending on which files were changed. Therefore, I had a choice. I could have had as many repos as I wanted, or one, with virtually no difference in the amount of effort in managing it. Build System at Genus AI was the prime example of that.

Scratching Below The Surface

When I started my professional career, I worked with PHP and 10-16 years ago, PHP was mocked for poor quality code of projects it was used in. It was never constructive criticism, but there was a point: PHP was easy to get started with but hard to master, and there was very little high-quality code to learn from.

Again, cue Phabricator. I remember spending afternoons and evenings just browsing and reading that code. There is a lot that can be learned from the way Phabricator code is structured. I wrote a few things at YPlan to link Jenkins with Phabricator, and it was a pleasure to work with. I can barely remember any other project that would be this amenable without presenting a massive amount of technical documentation first. Phabricator had little explanation in terms of its internal technical workings, but it was readable.

Database Design

I am not in the know, but I want to conjecture: the fact that it came out of Facebook, and the size of the Facebook organization and its projects, there must be many lessons embedded in Phabricator of how to scale databases.

Phabricator has quite an intriguing database(s) layout and relationship tracking model (edges). While at YPlan, I tried to replicate the edges model to drive the users’ timelines. I was the only enthusiast of that model, but it worked pretty well.

At YPlan, we also got rid of the database migration graph and relied on alphabetically sorting migrations. I got the idea after reading the Phabricator code, and it worked wonderfully. Up to this day, my conviction is that DAGs for database migrations in small and medium projects operated by small to medium teams causes more waste of time and annoyance than the value they provide.

A Sad Loss

I find it slightly ironic that there is a lot of talking about being cloud or platform-agnostic only to go back to their corporate GitHub accounts and sprint boards on Jira. The de facto monopoly of how software is made.

GitHub has done all the good things to make source control and open-source available to everyone, but it also made the open-source development model the new normal for private companies of 2, 10, 50, 100, & more engineers. There is essentially only one way to build software right now - the GitHub Pull Requests + Jira. Whether it’s GitLab Merge Requests or Monday/Asana is irrelevant - they are all essentially equivalent. I think this unseen monopoly is unfair.

Phabricator was the beacon of something radically different that feels uncomfortable at first, is rough around the edges, but might as well work better than what we have now. It feels sad and looks like an end of an era, but while the announcement said the official maintainers stepped down - I am keeping fingers crossed that there is hope in the community.

I think there always should be an alternative.

Merits of Mentoring

2021-03-22T00:00:00+00:00

This is an abridged version of an internal presentation I have given at work. In the slides, I used frames and art from the movie Back to the Future, and this is where the name Marty comes from.

Let me start with a story. Once upon a time, I was the first engineer at a small tech startup. Just five people in a six desk room. As you can imagine, I had all the freedom of drawing the line between being pragmatic and trying cool new things.

One day I am told that we’re going to have a summer intern to work with me, let’s call him Marty. I was excited, but I was also flustered a bit - I knew nothing of him, except that he recently got a BA in Political Science. That’s how startups work sometimes.

The next day Marty showed up and proved he couldn’t meet my expectations for an entry-level position. At the same time, he’s very humble, kind, and tried to compensate for his lack of experience by working long and hard.

Over the next few months, Marty and I would come in 1 hour earlier to the office. We would solve programming exercises, or draw diagrams of computer science concepts on the whiteboard.

In the beginning, Marty could not read an error traceback, and would repeatedly adjust his code until something changed for the better. After a 30min overview of what a stack is and how the CPU runs a program something clicked.

A puzzle of these disconnected pieces assembled into a coherent picture for Marty. It became clear what a traceback is, how to read it, and what exactly needs to be changed in the code. At the same time, I was reminded to not underestimate the importance of the fundamentals of the craft.

His first change to the codebase took about a month, the second one - two weeks. By the end of the summer, Marty was making about 1-2 changes per week, automating his work, with full test coverage, and just a little guidance. Marty was hired full-time.

I also had a small epiphany. I cannot always choose who I will have to work with, but I always have a choice to help people around me to be better. This opened me up to an option of volunteering that I had not seen a lot of purpose in before.

First Steps In Volunteering

I work at a different company than Marty now. In January this year, a colleague posted a message in diversity channel, with a link to Codebar and I registered as a coach.

But it also connected my previous experience with the realization that being at a bigger company comes with a privilege. I am working among people who went through a rigorous hiring process. The thought applies to every tech company out there.

It means that roles for people like Marty on his first day open up rarely. There are many more Marty’s outside the industry who are as hardworking, determined, kind, and humble. Yet, they are not getting a chance to pass that entry bar.

These “other Marty’s” are minority group members. They are looking for a safe and collaborative environment to develop their skills and expand career opportunities. That’s how Codebar, the charity I registered with, defines it. It is largely true, but this development of skills comes in a variety of ways and forms.

There are people learning to program. Some are completely new. Others know a little already and are struggling with a concept and need help finding the answer.

Some have a personal project and need expert advice or reassurance about what is good and what is not, or what to do next, what library to use, etc.

Others are trying to prepare themselves for that first technical interview that may open them to an exciting career switch.

Some students have limited technical jargon, have strong accents (mentors, like me, too), or are genuinely less chatty than others. You must be prepared to take whatever skillset or whoever it is that comes your way. Accept it in a cheerful stride and be focused only on helping the student move forward.

So Why Do It?

Coming back to the merits of mentoring. By doing it you help someone else to get better. If nothing else changes, as hard as your day was, after helping someone else you will feel a positivity boost.

Just as an example, I don’t work with Marty anymore, but we are in touch. He has been promoted and is now managing someone, which is an exciting new experience for him. He is also very appreciative to no end for having had a chance to learn from me.

It may sound cheesy - I cherish the thought that I helped him early in the process, but he can only blame himself for that promotion. :) Nonetheless, this is a genuine reaction and gives a positive boost for me too. But that is not all.

This experience of mentoring Marty, other colleagues, as well as volunteering at Codebar has taught me. It taught me to be more patient when someone takes time to grasp new ideas. It taught me to be humble and inquisitive when I cannot understand or explain something myself. It taught me to break down and communicate hard concepts in a way someone can gradually build an understanding. The sole reason I came back to writing to this blog is because of this experience.

There are many ways to start mentoring - formally and informally with a person, or even alone, writing a blog. I want to leave you with one thought that hopefully inspires you.

Regardless of where you are at in your career, remember that there is always someone trying to walk the same path that you have walked. When you listen to someone who is ahead of you and who you aspire to be, remember to reflect. Anything you know now is equally aspirational for someone else, like other Marty’s who want to make that step too.

You could be the hero who helped them grow and be better.

If you have questions, or if you’d like to work alongside me, in a company that cares about and actively works on this - reach out to me on LinkedIn and mention this blog post.

Book Review Of 2020

2020-12-16T00:00:00+00:00

As is fitting for the end of year blog posts, this one will cover some notable books I have read in 2020. This year I barely read fifteen books, compared to 23 in 2019. The primary reason for this number being low is the COVID confinement. To make it more enjoyable, than just a list of book names, I’ll chop the list into three groups: books I learned the most from this year, books I most enjoyed this year, and whatever is left - will get an honorable mention. Most of the book titles have affiliate links to bookshop.org, in case you would want to support independent booksellers. For convenience you can open the list of all these books in one page.

Books I Learnt The Most From

Atomic Habits - Tiny Changes, Remarkable Results by James Clear.

It is one of those books with a simple premise, but profound impact. The premise and arguments are very enticing. I have made a lot of notes but must admit - have not built any new habits myself. I was fascinated by two things. First, how it can be applied at a group level - which I did at least on one occasion. Second, how many parallels I found with the way I build software. My posts about Bottom-Up Problem Solving Part I and Part II share the same spirit used for habit building.

Software Engineering at Google: Lessons Learned from Programming Over Time by Titum Winter, Hyrum Wright, and Tom Manshrek.

This is by far my favourite Software Engineering book of 2020. At first, I was very reluctant to read it - Google has virtually unlimited resources to do, or at least try, optimal development practices. And there was the mandatory “but it is for Google (scale)” thought. However, at least some parts are undeniably universal for an engineering organization of any size. I introduced aspects of knowledge sharing, culture, and leadership chapters in a startup team. The book also demonstrated to me why Kubernetes is a good thing - at Google scale, but (trigger warning) I remain unconvinced that it’s anything beyond a hype train for a team of 5 engineers in a room building a product that has 5 clients. But that’s a story for another blog post.

Speed Up Your Django Tests by Adam Johnson

This is my favourite book on a specific technical topic, and so far - the only one I reviewed on this blog. I won’t repeat the review, but the book has a diverse set of practicable improvements someone can do not only to make the tests run faster but improve the project health generally - from database configuration to CI/CD setup. It’s well worth the price. I recommend following Adam on Twitter as he does sizeable discounts.

The Great Influenza by John M. Barry

I saw this book recommendation back in March, but avoided reading it, lest it will cause me to worry about how a real pandemic may look like. Having survived so far, grateful that I did, and with the courage to understand the current situation, I opened this book in early December. Highly. Recommend it. I learned two important things. First, 1918 flu must have been orders of magnitude more harrowing to live through. Second, it did not start as a killing machine and has been largely ignored as mild influenza in early 1918. It evolved to become an efficient killer by going through the population. The more people the virus infects - the more chances it has to mutate into a more efficient killer. Reading the very well narrated story and documented experiences of people from 1918, and putting it into the present context, causes chills to go through the back.

Books I Most Enjoyed

Velocity Weapon and Chaos Vector by Megan E. O’Keefe

It has been a while since I read and enjoyed a sci-fi story. I don’t know how I found the first book - Velocity Weapon - in the Protectorate trilogy, but I am happy that I did because I was hooked. The second book was published this July and it ended with a massive cliff-hanger. The third comes out next year. I am giddy and can’t wait to get my hands on it. I hope someone makes it into a graphic novel - it would augment the already brilliant story with an excellent cyberpunk/sci-fi noir visual feast. I also had my first fanboy moment tweeting the author to complain about how good the cliff-hanger was.

The Boy, the Mole, the Fox and the Horse by Charlie Mackesy

My review won’t do justice here. You just have to read it, and that’s that. I bought it as a present twice. It is a collection of life lessons everyone should know, told in a very empathetic way, with few words and many unique illustrations. Every page hits with powerful emotion, and I admit - I cried. But I also felt good about it. The experience is comparable to a teary and powerful movie ending, but whereas the movie ends and no one remembers what they cried for, this book is very personal and hits home hard. Just get it, alright!?

Harry Potter and the Cursed Child by Jack Thorne, J. K. Rowling, John Tiffany

I don’t think this needs an introduction. Having read through all of the other HP books in 2019, this script was a wonderful closure to the original story. When West End reopens I sure will try to see the actual play.

Slaughterhouse-Five by Kurt Vonnegut

I had very mixed feelings about this one. Found it difficult to follow the story in the first 50 pages, but afterward, it started to make sense, and I mostly enjoyed the time-traveling aspect of the story. However, considering how high, and for what, this book is regarded - it was an intriguing story, but it did not turn me into a total pacifist - and because of that, I feel like I missed out on something important.

Hit Refresh: The Quest to Rediscover Microsoft’s Soul and Imagine a Better Future for Everyone by Greg Shaw, Jill Tracie Nichols, and Satya Nadella

I think Satya Nadella is the reason Microsoft turned into a cool company after 20-30 years of having a corporate / enterprise vibe. This book describes his humble beginnings and a vision for the future of technology, but for me - the most interesting part was the story of how Satya transformed Microsoft culture. Building a company culture successfully requires a conscious effort. Once established - the culture is virtually impossible to change, without damage. Therefore it is an extraordinary feat to redefine the culture of a company that has 150K employees around the world.

Grow Your Own Vegetables in Pots and Containers by Paul Peacock

Something non-technical, for a change. When work from home began in March I craved to do something different, as a hobby. This year I lived in a top floor apartment with a spacious balcony, and so I used this as an opportunity to grow some vegetables. This book was an excellent guide throughout the process. It is priced very generously, and the writing style is sometimes really amusing - so it is not a boring read. I started with cherry tomatoes (Tiny Tim variety) but quickly branched out. By the end of the summer, the yield was about 1l of cherry tomatoes per day for about two weeks, 30 carrots grown in ten 4-pint plastic milk containers, and a few radishes. I have already started thinking about next year’s plants.

Honorable Mentions

The Harvard Business Review Manager’s Handbook

A collection of 17 different chapters and 17 different aspects of managerial activities. Quite dense on actionable steps, but readable and easily applicable.

Giving Effective Feedback by Harvard Business Review

A very short walkthrough of when and how to give feedback, what to do in difficult situations, and how to build an environment receptive to feedback.

A Philosophy of Software Design by John Ousterhout

Being a big fan of “The Psychology of Computer Programming” book, I thought this might be a philosophical take at building software systems. However, what I found was a list of good software engineering practices and a very dry read. The content itself is not wrong in any way, it is I just found it difficult to continue reading past 1/3rd of the book and skimmed through the rest.

Machine Learning Engineering by Andriy Burkov

This is an excellent addition to the “The Hundred-Page Machine Learning Book” by the same author. I use both as quick reference books to remind good practices, how various ML algorithms work, common problems, and ways to approach them.

The Art of Statistics by David Spiegelhalter

The book starts with a murder solved using statistical methods. And most of it is written in a way any person should understand with only basic math knowledge. There are no formulas. It explains how statistics can be manipulated and why we should all be wary of anyone who claims to be making decisions “following the data.” Data does not make decisions - decisions are made by the interpreters. Humans.

Seven Brief Lessons on Physics by Carlo Rovelli

A booklet explaining seven fascinating aspects of modern physics. Honestly, do not recall much about what they are.

No Man’s Land by David Baldacci

This is the last book in the John Puller series. John Puller is a military investigator, and in the good old Angela Lansbury style, he runs into various murders or mysteries to solve. The book (and all of them in the series) has a good hook, starts slow, but keeps increasing the pace as you go with the story. Eventually, it causes the curiosity to peak so much, that it becomes very hard to put the book down. Recommend it.

The Unicorn Project by Gene Kim

This is a follow-up book to “The Phoenix Project.” The Phoenix project felt like a fantastic representation of a company’s internal struggles with effective IT operations and was very educational. The Unicorn project was just a complete opposite. A miserable read, full of deus-ex machina that caused the story to drag on and on, just to prove a point. Even though it was acknowledged in the book, but it was abused too much. Whatever educational points this book was making got lost, between “oh… get over it already!” and “please, just end this misery.”

The Five People You Meet In Heaven by Mitch Albom

My first introduction to Mitch Albom was in “Tuesdays with Morrie,” which I read in my twenties and it brought me to The Last Lecture by Randy Pausch. Both had a big influence on me. However, I cannot say, that “The Five People You Meet In Heaven” made a big influence now. Maybe it is because I am now older and more cynical or I read the other ones first, that this one did not influence me. However, it was a nice and relaxing read.

Peter’s Empress by Kristina Sabaliauskaite

Written by a Lithuanian art historian and novelist, Kristina Sabaliauskaite, who is regarded well around the Baltics for her previous historical fiction. This book follows a woman called Marta Skawronska, an orphan, who became Empress Catherine I of Russia. It starts with an objectionable gimmick, but I got over it and found the book pleasantly entertaining. Sabaliauskaite is notorious (at least in Lithuania) for recreating very realistic 16-18th century environments and characters in her writing. I read it in Lithuanian, but I see there is an English available in the Kindle store. Assuming the translation is good, it should be a pleasant read for a historical fiction enthusiast.

On Continuous Automation

2020-12-08T00:00:00+00:00

In this post, I will summarize my approach to CI/CD automation. I believe that it is pragmatic enough, and blends in well with the primary focus on product, not tooling, and gradually sets-up small to medium teams for success with quality engineering operations. To learn about concrete instance of one such automation evolution - you can read Build System At Genus AI, where it sustained automation for 3 years and more.

Commands

The natural first step, before any automation, is when development tasks, like running tests, building artifacts, or deploying them, is just a sequence of commands everyone in the team knows and runs. It is simple, straightforward, and pragmatic. However, over time it can lead to problems (more on those - later).

During this phase, my focus is usually to work out the sequence of commands for each use-case - testing, building, deployment. Also, understand the differences, if any, between executing something locally vs remote. Everything should be documented in some structured format - markdown, restructured text, wiki, or notion.

Local Automation

At some point, it should feel a bit repetitive having to execute the same sequence of multiple commands over and over again. Time to bring all the different commands together as a bash script, a Makefile, an Ansible playbook, or some other multi-purpose and robust tool. This stage takes zero time because the commands are already known, and the tools to glue them together should be simple.

This is the local automation stage. It is about building one script to execute all the commands on a local machine. That is probably an excellent point to be for a small team, but it also sets this team for the next step - to run it on automation service.

Remote Automation

Depending on how the above two steps went, this one is usually the easiest. It requires understanding the differences between running commands locally vs on a CI service and parametrizing scripts. Once the scripts are ready - use the CI services, like GitHub Actions, CircleCI, AWS CodeBuild, Jenkins, or others, and invoke the same local automation commands. I want to emphasize this: the continuous integration service should run only the same script as an engineer would locally.

For a concrete example, when I was deploying my blog using CodeBuild (before the billing incident), I had Ansible playbook blog_deploy.yml that I could use from my machine. However, I only used that early on, and instead set-up AWS CodeBuild service to do it automatically when I push a new article to the repository. My CodeBuild buildspec file looked like this:

version: 0.2
phases:
   install:
      runtime-versions:
         ruby: 2.6
         python: 3.7
      commands:
      - pip3 install -r requirements.txt --no-deps  # install ansible
      - (cd blogs/seporaitis-net; bundle install)   # install jekyll
   build:
      commands:
      - (cd ansible; ./blog_deploy.yml)             # deploy

There are two phases - install dependencies and execute the same command I would use locally. That is all.

Isn’t this obvious?

It turns out - it is not. DevOps Research Report says that although automated builds are popular at 64% of the companies surveyed, this number goes down quickly with each stage of automation closer to production. Down to just 17% of companies having an automated deployment. Implementation details are not clear from the survey results, but undoubtedly some of the automation suffers from problems this article could and can solve.

Personal anecdata - in one medium sized company I worked at, the deployment process was documented as a list of commands to run. Engineers would have to sit and wait 20-30mins to go through the process, looking at the screen just watching output, then invoking the next command.

Think about it, if a team consists of four engineers, deploying one change per day each, that is 10hours of productive time per week lost to mindless staring at the screen. Not including mistakes, regressions, or the number of environments (staging, QA, etc.). Each of these further amplifying this waste.

Doing this for a year is equivalent to losing almost 2 months of productive person-time, if not more.

What you get for free

On the flip side - this approach brings some benefits for free.

This setup is not intrusive to product work and should require very little approval to implement. Commands and local automation should flow naturally out of the daily work. The remote automation with CI can be just a simple YAML file to checkout code and execute the commands (e.g. GitHub Actions, or CodeBuild as shown above).
Deployments become consistent. Based on my experience, once set and settled, this automation reaches a maturity level where material changes happen only when a big upgrade is needed or when the program significantly changes. Meanwhile, it remains untouched, and the chance of human-error running the list of commands is almost zero.
Fallback is local. If CI service is having a bad day and does not work, you can be 100% certain that running that deployment command locally will work, because it is the same command that has been exercised daily. At the same time, if some commands start failing for some reason, it will most often be simpler to debug by running them locally.
More productivity. Time spent waiting for a test-suite to finish could be spent working on some small improvement or a follow-up task. Essentially, swapping time spent watching and waiting, to product building or innovating.
Less coordination. Similar to the above example, having two engineers deploy two separate changes and coordinate the process can be turned into a productive time. Instead of waiting on each other to finish, modern CI services can ensure that all the changes go out one by one. In this case make sure that error alerts can reach the engineers automatically, e.g. by a Slack message or email.
Rollbacks are easy. Triggering deployment for a previous code version can be as trivial as reverting the commit and deploying it.
Diverse workflows. Test, build, and deploy are just the obvious ones. This model allows easily add stuff like side-deploy, canary testing, anything you can think of. As long as it is scriptable - it is automatable. This in turn enables GitOps.
Single source of truth. There is only one way to execute an action - the way documented in a single place in the code.

When does this not work?

The most common scenario when this does not work is when the CI/CD environment and local environment do not match, e.g. use a different interpreter, operating system. Or when the commands executed in CI/CD are different from those that engineers use locally. There is an implicit, and I think natural, assumption that if CI/CD is running a test suite, it is doing the same thing as an engineer would do locally. It leads to “but it works on my machine” frustration and sometimes hours wasted fighting something that is not even broken.

Conclusion

All the arguments boil down to this - CI/CD services should run the same commands an engineer would run locally. I hope this article convinced, or at least planted a seed, that by using one single source of truth for your automation, you can make yourself and your team more productive.

On Continuous Improvement Of Code

2020-11-23T00:00:00+00:00

On my most productive weeks as an individual contributor, I was making between 15 and 30 changes. Most of them were small iterative changes towards whatever the product backlog would define. However, I would always squeeze in 1, 2 or more small improvements that made the project healthier, made my team or myself happier working with it, or solved a problem for someone else. Not long ago, my colleague asked how I manage to find time for that. My reply was that I simply used fifteen minute breaks and looked for something with a low context switching threshold, and usually had some minor ideas on what to fix. But that is not the full story and upon a deeper reflection, I distilled it. This post is the actual framework for continuous improvement of code that I use.

Make A Note

Next to my keyboard I always have a notebook and on the sidelines of the page I continuously make one or two word notes of the minor annoyances I encounter, in very few words. Sometimes as little as one or two, examples that I remember - “dependency management”, “exception formatting”, “excessive alerts”. Minor does not represent the scope of the problem, just means that I encountered it and overcame it at the time. It is something that would fall as an answer under “where do you spend the time that you think you should not be?” or “what things you find hard, that you think should be easy?” during a retro. Listening to the team can also show areas of these minor annoyances.

Key thing is that writing those two words down is never a commitment to fix it. The purpose of making that vague note is twofold:

Remember. Writing the issue down helps me not to proverbially boil my performance, by getting used to something annoying and time wasting.
Prioritize. I can add marks to this written note, if something happened more than once, was mentioned by more than one person, or has a broader company-wide impact.

Since there is no pressure to fix the problem it means that if I write something vague, and cannot remember what it was about - it was not important. At the same time, if the original issue does resurface - some or all of the vagueness will reconnect due to the contextual memory.

Make It Visible

If the problem persists - write it down to the backlog, bring it up during tech debt discussions and retrospectives - make it visible, and hear the feedback. Three things may happen that are quite serendipitous.

Someone will say it’s not an issue and will propose an alternative. Problem solved.
Someone will say it is an issue and give more details. It will reaffirm your prioritization is aligned with the team’s understanding, as well as provide additional information.
Someone will propose an idea on how to solve it. It might be better than whatever you have thought.

On one occasion, a project I was working on relied on AWS Lambdas and occasionally we would get initialization failures in production, while continuos integration tests would pass. The test environment was importing all the dependencies slightly differently than the real AWS Lambda environment. The problem happened not very often, once every few months, but when it would happen it was a full, not partial, outage. I have documented the problem into a tech debt task, but all the solutions seemed disproportionately large in scope and did not justify the time. Until one day someone said, “Hey, have you heard of python-lambda package?” Fifteen minutes later we had a simple test case that covered this regression and it did not repeat.

Start Small

If it is not clear how to fix the problem or it would take a long time to debug it, and it is not a product priority, it may be hard to get an endorsement to work on it. However, it may be better to, instead, ask the question: “what would make this easier to do, with little effort now?” Some examples of simplifications I used:

Additional logging. If it is an evasive bug, instead of spending 5h debugging it, it may be easier to add logging, and work on something else for 5h, then come back and investigate the data.
Additional research. If a problem is more process or third-party service driven, I would spend a 15min break looking for and reading a blog post or documentation about it.
Additional visualization. If the relationships between systems (especially new ones) cause a brain freeze - a good diagram always helped me.

If the solution did not surface, usually the next step was still clearer. In either case, I would also document my findings, increasing the chances of serendipitous insight.

Make It Part Of The Flow

Doing this regularly can help break the monotony of a boring task. Personally, after I submit something for code review, if I want a break - I reward myself with fifteen minutes of looking at the notebook, backlog, or open branches, for small improvements I can make a step towards. Since there is no pressure to solve anything right there and then, even if nothing happens, I switch back to work mode. Integrating these short breaks into the daily workflow accumulates a few work-in-progress improvement branches, eventually leading to one or more small improvements over per week.

Lessons From Two Years Of Depression

2020-11-09T00:00:00+00:00

Mental health has a profound impact on work performance (not to say on life in general), and yet it does not get a lot of attention in software engineering circles. I think that this is partly due to a combination of the anxiety of being misunderstood, shame of even having had a problem, or fear of reprimand at work. I felt like this from 2016 to 2018, when I suffered from depression, and at the time only a handful of people knew. This post is a reflection on how I felt, and with some thoughts about what I learned along the way. As well as this, and probably more importantly, I want to set an example for others that it is okay to talk about mental health, even if, as it is in this case, it happened a while ago. If this post helps even one person to understand that depression is an illness that can be coped with, alongside providing them with some ideas of how to cope, then this post has served its purpose.

The written form only allows one type of structure, so while the sections are separate please think of them as overlapping and in some cases simultaneous.

The Plunge

I cannot identify a specific point in time when the onset happened. It must have been slow and related to constant tiredness from work. However, with hindsight, I can identify what I will define as ‘the plunge’ and what I learned from it. In early 2017 I quit a job at a company that I had worked at for 5 years and had enough savings to do a three month sabbatical. I went into this thinking that: (a) I would get some rest and get ‘better’; (b) my skillset was in demand and I wouldn’t have a problem getting back into the market.

Part (a) came to fruition and I did get some rest, but all the benefits were negated by (b), the subsequent job search. Contrary to my expectations, I could not get interviews or past the first interviews. Very quickly it started to feel like I was some sort of broken good and this was ‘the plunge’. Those were the darkest days. I had to go through each interview process as presentable, smiley, and friendly, but after the call I would put my hoodie on, draw the curtains, turn off the lights, and lie in the darkness with my mind buzzing and echoing all the mistakes I must have made. Until the next interview, it felt like I was lying to the whole world about who I was because I was putting on a face when really I felt terrible.

To increase my chances I started to apply for more junior positions where I would have had to learn a new programming language from scratch. In one instance after submitting my homework, using a new language, I was invited for a follow-up interview with the CTO of a very popular startup in London at the time. My excitement quickly turned to a floundering, as the CTO used half an hour to point out every single problem with the homework solution, of which there were more than enough. My solution worked, but it was not written idiomatically. Needless to say I did not progress to the next stage. I was devastated and in tears.

Although it was the most grueling interview I had experienced it helped me to realize something about myself. A lot of my identity, who I thought I was, had blended with my professional work. I saw myself as a successful, experienced, and self-confident developer who had some funky hobbies on the side. I had been programming since the age of three on my dad’s lap, and followed that passion into adulthood. When I struggled to find a job, rejection after rejection did not just upset me, it pierced the core of my being. I thought I knew who I was, and yet my perception was that the whole world was telling me that I was not.

It sounds trivial in hindsight, but at the time it was one of my biggest steps to healing. In a world where it seemed to me that everyone was told to follow their passion, it was extremely helpful to realize that passion is a powerful emotion, and that it is also often short-lived. Cal Newport does an excellent job explaining how to think about it in the book “So Good They Can’t Ignore You”.

Lesson: You are not your work. Work does not have to define you. There is a lot more to you.

The Drowning

Everyone experiences depression in slightly different ways, in my case it was apathy and a buzz of negative thoughts. It is exceptionally difficult to explain how it feels, but I will try. During every spare moment a wave of thoughts would flood through me, taunting me with things I had done badly (both recently and in the distant past). Pointing out mistakes, turning all the ambiguous statements that someone said into an insult, reminding me of all my personal and professional failures.

As the onset is gradual, at first I just brushed these thoughts away, but over time there was more and more. Eventually, unless I was focused intently on something, I would hear these constant echoes. Keeping focus takes effort and is exhausting. At times, even looking sociable and friendly requires focus and insurmountable effort. Putting on a face is like gasping for air, afterwards the mind submerges itself back into the monologue of self-torture.

This cycle became the new normal. Firstly putting on a face in the fake “at peace” moment, then moving back into the negative thoughts. Everything is okay, as long as I don’t care, don’t make any sudden moves to be happy, I can bear it. This leads to apathy and acceptance of this way of being. But unlike breathing under the water, there are moments where you have to go up and gasp for air. I was not doing this so this is what I will name The Drowning. The Drowning does not lead to life. In a final gasp it leads to one or the other, and, as sad as it is to admit, back in 2017 I reached a stage where I researched both options with indifference about the result. Two key things helped me get out of this situation; an exercise from a book and medication.

The book was called “Brain-Switch: Out Of Depression” by Arline Curtiss. I read a Lithuanian translation, and it may be that that is why I found the book so dry, boring, and exceptionally repetitive that I did not even finish reading it. Not a great review, I admit, but there was one principle repeated over and over again: whenever a fit of negative thoughts comes, drown it out by shouting something else louder inside your thoughts.

The book recommends repeating a phrase in your thoughts (it can be anything, e.g. “green turtle, green turtle, green turtle, …. and on and on.” or “nice sun, nice sun, nice sun, ….”). Admittedly, this sounds silly, but trust me, it works. I do not understand the neurological or psychological principle at hand, but I guess it works in a similar way to calming breathing. Mindfulness is an important tool on your belt that can help with this too, although I have less experience with this.

Lesson: Be aware of your thoughts, and don’t be afraid to consciously think of something, even if it sounds silly.

The Pain

There were moments during The Drowning when I was seeking powerful emotions to get through the apathy, to feel like a normal human being for a moment. I therefore decided to walk the Great Glen Way - a well-known 125km multi-day hike across Scotland. As it turned out, my hard walking shoes were too small, and lead to blistered and bloody feet from day two. I had a choice to go back, or to try and finish it. The walk was slow, the pain was excruciating and on multiple occasions passers-by stopped to ask if I needed help. Yet in some strange way I felt alive as I had not felt in a long time. Reaching the finish line was anticlimactic with no sudden uplift or celebration, I just lay down on the grass until I got cold and then limped away looking for my hostel.

While the wave of negative thoughts taunted me for my failure to prepare for the trip, I also realized that I was stronger, psychologically and physically, than I thought I was. I walked alone in nature for 9 days, carrying everything on my back, in one of the most beautiful places in the world. Herein surfaced a lesson. I am capable of much more than I was giving myself credit for. Two years later, with shoes that fit me, I planned and completed the West Highland Way, and it was a brilliant adventure.

Another realization from The Drowning is that it is not the pain that the brain seeks, but excitement. There are probably multiple ways, both good and bad, to excite the brain. In my case, at the time I found myself interested in close-up magic and illusions. Circumstances allowed me to go to The Magic Circle for their Monday socials, where professional magicians would share their knowledge and show off new tricks, trying to fool their peers. Similarly, the International Magic Shop near my office was my lunch break hide-away. For what a lot of people consider a child’s play, I found a hobby of depth, breadth, and skill that excited my mind. I also enjoyed the illusions others showed me, and kept focused while learning the skills myself. I have never told these people, but they really kept me afloat during this dark period.

Beware the easy path and take the harder one. It will be more challenging, but it keeps the mind busy. Where previously I had to forcefully repeat gibberish to drown out bad thoughts, a good hobby reduced the amount of effort required to keep me focused. Plus I can still do a few fun tricks for friends and family, which is good fun.

Lesson: You are much stronger than you think you are and the brain seeks excitement. Avoid the easy path and go with the positive.

The Meds

I looked at medication as a surrender or defeat, a final confirmation that something was wrong with me when all I wanted was to be a ‘normal’ human being. This partly came down to the fact that my mum is a psychiatrist and I was under the impression that only someone with serious mental illness came to her to get medication. I was never going to admit to myself that I reached that stage.

And yet when the apathy of The Drowning came, I knew that there were only two ways out of this misery, and I wasn’t going to leave without trying everything I could. I had to get over my unfounded principles, face my embarrassment, and take anti-anxiety and anti-depressant pills every day. I did not expect to feel the effects immediately and I was told it may take at least a month before I noticed anything.

And it did help. Gradually. There was less of the buzz and more of the happiness; less effort to pretend to be content, and more real contentedness about who I was.

Lesson: Whatever you may think of medication - it is worth trying. It is completely irrelevant what anyone else thinks about anyone taking medication. Including yourself.

The Empathy

Finally, and I think the most important lesson I have learned. Talking with other people helps.

I did not wear it as a badge, but I remember I met my former manager for drinks and just told him about how I felt and why. It was probably the first time when I decided to tell someone outside the family how I really felt. He did not judge, he did not try to find a silver lining. Instead, he shared something about himself. Where depression may come across as self-centeredness, in reality for the person who is feeling that way, talking is a really powerful antidote. Listening is too.

It seems like common sense, but let’s face it; everyone tries hard to hide the uglier side of life. Talking about it, mentioning things you find hard, and listening and validating the struggle of others helps.

Lesson: Everyone is struggling with something. You are not alone.

Final Thoughts

I have wanted to share my story to raise awareness for a while, but could not work out how to structure it. This post started as five bullet points I noted down before I went to sleep one night and expanded to include a lot more than I expected. Some of it is slightly uncomfortable and personal, yet I will stick to my original point. It is vitally important to talk about mental health. I got through The Plunge, The Pain, and The Drowning but too many people do not. In particular, too many young people. For me it was through medication, empathy and hobbies - for someone else it may be different. If anything here resonates and you feel similarly to how I felt - reach out. Depression is an illness and that means it can be treated.

Saving My Wallet With AWS Billing Alerts

2020-10-19T00:00:00+00:00

Well, this is a post that I did not expect to ever be writing, but here it is… Like the good DIY software engineer that I am, I was running my blog on AWS, with S3 for storage behind CloudFront CDN. The total weight of the Jekyll’s _site folder is 3.5Mb and my monthly bill was always below $3. What could go wrong? Cue AWS Billing alert.

Alert

Seeing the email subject line on my mobile phone screen, at first I discarded it, thinking it is an alert for a work account, which I expected to increase a bit this month. The next day, however, rummaging through my inbox I notice it again and with a raised eyebrow, try to log in to see my billing dashboard. I find a month-to-date bill of $75 with a forecast of $111. That’s not an exceptionally large sum for me but still - a stark jump (that grew even more - read on).

My first thought? Someone hacked my account. Although I do have free AWS security alerts setup amongst other things, but this is not infallible. As my eyes wander towards the breakdown by service the CloudFront CDN cost was around $60.

Next thought: did one of my posts get very popular? I thought the “AWS Like Service On boto3” was a neat idea worth a few more eyeballs enjoying it. Am I on HackerNews? A quick check of Google Analytics fizzle out this hope:

My blog is still a regular Joe. No new post = no visitors. So what is going on?

Forensics

First, what are the data transfer patterns? Maybe it’s just a temporary spike for whatever reason? Not really. After a slow ramp-up, it’s been 140GB/hour consistently for almost 24 hours.

Next up, where is it coming from? Virtually all requests were from France. I do love brie and baguettes, but I don’t think the French love my blog that much.

The average viewer is, supposedly, on a Desktop Windows machine and uses Firefox. As for more in-depth logs about viewers - my CloudFront distribution did not have any logging turned on. Probably for the best, because it would’ve racked up a bigger bill faster.

As unlikely as it is, maybe I too had forgotten some large file on S3, as happened to Chris Short in July. However, opening the popular objects page presented a different picture. Each URL on the website - HTML, CSS, JS & images - had been opened more than a million times, accumulating gigabytes of total data transfer, contributing to my monthly bill.

CloudFront tracks referrers, but referrer is notoriously unreliable as is the case this time - the referrer is my website or not-specified at all.

This is all I have picked up. As for whether this is malicious or crawler-that-went-rogue, I lean towards malicious. It is hard to believe something of such scale would’ve been able to continue for 24 hours and more so accidentally.

AWS Support

So the initial bill of $75 did throw me off, althought it would be okay if it were caused by legitimate traffic. However, it looks like a DDoS attack, and since CloudFront advertises basic coverage against DDoS, this is just something they have not caught up to yet.

I’ll just open a support ticket with billing to bring attention and move on, right? Well, not quite. In retrospect, this might have been naive, but “Amazon CloudFront, AWS Shield, WAF, and Route 53 work seamlessly to create <…> security perimeter against <…> application layer DDoS attacks” not of the goodness of their hearts. As noted more accurately in DDoS resiliency whitepaper - out of the box it only protects against transport and network layer attacks. This makes me more worried and I check the billing dashboard again.

So within hours things have escalated quickly to $375. Dreading the idea of having to pay this, I disable my CloudFront distribution, contemplate for a bit, and decide that email support is probably not going to cut it. It is time for a serious 1:1 chat with support.

Andrew from AWS answers within minutes, asks whether I expect(ed) that usage, checks in with an internal engineer, and confirms that AWS will cover the bill with credits. In the interim, I get pep talked by AWS Support engineer. :)

Conclusions

I have three takeaways from this incident.

First, this snippet has probably saved my eyes from popping out at the end of the month:

AWSTemplateFormatVersion: '2010-09-09'

Resources:

  Budget:
    Type: AWS::Budgets::Budget
    Properties:
      NotificationsWithSubscribers:
      - Subscribers:
        - SubscriptionType: SNS
          Address: !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:MySNSTopicForBudgetAlerts
        Notification:
          ComparisonOperator: GREATER_THAN
          NotificationType: ACTUAL
          Threshold: 80
          ThresholdType: PERCENTAGE
      - Subscribers:
        - SubscriptionType: SNS
          Address: !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:MySNSTopicForBudgetAlerts
        Notification:
          ComparisonOperator: GREATER_THAN
          NotificationType: FORECASTED
          Threshold: 100
          ThresholdType: PERCENTAGE
      Budget:
        BudgetLimit:
          Amount: 25
          Unit: USD
        TimeUnit: MONTHLY
        CostTypes:
          IncludeSupport: true
          IncludeOtherSubscription: true
          IncludeTax: true
          IncludeSubscription: true
          UseBlended: false
          IncludeUpfront: true
          IncludeDiscount: true
          IncludeCredit: false
          UseAmortized: false
          IncludeRefund: false
        BudgetType: COST

Second, in case of emergency, and for a private AWS account, choosing the chat option is best when creating a support case.

Third, the reasons why someone decided to do this remain unknown to me. I am still curious as an attack like this must have cost money. A quick google search leads to “The Cost Of Launching a DDoS Attack” on Securelist:

The cost of a five-minute attack on a large online store is about $5

Attack on this blog took at least 24 hours before I turned the domain off for a day, and generated a transfer rate of 140GB/hour. It definitely must have cost well above $100, if not more. And if that is true, someone must have been very proud of taking a $2/mo website down paying orders of magnitude more. Bravo! Unless the target was not me directly, but then again - who knows?

Build System At Genus AI

2020-07-08T00:00:00+00:00

Having worked at two startups from day one - YPlan and Genus AI - there is this exciting period of high-velocity and high-productivity when changes are fast, staging is the development machine, and release is just a few quick commands away. It is energizing, invigorating, and rewarding as you can see the prototype, MVP, beta, or v1, materialize almost out of thin air.

Yet, as the project and the team grows, this process gradually becomes slower and more complex. There is more to coordinate with the team, the stakes of making a change are also getting higher. If unchecked, this emergent friction results in release times of one day, a week, maybe even a month. In some cases that may be fine, but I am a firm believer that it is unacceptable for a startup to have a lead time¹ longer than a day.

My obsession to ship fast comes from seeing first-hand, at YPlan, how customers, both internally (tools) and externally (product), light up when they see their problem fixed or solved sooner, rather than later. Joining Genus AI as the first engineer in 2018 I did enjoy those vigorous first months, and as the talented Adam Chainz (of the Speed Up Your Django Tests fame) later briefly joined us - we both wanted to keep the same pace. We not just succeeded, but even improved the setup, as compared to that of YPlan. Here’s how it looks in 2020.

Builds And Deployment Frequency

This graph shows monthly build and deployment counts for the past 15months.

Some key changes visible in the graph:

There’s a dip before July 2019, when I was a single engineer behind the project and focus was on hiring the team.
From July 2019, the engineering team size increased by one person roughly every month until October. The builds and deployments gradually increased and plateaued at that time.
January 2020 is when our first full-stack engineer joined the team and we started rapidly converting the website from hard to maintain Django templates, to a React-based website running on top of a GraphQL API.

Most recently we have been doing about 1500 test builds and between 150 and 200 deployments per month, or, assuming 20 workdays per month, about 75 test builds per day and 10 deployments. That is not a bad frequency for a team of five people, considering that our industry-standard, according to DevOps Research Association² (DORA), is between once a week and once a month.

Something that might be interesting to look at is a further breakdown of builds and deployments.

It largely varies, but we aim to submit our work early for review feedback. Sometimes changes may not even pass all the tests, sometimes the feedback involves partial adjustments. One thing worth mentioning is that we try to keep our build hierarchy flat. Here’s what I mean - whereas if applications A & B both depend on some package C. If A or B are changed - only their test builds start. If C is changed - A, B, and C test builds start; this primarily works because we use a monorepo and explains the high succeeded-to-failed build ratio in the graph. One change may result in more than one test build.

Next up, is the breakdown of deployments.

A really important factor in shipping to production is the confidence and safety of doing so. This is where a continuous deployment pipeline comes into play. Deployments at Genus AI are done in two all-at-once stages - backend (asynchronous workers) and frontend (website and API). After deployment to the backend, final verification checks are done, and if successful - the process continues with deployment to frontend. These checks can fail and backend operates at a reduced capacity - part or all of it is not working. Such failures are represented by the failed deployments in the graph. However, the frontend remains operational. It is certainly not perfect, and I imagine we will have to improve this in the future.

This deployment approach is primarily enabled by the habit of doing small changes. Small changes, individually, have a small impact radius. Fixing an issue majority of the time is a simple revert, and releasing it takes less than 15mins, but more on that later. If something went through and broke frontend - we usually looked for ways to catch regressions at test build time. That is a contributing factor, why our failed deployments are relatively low, but failed builds, in the “Builds per Month” graph, quite high.

When new team members join, they have to do a warm-up exercise on day one and get some code deployed as part of that. The warmup exercise ends with a note: “Don’t worry if something doesn’t work and feel free to make mistakes.” On one side, the purpose is to get the new person ship code fast, and not get terrified of failure. At the same time, it is a test of the verifications done in the deployment pipeline. As the graph shows - it stopped deployment of a few changes that would’ve broken frontend.

Time To Production

According to DORA, the industry average lead time, from code accepted and committed to deployed to production, is between once a week and once a month. Below is the average Genus AI build and deployment times for the last 15 months. It is less than five minutes:

It is worth emphasizing this average does not capture variability. We do have some builds and deployments that take more time. Most of this speed in test builds is achieved by executing multiple different types of builds, e.g. code tests, integration tests, migration checks, security checks, merge checks, in parallel. An interesting effect of this speed is that our major bottleneck to deploying code is not the release process, but the code review process. However, bottleneck being there allows some flexibility: trivial changes - go out fast; changes that are chunkier - may take longer. Hence why breaking things down (Part I & Part II) is so important for us.

Availability

We operate on a very loosely interpreted rule - “it should not break twice.” We accept the fact that things will break, and when they do, we triage it as something low-impact, and keep track of occurrences openly, or high-impact, and look for ways to fix it immediately, and with a regression test.

As mentioned earlier, we do have partial failures regularly, that get reverted. The graph above shows the percentage of successful responses vs 5xx errors. It fluctuates just below 100%, with the lowest point being 99.6%. However, considering the test and deployment pipeline weeds a lot of the issues, and the changes we do are small, the issues causing dips in this graph are usually low-impact, localized to a specific part of the product, result in partial performance downgrade or a delayed background task processing.

Another important factor is the engineering team is located in a timezone significantly ahead of our clients. We use this to our advantage - failures are fixed before they become visible. Considering our lead time to production is about 15 minutes, we usually roll-forward issues with revert commit, then take time to reflect, and implement a fix.

We are small, growing, startup so this nimbleness allows us to make a trade-off like that. A larger-scale B2C startup, or a very large established product, might not have such luxury, but we do for now and use it to our advantage. Considering the focus of the industry towards high availability, it is easy to forget that sometimes the cost of not-fixing something (or letting it fail for a short time) is lower than the cost of not implementing an important product improvement somewhere. I think the affinity towards 0% downtime at a time when product is growing, without any regard to context, is a waste of resources, but mileage may vary case-by-case.

The Setup

At a very high level - these are the principal components enabling all of the mentioned traits.

We use a monorepo codebase for the same advantages Google does ³, it is just a natural thing to start with on a new project, and I had seen it work. Phabricator is the tool for code reviews and build orchestration. Arcanist command-line tool (arc) provides a very simple developer workflow:

Engineers work on code locally, on top of the trunk, and when they want to test it automatically and/or get their code reviewed, they run arc diff. This command submits the changes for review, as well as triggers the automated build and testing pipeline. Engineers can break down big changes into a stack⁴ of smaller ones. When changes are reviewed and accepted, the engineer runs arc land, and deployment to production is taken care of automatically.

It is so boring⁵, that since the inception in 2018, this setup required virtually zero maintenance and scaled well with the team and the product. The only changes required are adjustments in build commands or adding/removing additional builds, once in a while.

Cost

We religiously tag our resources on AWS, so Cost & Usage Report section in the AWS Console can show a pretty accurate report for the build system. The 0 in June 2019 is due to the cost being covered by AWS credits.

Back in 2018, we invested about 2hours at the end of the day for two weeks to build this. The low cost of this, and the fact that this system scaled so well for 2 years in a row at such a low monetary and maintenance cost - it is not a bad investment.

Interestingly, this might not have happened. When we were setting up the CI/CD pipeline in 2018, Phabricator had a ready-made CircleCI plugin, that used APIv1, that we installed. However, CircleCI decided to sunset its APIv1 and gave about 30 days to migrate. Since we didn’t feel confident writing robust PHP - rewriting the plugin was not an option. We decided to look for alternatives and settled on AWS CodeBuild.

Conclusion

I wanted to show you that for a small and nimble product startup build and deployment pipeline does not require a heavy upfront investment, expensive 3rd party services, dedicated build clusters, Kubernetes, or toolchains large organizations use. What is needed is a robust way to launch a build container, execute commands, and get the results in some form. That’s what AWS CodeBuild, AWS EventBus, AWS Lambda, and Phabricator - a boring infrastructure - enabled us to do at Genus AI: focus on building a cool product.

[Footnotes]

Lead Time - time from accepted code change to release to production. ↩︎
DORA - DevOps Research Association. A Google owned research body researching and promoting good DevOps practices. ↩︎
Why Google Stores Millions Lines Of Code in a Single Repository from Communications of the ACM:
- Unified versioning, one source of truth;
- Extensive code sharing and reuse;
- Simplified dependency management;
- Atomic changes;
- Large-scale refactoring; (our use-case: Large Scale Refactoring With PyBowler)
- Collaboration across teams;
- Flexible team boundaries and code ownership; and
- Code visibility and clear tree structure providing implicit team namespacing.
↩︎
Stacked Diffs Versus Pull Requests - Jackson Gabbard has done an excellent write-up. ↩︎
Boring Technology Club - I am a massive fan. ↩︎

How To Make An AWS-like Service Accessible Through boto3?

2020-06-29T00:00:00+00:00

A product or service on AWS may rely on countless AWS native services. However, quite often it is not enough, and bespoke built or configured systems are deployed. Usually, to use such a system a custom permission management is needed, custom authentication protocols have to be deployed, custom APIs are exposed, and used with custom client libraries, even though everything runs on AWS.

In October 2019, I had a lightbulb moment. What would it take to write a custom service on AWS and use it as an AWS service, through boto3 client? Codename “Elastic Unicorn Service”, like this:

import boto3

# Create Elastic Unicorn Service (EUS) client and
# print the name of unicorn with id 'u-00001'.
eus = boto3.client('eus')
unicorn = eus.get_unicorn(UnicornId="u-00001")
print(f"UnicornName = {unicorn['UnicornName']}")
print(f"HornLengthInFeet = {unicorn['HornLengthInFeet']}")
# Output:
# UnicornName = Calypso
# HornLengthInFeet = 3

After a brief period of going through the botocore source and experimentation, I managed to build a working prototype of “Elastic Unicorn Service”, and I am really excited to share the results in this blog post. I will dive into each part individually below, but it is surprisingly not hard to achieve this. There are only two principal components in this prototype:

botocore loaders
Amazon API Gateway service.

botocore loaders

At the time of writing this blog post Wikipedia page claims that “in 2020, AWS comprised more than 212 services.” So it may not come as a surprise that managing an API client for each service would be a behemoth task, and there are no S3Client or EC2Client classes in the code. Instead, boto3 uses botocore to read something called service model files. They describe the AWS service API call structure and are read by a class called Loader. This information is used to generate a client class at run-time (see this method). botocore also provides an event system with ample points of extension, to customize the generated classes. I have not looked at non-Python AWS SDKs, but presumably, client code generation is automated to some degree.

There is not a lot of information about the structure of the model files, apart from the explanation of the directory layout they are in, under loader documentation. But browsing through the existing service models as well as turning on boto3 debug output with boto3.set_stream_logger('', logging.DEBUG), can give an idea of what’s what.

For the purposes of this experiment three things are important:

endpoints.json file, like this one, contains the common attributes of AWS services - hostname pattern, signature algorithm, regions in which services are present and various custom overrides.
service-2.json file, like for AWS Config service. It defines the API calls and the structures for inputs, outputs, and error responses.
AWS_DATA_PATH environment variable or ~/.aws/models/ directory, where botocore looks for the above files to be able to generate the client classes.

I won’t go into a lot of detail about the structure, because I could not do a good service here - it is not documented and I just worked out most of it. Some of the properties and values were self-explanatory. When not - a good look at the source code and debug output usually revealed more detail. Instead, I will provide final examples.

API Gateway

The API Gateway service is used to provide the AWS IAM authentication and permission management. The actual API running behind API Gateway could be hosted anywhere. That is it.

The Code

My original experiment in 2019 used API Gateway MOCK integration to return a static response. While writing this blog post I realized I wanted to make the experiment more interactive. Hence, I have added an inline Lambda to provide a very basic Elastic Unicorn Service API with two calls: GetUnicorn and DescribeUnicorns. A short recording of how it looks in action:

The repository for the “Elastic Unicorn Service” can be found on GitHub and the README file should provide enough instructions on how to try it out.

Outcomes

In the end, this remains an experiment and I did not apply it anywhere. Mostly, because the line between whether this is a little known low-level API or an undocumented internal API is very blurred. It would not have been prudent to build out a service on an internal/undocumented API.

If this would be accessible, I can see a few immediately appealing benefits:

No need for custom authorization or authentication backends. Permissions can be controlled using identity IAM policies for API execution. Consider a policy statement like this attached to a role or user:
```
- Effect: Allow
  Action: execute-api:Invoke
  Resource:
  - arn:aws:execute-api:eu-west-1:012:api-id/Prod/GetUnicorn
  - arn:aws:execute-api:eu-west-1:012:api-id/Prod/DescribeUnicorns
```
Granted, it is not as nice as a named permission action could be (Action: eus:GetUnicorn), but it allows as granular permission management as the real AWS services.

The botocore Stubber infrastructure comes “for free”, with validation against the service model, simplifying unit testing.

import boto3
from botocore.stub import Stubber

eus = boto3.client("eus")

@pytest.fixture(autouse=True)
def eus_stub():
  with Stubber(eus) as stubber:
    yield stubber
    stubber.assert_no_pending_responses()

def test_get_unicorn(eus_stub):
 expected = {
     "Unicorn": {"UnicornId": "u-abcdef0123", "UnicornName": "Prongs"},
 }

 eus_stub.add_response(
     "get_unicorn", expected, {"UnicornId": "u-abcdef0123"},
 )

 actual = eus.get_unicorn(UnicornId="u-abcdef0123")

 assert actual == expected

Increased consistency between client libraries within the AWS Cloud.
There is no client code written at all. It is generated at run-time from service model specification.

Two immediate cons are:

Lack of documentation about the custom service models structure.
Lack information, whether the custom service models are considered a public API or not.

However, I do find this pattern of building AWS-like services on AWS very intriguing and I wonder what the official botocore maintainers would say about this. As far as I am aware, or was aware back in 2019, this has not been tried publicly before. I am intrigued to hear what you think. You can get in touch on Twitter or LinkedIn.

Making A Meal Planner With AWS Honeycode

2020-06-25T00:00:00+00:00

Yesterday afternoon AWS has announced a new service named AWS Honeycode (beta). Honeycode lets people build mobile and web applications without writing a single line of code. Many people find coding esoteric, hard to fathom, or just plain too steep to learn to get value out of. I find it an exciting proposition that many people could benefit from simple automations that could save them time. Until Honeycode, no one would be making them. So just to try it, I set out to build a simple application for myself and this post is about my experience.

My first impression of Honeycode was “Google Forms On Steroids.” Do not get me wrong, Google Forms is a great product to be compared to. Like Google Forms, Honeycode provides a spreadsheet as a data source, but it takes it further. You can build not only a data entry form around it, but a list page with search, details page, and connect them into a simple, but cohesive app. Second, Honeycode uses the all too familiar spreadsheet macro language to adjust the data here and there (filter a sheet, vlookup something). This is a brilliant idea because I think it flattens the learning curve in a major way. Anyone who has seen Google Sheets or Microsoft Excel, and went through a basic computer literacy course at school should be able to connect the dots. Finally, there are automations - send an email on data entry, or update a record in a different sheet, on some condition. Without automations, it would just be a spreadsheet with a form and maybe a few other sheets to list data in different ways. With automations, something can happen after the data was entered or changed. I think this is what closes the circle and makes it into a real application building tool.

AWS Honeycode has a good list of starter templates, from a to-do list to event planning, to weekly team organizer, but one use-case I tried to address with Honeycode is to build a meal planning application. At home, my partner and I try to preplan most of the dinners for a week ahead and I must confess I find the task daunting, for a few reasons:

I have to remember a chunky list of recipes I’ve made in the past;
I have to then check my recipe notes in Google Keep, Google new ones, or just plain scour through our recipe bookshelf to get a list of ingredients.

Of course, there should be an app for that. Maybe there already is, but I am not willing to spend $15/mo on subscription I know I will only use on Sunday evenings, but that will nag me daily with notifications. I’m thrifty like that. But then again, this sounds like a perfectly valid use-case for a tiny app on AWS Honeycode. Lo and behold, after about an hour playing with bells and whistles a not very cleverly named “Home Organizer - Meal Planner” web application was born. And in a true developer fashion, I have “Test Meal” and “Another Dish” recipes planned for today and yesterday.

And when I click on a recipe - it shows info where I can find that recipe in a book:

And if I don’t know what recipe to pick - I can see the list and search by ingredients:

Considering that it took me about an hour to make this prototype and have it live, on a service I haven’t seen before, I think it is great. However, I am very curious to try and get my partner on board to develop this prototype together further. And possibly inspire people, who don’t know how to program, to try this too, considering that it is virtually free for up to 2500 rows in spreadsheets and 20 users. That is plenty for recipe references and meal plans for just the two of us.

Of course, not all is flying colours. There is room for improvement too:

My biggest disappointment came when I realized the Android application was not available in the UK. This meant that I could not use the Honeycode on mobile (logging in on phone just points you to download the app).

Help was hard to navigate sometimes. Especially finding the list of spreadsheet functions for reference. The autocomplete tooltips could show not the parameter list, but the explanation too. Google Sheets do it quite well.

I did not use automations for meal planner, and there seemed not very many options available, past the basic: update a row, send an email notification, add a new row. I am sure the developers have a backlog of ideas and I hope one of them is an ability to invoke an AWS Lambda - that would open up interesting automation options for people with some coding basics. (e.g. I’d happily write a function to split the ingredients list into components, or a function to automatically search and add recipes).

Finally, I thought the app wizard was great. So great, that once I knew how to use it, I found it difficult to create screens from scratch. Which lead me to some confusion about how to pass around variables between screens. In the end, whenever I found myself struggling, I just created a temporary app with a wizard and copy-pasted the screen to my main app, and deleted the temporary one.

None of these are permanent problems. Considering AWS Honeycode is a beta product and the way Amazon builds things - ship fast, listen to feedback, and iterate - it will improve. But most importantly I am excited about the ability for people to build apps without writing code. An environment that enabled building this builder (sic) mentality, ability to try, break and fix things, was what lead me into the world of programming, software engineering, and constant learning. AWS Honeycode can become such an environment for a new generation of builders, whatever their age, race, gender, or economic status.

Bottom Up Problem Solving - Part II

2020-06-21T00:00:00+00:00

In the previous post, I have explained how a lot of programming is simplifying big problems and solving smaller ones in a sequence of changes. In this post, I will share my thoughts about the practical significance of following that model in a software engineering environment.

This bottom-up process optimizes towards making small iterative changes and derisking releases. Small changes are desirable because they are easier to understand and evaluate for code reviewers. Smaller changes are also inherently less risky. In a safe and fast continuous deployment pipeline shorter reviews and lower risk naturally drive towards faster release cycles. Faster release cycles mean faster feedback.

Having a fast release and feedback cycle can make the process of simplifying top-down and building bottom-up much easier in light of uncertainty. There are scenarios when trying to break down the big problem top-down may:

not be possible; e.g. not enough knowledge about the big picture;
require a lot of resources; e.g. a big or expensive undertaking; or
the end-result may not guarantee success; e.g. it has never been tried before;

Yet the fast feedback loop enables everyone to focus on the next iterative step towards a larger goal. This is when such small changes can drive results at the same time letting engineering and product team keep enough agility.

I have worked with people who, in an environment that facilitated this, were able to consistently do 20-30 changes per week. They did it by making small changes, that got reviewed quickly and released automatically. Each of those modifications carried enough weight to make the change visible too, and at the end of the week all these small marginal gains were adding up to a lot.

As a practical example, a serverless web application I work with, had its early changes releases staged like this:

Empty CloudFormation stack plugged into an automated deployment system.
Deployment of cloud resources required to run a serverless app.
Deployment of a “Hello, World!” app, with the beginnings of a test suite.
Deployment of a non-functional login form.
Deployment of authentication backend.
Deployment of login implementation.
Deployment of a non-functional user list.
Deployment of user creation form.
Deployment of an user list implementation.
So on…

This happened over two days and, by the end of a week, we had a prototype of a system that was ready to be put in front of clients. This is a specific example and doesn’t carry a lot of detail past being an abstract list, but it does show that this type of bottom-up building is extremely efficient if you want to move fast. In a startup environment, where most of my experience comes from, showing results fast can be paramount for company survival. Or, less dramatically, can make a difference between signing a client today, or “reconnecting in 6 months, when you have the feature ready.”

Nor is this a new idea or specific to software engineering. I am reading Atomic Habits by James Clear and in one of the chapters he writes about building “big” habits. The gist is:

When you start a new habit, it should take less than two minutes to do. Nearly any habit can be scaled down into a two-minute version, e.g.: “Read a book before sleep” starts with “Read one page.” <…> You have to establish a habit before you optimize it. Instead of trying to engineer a perfect habit, start with an easy thing. You have to standardize before you can optimize.

This is the same principle, applied to software engineering. There should be no hesitation or feeling of inadequacy by starting small and simple. When you join a new company, with an established project or team, it is very easy to be overwhelmed by the apparent complexity of the system and the tools. This may lead to attempts to match that complexity. Resist and remember - every complex system evolved from a simple one¹. Therefore it is good to imagine the end goal but always focus on the first step forward.

A non-obvious aspect of this is the need to break the parity between product features and code changes. I have worked in one or two companies, where the expectation was that one feature-request, bug report, or another task for product improvement requires one code change (in a form of a single PR), and needs to be released at once, or in a bundle with a few others. There are a few subtle problems with this approach.

First, the way the big picture is broken down by someone outside engineering organization may carry assumptions that do not match the code structure or even engineering processes. That is not to say that the product manager is doing something wrong - no. What it means is that an engineer should still be able to apply this bottom-up principle and break a task further down, if it makes sense.

For example, an “Implement a Forgot Password” feature request could become:

Deploy an empty page at /forgot-password.
Update database user model with reset token and timestamp.
Implement, test, and deploy password reset functions for verifying token and setting a new password.
Implement the form at /forgot-password and connect it to the code from step #3.

Second, even if the changes are broken down into smaller chunks, this expectation of feature-change parity may lead to bundling. Bundling is releasing related code changes all at once, even if they are individually small. It immediately increases the risk that something will go wrong. And since any individual part in the bundle can break - it may be unpredictably difficult to roll-back or roll-forward a fix.

Derisking means breaking down and releasing changes not necessarily by what the task asks an engineer to do, but breaking it down by what the developer thinks they can confidently get reviewed and released fast. In some ways, the size of the change list should represent developers’ confidence with the system and the shared understanding with the team of what constitutes a simple vs. risky change.

As a second practical example - a very underspecified task for a proof-of-concept project at work has about 60 individual code changes attached to it. The reason is that engineers at that time acknowledged the fact that it was not clear which direction the product was going and opted for a series of tiny changes, verifying their assumptions at each step. At another time, an improvement to an existing feature had about 10 changes. Although in terms of new and changed code - both instances were of about the same size.

In summary:

resist complexity, even when you see it;
have a rough idea of what you want to achieve;
focus on the small and simple step that would move you there.

I hope that if you ever felt intimidated by a programming exercise, library, tool or a task at work, that this post gave you enough confidence, through my practical examples, to look for ways to simplify, break it down, and look for the next small change that would move you towards your bigger target.

Good luck!

[Footnotes]

In fact, this is better known as Gall’s Law, named after John Gall, author of “The Systems Bible”:

A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.

↩︎

Bottom Up Problem Solving - Part I

2020-06-08T00:00:00+00:00

Among the variety of strategies for solving various problems that apply to software engineering are top-down and bottom-up approaches. The top-down approach emphasizes planning and complete understanding of the system, and coding begins only after a sufficient level of detail has been reached in the design of the system. Bottom-up emphasizes coding and early testing but runs the risk that parts of the system may be coded without having a clear idea of how they connect. Top-down is the what and bottom-up is the how.

Both are indispensable tools in a system or product-building toolbelt, both transcend software engineering and go into adjacent areas. There are probably countless books and blog posts written about both already. So what is this post about? I want to use a simple example to show the difference between knowing what to do and how to do it. The first one is usually straightforward to express in words - “a system for …” or “a module to do …”. The second is what makes the first happen, and people just starting in programming, using a new library, or working on a new project may find it very intimidating.

Why do I want to show that difference? Because someone just starting may feel inadequate, looking at some simple exercise, understanding pretty clearly what outcome they want, but feeling frozen without any idea how to get there. The important message I want you to internalize, if you ever find yourself in this situation, is that there is absolutely nothing shameful or noob about starting with a “Hello, World!” application. Every software engineer I know goes through this process. Experience and practice only help with making short-cuts and broader leaps in the process I am about to show you. Sometimes these short-cuts may indeed backfire on the most experienced engineer as the times, tools, and technologies change. But that’s another topic.

Right now, let’s start by concerning ourselves with an exercise that should put us through our paces through this bottom-up process. As it happens, Harvard summer school CS50 course has a problem called Mario - it requires us to implement the following program using more or less only for loops in the C programming language:

Implement a program that prints out a double half-pyramid of a specified height, per the below.

$ ./mario
Height: 4
   #  #
  ##  ##
 ###  ###
####  ####

The first thing to note is that it should be obvious what result is expected. How to do it is a different matter. To put you at ease, I genuinely am not confident that I could write the solution without going through this process in some form first.

Simplify Top-Down

One of my teachers at university was notorious for collecting quirky and challenging maths problems and sharing them with students. He eventually published a little book with a collection of these problems and one thing in the introduction left an impression in my memory. He wrote: if you are genuinely stuck and don’t know how to move towards a solution, do not bang your head against a wall, but look at the problem statement again and see if you can relax some conditions and solve a more straightforward problem. Usually, an insight into how to solve the big one comes more naturally after the simple one is solved. It is not a new idea, but I think it’s worth re-iterating, and it applies brilliantly here.

How does it apply in this case? If the final solution is the “top”, we will try to simplify the original statement multiple times, and in the following section, we will “walk” bottom-up to write the full solution. Let’s begin simplifying.

Implement a program that prints out two squares of a specified height, per the below.
```
$ ./mario
Height: 4
    ####  ####
    ####  ####
    ####  ####
    ####  ####
```
This simplification identifies that at each line, a certain number of spaces and # symbols are printed in alternating sequence: spaces, #, spaces, #. However, there is no requirement for the pyramid.
Implement a program that prints out one rectangle of a specified height.
```
$ ./mario
Height: 4
########
########
########
########
```
This simplification takes away the spaces and alternation.
Implement a program that prints a line of a specified length.
```
$ ./mario
Height: 4
####
```
This simplification removes the requirement of printing multiple lines.
Implement a program that prints out the specified length.
```
$ ./mario
Height: 4
4
```
This change replaces a sequence of symbols with just output.
Implement a program that takes input.
```
$ ./mario
Height: 4
```
This simplification removes the printing of the input value.
Implement a program that prints “Hello, World!”
```
$ ./mario
Hello, World!
```
This simplification removes the value input and is the standard starting point of any language.

Implement Bottom-Up

Starting at the “bottom”, the first step is to write a “Hello, World!” application:
```
#include <stdio.h>

int main(void) {
  printf("Hello, World!\n");
  return 0;
}
```
Assuming the above code is in mario.c file, compile with gcc mario.c -o mario and run ./mario. All subsequent examples will be changes to mario.c, so I will not repeat the compilation instructions.

Implement a program that takes input.

#include <stdio.h>

int get_int(const char* text) {
  int num = -1;
  do {
    printf("%s", text);
    scanf("%d", &num);
  } while (num < 0 || num > 24);
  return num;
}

int main(void) {
  int height = get_int("Height: ");
  return 0;
}

The get_int function does not change from now onwards, so I will omit it from example code and leave // ... comment where the function would be instead.

Implement a program that prints the specified length.

#include <stdio.h>

// ...

int main(void) {
  int height = get_int("Height: ");
  printf("%d\n", height);
  return 0;
}

Implement a program that prints a line of specified length.

#include <stdio.h>

// ...

int main(void) {
  int height = get_int("Height: ");
  for (int ii = 0; ii < height; ii++) {
    printf("#");
  }
  printf("\n");
  return 0;
}

Looking at this loop and knowing the next step is to print a rectangle should already hint that a loop inside a loop is needed.

Implement a program that prints out one rectangle of specified height.

#include <stdio.h>

// ...

int main(void) {
  int height = get_int("Height: ");
  for (int line = 0; line < height; line++) {
    for (int ii = 0; ii < height; ii++) {  // (1)
      printf("#");
    }
    for (int ii = 0; ii < height; ii++) {  // (2)
      printf("#");
    }
    printf("\n");
  }
  return 0;
}

There are two inner loops (1 & 2), even though the output is a rectangle, we know that the final result will have to have two halfs, so two loops here will make that easier.

Implement a program that prints out two squares of a specified height, per the below.

#include <stdio.h>

// ...

int main(void) {
  int height = get_int("Height: ");
  for (int line = 0; line < height; line++) {
    for (int ii = 0; ii < height; ii++) {
      printf(" ");  // (1)
    }
    for (int ii = 0; ii < height; ii++) {
      printf("#");
    }

    printf("  ");  // (2)

    for (int ii = 0; ii < height; ii++) {
      printf("#");
    }
    printf("\n");
  }
  return 0;
}

Rectangle is split into two squares by putting two space characters between two loops on each line (2). In addition, this adds spaces before the first square (1). This will help make the pyramid steps in the final step below.

Implement a program that prints out a double half-pyramid of a specified height.

#include <stdio.h>

// ...

int main(void) {
  int height = get_int("Height: ");
  for (int line = 0; line < height; line++) {
    for (int ii = height - line; ii > 0; ii--) {  // (1)
      printf(" ");
    }
    for (int ii = 0; ii <= line; ii++) {  // (2)
      printf("#");
    }

    printf("  ");

    for (int ii = 0; ii <= line; ii++) { // (2)
      printf("#");
    }
    printf("\n");
  }
  return 0;
}

The thing to note here is that the step (#) symbols increase with each line (2). The opposite is true about the space characters in the first sub-loop (1) - their number decreases with each line, relative to the height. Hence, ii = height - line.

By this time - the problem was simplified gradually and solution engineered from the ground up and, hopefully, the difference at each step is easy to follow and understand. For a more convenient medium of trying this code - I have put all of the above code steps in this repository.

If you liked this example and would like to practice something more challenging you might find the post about kilo - an uncomplicated text editor in 1000 lines of code - by Antirez¹ enjoyable. It has been turned into a series of short tutorials that anyone who prefers text and code examples can follow, build a text editor, and, most importantly, understand how each step combines into the final application.

It may not seem like much, but this type of problem-solving has a great practical significance in a real software engineering environment, but I will expand on that subject in a follow-up post.

[Footnotes]

Antirez is the author of a wildly popular in-memory key-value storage engine Redis. ↩︎

Review: Speeding Up Your Django Tests

2020-05-25T00:00:00+00:00

Disclosure: Adam is a dear friend, and I had the pleasure of reviewing the final draft of the book. Therefore, this post is very biased, but hopefully in a positive way.

Rarely have I read a technical book over the weekend and felt invigorated enough to go back to work on Monday morning to share and apply that newfound knowledge immediately. “Speeding Up Your Django Tests” by Adam Johnson is one of those rare reads. It may as well be the best practical technical book this year I laid my eyes on.

Having worked with Adam, I know him as the real 10x engineer. Someone who always leaves things better than he found them and can seemingly effortlessly ship code that gives the most impact. And he does that with speed, precision, and superb technical conviction. I am in awe of how he does it, so I am excited to have a book that gives a little insight into one aspect of Adam’s thought process.

This book draws out an easy to use blueprint for achieving the goal set out in the title. Regardless of whether you are a seasoned Django guru, someone who is just starting, or someone deployed behind legacy code lines - you should find something useful. Follow the “In a rush?” section and “Easy wins” chapter for 30mins or go through the whole book chapter by chapter, follow some of the tips, and you will notice results.

Easy wins aside, testing is not just about test suites and test cases, and “Speeding Up Your Django Tests” covers the vast surface area of this subject comprehensively: from test structure and fixtures to continuous integration, database configuration. The most beautiful part is that this inspired me to think “oh, I might just be able to pull this off!”

Whenever you find yourself waiting for that test build to finish running - flip a chapter, and you’ll find something to improve. For those who want to go deeper down the rabbit hole, the book is sprinkled generously with references and links.

The chapter on upgrade importance gives unequivocal data-driven evidence why keeping everything up to date is practically beneficial and shares a beautiful metaphor about it. Keeping things up-to-date works both ways and, knowing Adam, I am 100% certain he will keep this book accurate and up to date for the foreseeable feature.

Summing it all up, regardless of whether you are working on one Django project or many, alone or in a team, this book is one of those rare investments that are guaranteed to help you reap results.

Go ahead, buy it, speed up your django tests, and make that project fun to work on again.

Large Scale Refactoring With PyBowler

2020-05-19T00:00:00+00:00

I have been quite intrigued by sophisticated refactoring tools for a while, but never found an opportunity that’d warrant the time investment in learning how to use one, since the changes were always small. Lately, I had to remove some old code. While some IDEs (like PyCharm) have some basic refactoring tools, I am a creature of habit and stuck with Spacemacs. For that reason, my primary refactoring swiss-army knife has for a long time been Facebook’s codemod¹ and a regular expression of some proportion, in this particular case, it seemed extremely difficult.

Cue Pybowler² - a Python tool for safe code refactoring (announcement talk). It seriously lacks good examples, but after a few failed attempts, I found it very helpful. This post provides example code for the situations I encountered.

If you want to follow along - visit and clone the pybowler-example-repo Github repository. The example code is elementary:

# src/classes.py

class FooClass:
    def __init__(self, value):
        self.value = value

    def run(self):
        logger.info(f"FooClass.run<value={self.value}>")
        return self.value + 1

# src/functions.py

def run(val1, val2):
    logger.info("run")

    foo = FooClass(value=val2)

    return val1 + foo.run()

There is a class FooClass with method run, a separate function named run, plus some tests for both. I hope the fact there is so little of the code makes the understanding of how things work much more accessible. If you have the example repository set up - let’s dive straight in.

Changing Arguments

One of the simplest things that Pybowler enables out of the box is adding/removing arguments.

Every refactoring using Pybowler starts by building a Query, followed with a fluid interface calls to selector(s) (e.g., select_function) and filter(s) to find and narrow down only to parts of code that’ll be changed by modifier(s) (e.g., add_argument) with the final execute(). There are a few more pieces to this that I am leaving out in this post, but you can read more in the documentation.

# examples/01-add-arguments.py
from bowler import Query


def main():
    (
        Query()
        .select_function("run")
        .add_argument(
            "auto_param", '"default_value"', positional=True,
        )
        .execute()
    )

Run bowler run examples/01-add-arguments.py src tests to see it in action:

--- ./src/classes.py
+++ ./src/classes.py
@@ -7,6 +7,6 @@
     def __init__(self, value):
         self.value = value

-    def run(self):
+    def run(self, auto_param):
         logger.info(f"FooClass.run<value={self.value}>")
         return self.value + 1
--- ./src/functions.py
+++ ./src/functions.py
@@ -5,7 +5,7 @@
 logger = logging.getLogger(__name__)


-def run(val1, val2):
+def run(val1, val2, auto_param):
     logger.info("run")

     foo = FooClass(value=val2)
--- ./tests/test_functions.py
+++ ./tests/test_functions.py
@@ -4,7 +4,7 @@


 def test_run(caplog):
-    assert run(1, 1) == 3
+    assert run(1, 1, "default_value") == 3

     assert caplog.record_tuples == [
         ("src.functions", logging.INFO, "run"),

Notice that it alters function and method named run, even though select_function was used. I am not 100% certain if this discrepancy is expected behaviour or a bug, but either way, it is something to keep an eye out.

Renaming Methods

Another common use-case for refactoring things is to rename a method. In this case, two queries can do the job. Notice that first one looks for a function inside a class FooClass, not a method. It is due to the discrepancy above, but it does the job all the same.

# examples/02-rename-method.py
from bowler import Query


def main():
    (
        Query()
        .select_function("run")
        .in_class("FooClass")
        .rename("increment")
        .execute()
    )
    (
        Query()
        .select_method("run")
        .is_call()
        .rename("increment")
        .execute()
    )

Execute bowler run examples/02-rename-method.py src tests to see the results.

--- ./src/classes.py
+++ ./src/classes.py
@@ -7,6 +7,6 @@
     def __init__(self, value):
         self.value = value

-    def run(self):
+    def increment(self):
         logger.info(f"FooClass.run<value={self.value}>")
         return self.value + 1
--- ./src/functions.py
+++ ./src/functions.py
@@ -10,4 +10,4 @@

     foo = FooClass(value=val2)

-    return val1 + foo.run()
+    return val1 + foo.increment()
--- ./tests/test_classes.py
+++ ./tests/test_classes.py
@@ -13,7 +13,7 @@
 def test_run(caplog):
     foo = FooClass(value=1)

-    assert foo.run() == 2
+    assert foo.increment() == 2

     assert caplog.record_tuples == [
         ("src.classes", logging.INFO, "FooClass.run<value=1>")

Notice the first Query renames the FooClass.run method definition into FooClass.increment and the second one changes the invocations on class instances from foo.run to foo.increment.

A real-world use-case for this could be DEP-201 - a Django Enhancement Proposal that may cause a requirement to refactor a lot of url() calls to re_path()s. With a little bit of creativity, it’s not hard to come up with this:

from bowler import Query


def main():
    (
        Query()
        .select_function("url")
        .is_filename(include="urls.py")
        .rename("re_path")
        .execute()
    )

Run bowler run examples/03-django-url.py django_example, which contains urls.py:

# django_example/urls.py
from django.urls import url

from foo import views

urlpatterns = [
    url(r"^all/$", views.all_view, name="all"),
    url(r"^update/$", views.update_view, name="update"),
    url(r"^mark/$", views.mark_view, name="mark"),
    url(r"^mark-all/$", views.mark_all_view, name="mark_all"),
    url(r"^delete/$", views.delete_view, name="delete"),
    url(r"^redirect/(?P<obj_id>[\d]+)/$", views.redirect_view, name="redirect"),
]

Observe some changes:

--- ./django_example/urls.py
+++ ./django_example/urls.py
@@ -1,12 +1,12 @@
-from django.urls import url
+from django.urls import re_path

 from foo import views

 urlpatterns = [
-    url(r"^all/$", views.all_view, name="all"),
-    url(r"^update/$", views.update_view, name="update"),
-    url(r"^mark/$", views.mark_view, name="mark"),
-    url(r"^mark-all/$", views.mark_all_view, name="mark_all"),
-    url(r"^delete/$", views.delete_view, name="delete"),
-    url(r"^redirect/(?P<obj_id>[\d]+)/$", views.redirect_view, name="redirect"),
+    re_path(r"^all/$", views.all_view, name="all"),
+    re_path(r"^update/$", views.update_view, name="update"),
+    re_path(r"^mark/$", views.mark_view, name="mark"),
+    re_path(r"^mark-all/$", views.mark_all_view, name="mark_all"),
+    re_path(r"^delete/$", views.delete_view, name="delete"),
+    re_path(r"^redirect/(?P<obj_id>[\d]+)/$", views.redirect_view, name="redirect"),
 ]

If you have a real Django project, you could try bowler run examples/03-django-url.py path/to/your/project.

Using Complex Selector Patterns

Sometimes, a chunk of code has to be removed, and it varies so much that a regex-based approach makes it difficult to do. A situation I found myself recently was to remove a high double-digit number of asserts over pytest caplog.record_tuples, similar to this one:

assert caplog.record_tuples == [
    ("some.module", logging.INFO, "informational message"),
]

The issue here is that the expected part of the assert varies massively from file to file and from test function to test function. Module, logging level, the message itself make it challenging to write a regex to search and replace. Here is where Pybowler was helpful beyond a simple find and replace.

To be able to execute this change, I first had to write a lib2to3 pattern. The documentation³ does not explain well how to assemble it and it wasn’t obvious how to proceed past the examples. However, the announcement talk had some insight. So here’s the example pattern writing process I followed:

First, run bowler dump tests/test_classes.py. If you are trying to do this for your your case, change the path to the file that contains a piece of code you want to change.

Next, scroll through the printed syntax tree and look out for the parts representing the code to be changed, in this example - the assert statement.

.  .  .  .  [assert_stmt] '\n    '
.  .  .  .  .  [NAME] '\n    ' 'assert'
.  .  .  .  .  [comparison] ' '
.  .  .  .  .  .  [power] ' '
.  .  .  .  .  .  .  [NAME] ' ' 'caplog'
.  .  .  .  .  .  .  [trailer] ''
.  .  .  .  .  .  .  .  [DOT] '' '.'
.  .  .  .  .  .  .  .  [NAME] '' 'record_tuples'
.  .  .  .  .  .  [EQEQUAL] ' ' '=='
.  .  .  .  .  .  [atom] ' '
.  .  .  .  .  .  .  [LSQB] ' ' '['
.  .  .  .  .  .  .  [atom] '\n        '
.  .  .  .  .  .  .  .  [LPAR] '\n        ' '('
.  .  .  .  .  .  .  .  [testlist_gexp] ''
.  .  .  .  .  .  .  .  .  [STRING] '' '"src.classes"'
.  .  .  .  .  .  .  .  .  [COMMA] '' ','
.  .  .  .  .  .  .  .  .  [power] ' '
.  .  .  .  .  .  .  .  .  .  [NAME] ' ' 'logging'
.  .  .  .  .  .  .  .  .  .  [trailer] ''
.  .  .  .  .  .  .  .  .  .  .  [DOT] '' '.'
.  .  .  .  .  .  .  .  .  .  .  [NAME] '' 'INFO'
.  .  .  .  .  .  .  .  .  [COMMA] '' ','
.  .  .  .  .  .  .  .  .  [STRING] ' ' '"FooClass.run<value=1>"'
.  .  .  .  .  .  .  .  [RPAR] '' ')'
.  .  .  .  .  .  .  [RSQB] '\n    ' ']'
.  .  .  .  [NEWLINE] '' '\n'

The above syntax tree represents the following line of code:

    assert caplog.record_tuples == [
        ("src.classes", logging.INFO, "FooClass.run<value=1>")
    ]

The first thing to note is the above piece of syntax tree starts with [assert_stmt] node and has a NAME and comparison children nodes. NAME has a value assert and comparison - has more children. For the time being, don’t care about the sub children and use any* to match the subtree. It leads to a pattern like this:

# examples/04-math-asserts.py
from bowler import Query

PATTERN = """\
assert_stmt< "assert"
  comparison< any * >
>
"""


def main():
    (Query().select(PATTERN).dump())

Run bowler run examples/04-match-asserts.py tests to see this pattern in action and notice it prints subtrees for all assert statements. Just one example:

./tests/test_classes.py
.  [NAME] '' 'assert'
.  [comparison] ' '
.  .  [power] ' '
[assert_stmt] '\n    '
.  .  .  [NAME] ' ' 'run'
.  .  .  [trailer] ''
.  [NAME] '\n    ' 'assert'
.  .  .  .  [LPAR] '' '('
.  [comparison] ' '
.  .  .  .  [arglist] ''
.  .  [power] ' '
.  .  .  .  .  [NUMBER] '' '1'
.  .  .  [NAME] ' ' 'foo'
.  .  .  .  .  [COMMA] '' ','
.  .  .  [trailer] ''
.  .  .  .  .  [NUMBER] ' ' '1'
.  .  .  .  [DOT] '' '.'
.  .  .  .  [NAME] '' 'value'
.  .  .  .  [RPAR] '' ')'
.  .  [EQEQUAL] ' ' '=='
.  .  [EQEQUAL] ' ' '=='
.  .  [NUMBER] ' ' '3'
.  .  [NUMBER] ' ' '1'

To select only the assert caplog.record_tuples, the pattern has to be narrowed down to focus on the children of comparison node. Looking at the tree output - there is a power node containing "caplog" and a trailer node containing "." "record_tuples". The finished pattern takes shape:

# examples/05-remove-code.py
from bowler import Query

PATTERN = """\
assert_stmt< "assert"
  comparison<
    power< "caplog"
      trailer< "." "record_tuples" any* >
    >
    any*
  >
>
"""


def remove_statement(node, capture, filename):
    node.remove()


def main():
    (Query().select(PATTERN).modify(remove_statement).idiff())

Using this pattern would return the asserts we’re targeting. The final part of the puzzle is to write a modifier⁴ to remove that piece of code. Since the pattern matches the assert statement - it is the node we want to remove in the modifier. Hence, node.remove() in the remove_statement above.

Running bowler run examples/05-remove-code.py tests should show it in action.

--- ./tests/test_classes.py
+++ ./tests/test_classes.py
@@ -7,7 +7,7 @@
     foo = FooClass(value=1)

     assert foo.value == 1
-    assert caplog.record_tuples == []

 def test_run(caplog):
--- ./tests/test_classes.py
+++ ./tests/test_classes.py
@@ -15,6 +15,3 @@

     assert foo.run() == 2
-
-    assert caplog.record_tuples == [
-        ("src.classes", logging.INFO, "FooClass.run<value=1>")
-    ]

--- ./tests/test_functions.py
+++ ./tests/test_functions.py
@@ -6,7 +6,3 @@
 def test_run(caplog):
     assert run(1, 1) == 3
-
-    assert caplog.record_tuples == [
-        ("src.functions", logging.INFO, "run"),
-        ("src.classes", logging.INFO, "FooClass.run<value=1>"),
-    ]

The thing to note is that modifiers do not return anything and have to remove/replace/update the node(s) in place. This pattern is elementary, but looking at the existing modifier implementations (e.g. add_argument_transform) should give some hints on how to write more complex ones.

If it is not entirely clear what is happening exactly, I recommend printing or using ipdb to inspect the node instance, as well as capture inside remove_statement.

Final Thoughts

Pybowler is built on top of fissix, a lib2to3 backport and lib2to3 is expected to be deprecated by Python 3.12. Are there any alternatives to Pybowler? Unsurprisingly, there are. One of them is RedBaron - a community-driven open source project for writing code to modify code - its documentation may be slightly more beginner-friendly. A similar project, coming out of Instagram, is LibCST and its codemods with quite friendly documentation too. If you are working on a Django project, you may be interested in keeping an eye on django-codemod - its goal is to provide LibCST codemods to help upgrade Django.

Having used Pybowler now in a couple of instances and finding alternatives later, I want to share a few observations:

In a relatively small codebase (~55K lines) a regex-search and replace using codemod¹ covered the vast majority of use-cases so far.
If another opportunity pops up to require similar code modifications, I may look at RedBaron or LibCST. These two have extended documentation, more examples, and are actively developed.
My impression where this could be very powerful is when it comes to transforming large blocks of code. An example I can come up would be to switch a framework or transform some duplicated code to use abstraction. Still, I imagine these have prerequisites to be effective, e.g. a style-guide that supports such changes and experience in writing them, to name a few.

Hope this article and the accompanying pybowler-example-repo repository with a simple hands-on introduction to automated refactoring inspires you to look at automated refactoring as an option when search and replace does not cut it.

[Footnotes]

Facebook’s codemod: a tool/library to assist you with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention. ↩︎ ↩︎²
Bowler: Safe code refactoring for modern Python ↩︎
Pattern Syntax:

Selector patterns follow a very simple syntax, as defined in the lib2to3 pattern grammar. Matching elements of the Python grammar is done by listing the grammar element, optionally followed by angle brackets containing nested match expressions. The any keyword can be used to match grammar elements, regardless of their type, while * denotes elements that repeat zero or more times. Make sure to include necessary string literal tokens when using nested expressions, and any* to match remaining grammar elements.

↩︎
Modifiers:

Modifiers in Bowler are functions that modify, add, remove, or replace syntax tree elements originally matched by selectors after elements have passed all filters. Modifications may occur anywhere in the syntax tree, either above or below the matched element, and may include multiple modifications.)

↩︎

Jigsaw As A Metaphor

2020-05-08T00:00:00+00:00

There are many strategies for completing a jigsaws, as I have discovered over the past few days while attempting a 1000 piece one myself.

Regardless of which piece is chosen, it is impossible to start and immediately know where it goes and what other pieces it connects to.

One of the first possible steps is to take a piece and spend a long time looking at the reference picture to find its relative place. This is laborious, and slower the more pieces there are.

In the process of doing the jigsaw, many pieces are looked at. Some - more than once. Sometimes you cannot tell where a piece goes or which other parts it connects to so you put it to one side. Gradually over time an interesting thing happens:

Picking up a piece leads to a tacit knowledge that its counterpart was seen before and you put it on the table. It might not be in exactly the right location, but in the right cluster of interconnected pieces. Each small piece develops into a larger group, until all the groups come together to form one beautiful picture.

Consider briefly the process of assembling an infinite puzzle as a metaphor for learned intuition. I think it delicately captures many aspects of startup culture, team building, knowledge sharing, successful leadership, and I suspect much more.

[Acknowledgements]

Photo by Hans-Peter Gauster on Unsplash

Thoughts On @staticmethod Usage In Python

2020-05-05T00:00:00+00:00

At least one popular Python style-guide, Google Python Styleguide, insists¹ on not-using @staticmethod decorator and suggests to use a module-level function instead. Guido van Rossum has called the static method a “somewhat mistake”². Still, neither of these sources provide much explanation in practical terms - why.

I have also been a pronounced critic of @staticmethod decorator in code reviews at work. However, my colleague recently asked why do I dislike static methods so much, so I thought I would put it down in writing. It’s not that I dislike them, but I consider them a code smell that leads to issues down the line.

One argument I heard for using @staticmethod is that because a method is a utility and does not need the class or instance to do its job. Or another - it’s a way to namespace the helper under the class which uses it. Both sound like the correct thing to do. However, when it comes to Python code - I disagree. Here are a few aspects to consider.

API Design

Any function, module, or package has an API that an engineer is responsible for defining. A good API clearly defines the operations that can be invoked and how to invoke them. Most OO programming languages have helpful language features to enforce that. Unfortunately, Python is not one of those languages, despite sharing some similar keywords. For instance, even when it claims to have private or protected methods, they are more of a convention than actual scope control.

It may seem obvious, but it comes at the cost of some familiar things (like static methods) contributing to counter-intuitive outcomes. Consider a simplified example:

class Foo:

    def run(self):
        # ...
        self.get_something(param1, param2)
        # ...

    @staticmethod
    def get_something(param1, param2):
        return param1 + param2

get_something is a helper method, and it does not need any instance variables to do its job - so @staticmethod makes this clear. However, what is lost here is the scope of this method. Is it supposed to be private or public? Private, in this context, means it should be called from a class instance. Let’s assume private and supposed to be used just in instances of Foo.

Everyone comes with personal predispositions about what’s acceptable and what’s not in code - this whole post is about mine. For someone, it may seem perfectly reasonable to import Foo and call Foo.get_something in some other package, module, or class, without asking or checking. Here lies an interesting disconnect between the author and the user of this method: Author of Foo abdicated the control of the method to anyone who finds it useful.

An engineer who uses this piece of code may not be aware of the original intent or scope of this method.

In other words: a rule in codebase not enforced is bound to be stretched at some point - it is easy to notice such a situation during code reviews in a team of 5. In a team of 10 or more, it is a reasonable assumption that, without any other scope control, a method that is not intended but is available for public use - will be used as such at least once.

The above proposition is better known as Hyrum’s Law³.

Hence, if there is a way to enforce that scope, it should be used. If the author does not intend a method to be called without instantiating the class or hopes to keep it for use within class instance - it should be an instance method (a.k.a. without the @staticmethod decorator). Regardless of whether it needs to access anything in the class or not. The obvious benefit is that the interpreter will enforce this rule automatically, and get_something will only be callable from Foo instances.

Maintenance

As business goals, priorities, and requirements change - so does the code. What was Foo one day, may need to become Bar another. It helps to think about this when writing code. With a @staticmethod it’s a question of how many search-and-replace operations in a codebase you (or someone else) will have to do.

Let’s assume now the opposite of above - Foo.get_something is indeed supposed to be a publicly callable method. It is now an obnoxious dependency to manage. Consider this piece of code:

from app.foo import Foo

class Bar:

    def do_something(self):
        # ...
        Foo.get_something(param1, param2)
        # ...

To call the static method any other module needs to import the class (from app.foo import Foo) and invoke Foo.get_something(). If get_something is indeed a utility, by its nature, it is unlikely to change much over the lifetime of the module. However, the code using it, Foo and Bar classes, is more than likely to evolve and change. Assuming this is a somewhat popular utility method, it can quickly become frustrating to refactor the code. Say having to change Foo to NewFoo will end up requiring changes in all sorts of unexpected places - collateral that takes time to resolve when it easily could be avoided.

Hence, if there is a way to save your time running a search and replace project-wide and having to update tests and other seemingly unrelated code, it should probably be used. In this case, a simple module-level function will do the work. The classes that use it can be changed as much as needed, and the utility and its other invocations can be left alone keeping a refactoring scoped to only what’s changed.

Pragmatism

From a purely pragmatic perspective of wanting to type less, having to type self.get_something or a SomeWhatDescriptiveClassName.get_something is just a waste of time and horizontal space in a language which relies heavily on indentation and where get_something would suffice. Python is good at namespacing things with packages and modules and regardless of the intent to be public or private - a module-level utility function will do the job.

Teamwork

There is a natural tendency to write and think of the code as something existing right now, written by you - the author. We make assumptions unconsciously, and there are many of them:

the editor and the environment the code will be changed in will be the same as now;
the author of the code will remain the same;
an auto-complete will always be present;
build tooling will always be present;
the author will still be able to review the code and catch any misusage; etc.

Some of these assumptions are natural, but the older the project - the less likely they are to stay true. Some of these things will erode, and the code with all these little inconsequential bits and pieces will become a nuisance to handle.

Hence, if some of these problems can be avoided early on, especially just by the force of style and habit, they should. I consider @staticmethod usage to be something from that area, and that’s why I raise it during code reviews.

[Footnotes]

From Google Python Styleguide:

Never use @staticmethod unless forced to in order to integrate with an API defined in an existing library. Write a module level function instead.

↩︎
Guido van Rossum in python-ideas mailing list:

Honestly, staticmethod was something of a mistake – I was trying to do something like Java class methods but once it was released I found what was really needed was classmethod. But it was too late to get rid of staticmethod.

↩︎
Definition of Hyrum’s Law:
- With a sufficient number of users of an API,
- it does not matter what you promise in the contract:
- all observable behaviors of your system
- will be depended on by somebody.
↩︎

Tracking OKRs On Phabricator

2019-01-19T00:00:00+00:00

Last year I had read John Doerr’s book “Measure What Matters.” For those who are hearing about it for the first time, the book is about Objectives and Key Results (OKRs), a data-driven objective tracking approach that he introduced to Google. OKRs were invented by Andy Grove at Intel and documented in his seminal work “High Throughput Management”, but received the most hype due to Google.

Update: A lot has changed since 2019. Among other things - it also changed how we track OKRs with Phabricator. We stopped using the complex project / milestone approach described below and stuck with a bunch of subtasks.

I will not be explaining OKRs, there is a guide from Google and a very good talk by Google Ventures that I always recommend to everyone who’s interested:

What I really want to write about is a setup for tracking company and team level OKRs. I have recently tried it and it worked with a relatively small team that was looking for a process to establish focused sprints of work.

One of the problems I had in the past with OKR tracking was the fact that they were not directly linked to my day to day work. I would establish my objectives and define key results, but they would be stored in one system, whereas the day to day tasks which lead to the key results would sit in another system. This, in my mind lead to some issues.

OKRs, by definition, are a tree structure where information flows top-down and bottom-up. So any bottom level task should be traceable up to the exact team objective or company key result. Likewise, any company level objective should provide visibility as to who is working on the nitty gritty of it and the amount of work that is happening.

Being able to trace tasks from the volume of work to the OKR itself is not critical, but it helps with engagement by providing visibility into how day to day work plays into the bigger picture.

Enter Phabricator - my favourite swiss army knife suite of tools for, but not only for, engineering teams. Having recently set up OKR processes for a small team, I opted for this setup:

A top level project named OKRS - this is not used for much, except as a container for OKR sessions.
A sub-project for individual OKR sessions, say 6 weeks, with an incremental counter. For example: OKRS1 for session one between Feb 1st and March 14th, OKRS2 - for session two between March 15th and May 1st.
Each session uses milestones to establish objectives. Keeping the name down to one or two words and expanding in the description increases clarity.
Key results for each objective are put in the respective milestone column as tasks.
Team takes and breaks these down into nitty gritty tree of subtasks required to get the key result. This can go as deep or as wide as needed.

This does not follow OKR prescription to the dot and on its own it is nothing particularly remarkable, but with other applications in the Phabricator suite it integrates with and a small team size it works sufficiently well without requiring to pay for some dedicated product.

Free AWS Account Security Monitoring

2019-01-12T00:00:00+00:00

Recently I was pointed in the direction of AWS CloudTrail documentation and, specifically, something that I consider quite a gem for anyone who wants to get an early heads up about security events on their AWS account. Be it personal or corporate.

The gem in question is here:

Creating CloudWatch Alarms with an AWS CloudFormation Template

As the name suggests - it actually contains a link to a CloudFormation template with pre-made list of alerts for when:

S3 bucket policies, replication, lifecycle or ACL rules change
Network events happen
EC2 instances are started/stopped
CloudTrail configuration changes
Failed console sign-ins to your account
Authorization failures for any API calls
IAM policy changes

The template assumes you have a CloudTrail Event forwarding to CloudWatch Logs already set-up, and alerts are triggered by metrics generated using metric filters.

The default set of alerts is quite good and it can be tweaked and tuned for individual accounts, but the best part with this - I think - it is possible to have all this for free.

The first CloudTrail in AWS account is free for management events (even if it tracks all-regions). CloudWatch Logs free tier is 5GB which should be enough for a personal account running on a tight budget. Finally, there are 10 metrics and alarms in the template, which happens to be exactly the free tier of CloudWatch Metrics and Alarms.

Of course it is possible to tweak and tune for individual cases. Personally, I find CloudFormation templates in YAML format much easier to handle, hence I have translated the original to YAML. Download it here.

Too many times someone exposes credentials or gets their account hacked, only to find out about the breach when a thousand dollar bill rolls in at the end of the month. Fingers crossed attitude can only bring stress and anxiety. I think almost everyone wants to be calm about security and I think this neat gem from AWS CloudTrail docs is a perfect tool to contribute to that calmness.

Being Friendly On Email

2014-11-26T00:00:00+00:00

Managing email is hard. All notifications, error messages and marketing email aside I get a few actionable emails from people, a day. The number itself is by no means big, but the keyword here is actionable. Actionable means - it has an important information inside - which triggers an action - a code review reminder, a bug hunt request, a check in the code to confirm that a feature is not a bug, a reply to a partner or coleague, unblocking their tasks.

All of it takes time and we always want to save it to have more time for our own tasks at hand. And I, for instance, can be charged guilty for trying to save this precious time by reducing the email output: write less emails or if I can - write less words. Less words mean less information and human touch. Less of both result in miscommunication, uncertainty and general grumpiness.

I used to do this extensively - reply when I have something to share, with just enough information. If an actionable email came in - I used to reply when I had done it, unless explicitly asked for an estimate. Well, it took one book and a period of silent experiments, until I think I finally can say that my email etiquette has made an almost 180 turn, not full 180 because when overwhelmed - I sin and not reply properly or in time, and my personal inbox still hurts. But here’s how the good things happened…

There are countless blog posts and books about inbox management, what might set this one as unique - is the fact that the book which helped me was not about email management, but rather - about making friends. In fact a neveraging classic - “How to Make Friends and Influence People” by Dale Carnegie.

It is a simple book about human(e) interactions. I honestly regret not reading this book much much earlier, but I am happy that I did eventually. What I liked the most about it - it had practical examples of situations. For me - lacking in people skills - this was the backbone of the change and I started applying the principles from the book, that could be applied to email, immediately.

In the similar way as the book, I’ll present a principle and how I applied it. Hopefully, you’ll find something new and useful.

Remember that a person’s name is to that person the sweetest and most important sound in any language

Writing a name in emails for me feels rude, but if an email is genuinely about the person why start it with plain ol’ “Hi,” or other, when you can write:

Hi Name,

Feels awkward to start with a name in a multi-person thread, but if I appreciate what person does - I would try and use the name at least once in the main body.

Talk in terms of other persons interests

I use this one at work, if I have to write an email to our partners - I spend couple of extra minutes thinking what my goal is and how to reflect it in terms of what the recipient might be interested in. It is very easy to just go with “Hi, I want this and that done by then, otherwise I cannot do mutual interest here,” it is much more effective to write:

Hi Name, I am close to delivering mutual interest, but before I can do that I need couple of questions answered.

Make the other person feel important - and do it sincerely

I recently read an article, of which one idea stuck to me: leaders avoid using first person pronouns. So, striving to emulate a small leadership quality, I started to avoid using first person pronouns, except when absolutely necessary. In turn - the result is more about the recipient. It felt unnatural and insincere at first. Just because I have never tried to do this, but after a little practice - multiple rewrites of the paragraph - I would get to a satisfying result and feel genuinely happy that I do not sound selfish and the person on the other end, whom I might have never seen, would feel important - I started to sincerely want that.

Hi Name, mutual interest is not far from being finished and your guidance could help me take the last step.

Begin with a praise and honest appreciation

Raise hand if you started email with your needs and requirements. Well, working day to day we all usually are at a mercy of various requests that we have to eventually fulfill. We all strive to feel important and appreciated, but do we appreciate others? I certainly feel like I did not and even as I write this I think - I could do better. Why not spend an extra minute or two and do a recap of things already done and write one or two sentences about that?

Hi Name, your guidance was really invaluable for us so far towards the mutual interest. We are not far from the finish line and if you could provide us with more information - that would really make a big difference.

If someone did a good job taking initiative - mention it and thank for it. If you are just a man in the middle delegating requests that others implement - mention how the implementation is doing and how it helped. If long weeks of mutual work is finally over - recap and reflect how good the collaboration was! Heck, if someone replied quickly - appreciate the quick response!

People already spent or will spend time for your interest - at the very least say “Thanks a lot!” I know this sort of example smells generic - it is, but the point holds: there is always something to appreciate.

Let the other person save face

Too often something is mistyped or forgotten. At work people are busy and overwhelmed with more than work - it is easy to forget that mistakes are natural. Thus, instead of pointing to an error, why not think - is it really that important?

Here’s one idea from computer science - idempotence - that I applied multiple times with a lot of success. Roughly speaking it means that you can do certain operation multiple times without changing the result. I apply it by thinking: is the mistake bad enough that if I would ask to repeat the same action - something would break? If the answer is “Not” - I rephrase the question as if nothing happend and send it back. No harm, no bitter.

Hi Name, your guidance was really invaluable for us so far towards the mutual interest. We are not far from the finish line and if you could provide us with more information - could you send us documentation on XYZ?

Call attention to people’s mistakes indirectly; and Talk about your own mistakes before criticizing the other person

Another tool I use - is to not assume too much about what happened at recipient side of things and assume it was my fault. I am not a native english speaker, but at least 8 hours a day I speak, think and write in english. Mistakes are destined to follow me. When something is done in a different way than I asked - I assume it was because I was not clear enough. Rephrasing the question in terms of what I failed to say and what I meant (not pointing to failure to understand me) usually helps a great deal.

Hi Name, your guidance was really invaluable for us so far towards the mutual interest. I have read the documentation and I failed to find the part explaining what I need to do, could you pinpoint me to a page or keyword I should search for?

By admitting that you misunderstood something or failed to express something - you will be right most of the time, even when you’re wrong.

Ask questions instead of giving direct orders

Do you respond to “do this and that” with enthusiasm? Or “From now on things will be this way”? I don’t, though I used to do it, not that harshly, but in a similar manner. The fact is - noone likes to take orders, especially if they are not justified. Save yourself some time arguing and ask a question: “What do you think if ..?”, “Could you find time for ..?”, “Do you think doing … makes sense?” - what is worst that could happen? You’ll hear something you haven’t thought about.

Throw down a challenge; and Let the other person do a great deal of the talking; and Let the other person feel that the idea is his or hers

Just recently after a nudge on the side I had another thought about how I write specs: why do I take the juiciest bit out of the task? The exploration? The ideas? The possibility of slipping, getting up and finishing it just in time? Oh the excitement! I don’t know! I work with smart bunch of people and I decided to give it a try - my latest spec is “where we are now” and “where we want to be” and couple of observations where I think slippery places might be. So far it’s going great - there’s now a plan which sounds better than I had been thinking of and it is about to kick off. Hope it goes well!

No but’s and yet’s

I think this one is not from the book, but a rule that I adopted to great extent.

What do you feel when you read “You are great, but …” or “Things are working fine, yet …” - you just wait for something negative. Alternatively, what do you think when you read “You are great and …” or “Things are working fine and …”. Your defenses are down! Red alert! No!

The ugly truth is that if I say something relatively negative after “and” it sounds much much milder than after a “but” or “yet”, even if the whole sentence feels weird. Compare “This is great, but could be better.” to “This is great and it could be even better.”

Conclusion

To most - all of this probably sounds obvious or natural, but for me - it wasn’t.

I can even share couple of success stories: The changes were noticed by colleagues. One partner replied even while on holiday, although the request I had was nothing urgent, I just wrote it in a friendly way - I felt happy and appreciated the reply. Another, a big tech company every year running a big show in London, said it was a pleasure to work with us - which indeed was from our side too and this appreciation was exchanged multiple times. Various technical support communications are now more exciting when you feel that people care and try to reply quickly and even follow up unprompted.

I have yet to bring all this into my personal inbox, which lacks attention, and generally into real life people skillset. Sometimes I forget, but I guess practice just takes time.

If you feel you can relate to all this - I cannot recommend enough “How to Make Friends and Influence People” by Dale Carnegie - it has made a profound impact to the quality of my people skills.

Most Url Dispatchers Are Redundant.

2014-02-26T00:00:00+00:00

Last week as I was reviewing a small snippet of code adding yet another url pattern to application route table, I got struck by this thought:

wait a second, this url routing - it should not be part of application, but rather - of webserver. Application is responsible for taking input and producing output, and url parsing and processing is not part of that process. Meanwhile, webservers are super efficient at taking a request and passing it to a correct handler functions, via gateways.

Immediately, I noted it to myself and over following days I began to tinker this idea in my mind. Why did it suddenly pop in my head? Does it make sense at all to do that? If it does - why it is not done and how it should be done? What would be some immediate problems if suddenly I decided to switch all my application routing to webservers?

As to why it caught my attention, the answer is pretty easy: at my startup I happen to be two-in-one person. As a software engineer I work on the APIs our servers provide for mobile application; as a site reliability engineer I make sure that our infrastructure is in perfect shape to run the server code - part of which is configuring web servers. So, as I am sharing my time between the two parts of the whole picture, I suppose, the inception of this idea was a natural thought process.

Pros

Does it make sense? The more time I spend thinking about it, the more I am convinced that it actually does. Hear me out:

1. Web servers are more efficient at routing urls.

Probably all modern day web frameworks or libraries implement a url routing component. This component usually works by applying request url to a list of patterns to find first most specific match. Some of them do not even try to optimize this process[1], some of them - do[2, 3]. But they are all bound by the performance of the language they are implemented in and the overhead of request processing and initialization.

I may as well be an ignorant fool, but I think in this exact case a compiled language solution would always win and so it happens that modern production grade web servers are written in compiled languages. Also they have more options for URL patterns, e.g. they can be static, prefix based or regular expressions. This leads to my next point…

2. Web servers have better low-level[4] http control options.

After the routing is done, web frameworks require application to parse request, either directly or with support from helpers, to give some control over which HTTP methods are supported. Also, some of the frameworks force you into doing response caching at application level.

Again, this could be done by webservers - they are good at low-level http control as well as working as a proxy cache. It would free the application of a lot of redundant code.

3. Web servers are good at rudimentary url processing.

Imagine, a webserver matches correct url pattern, extracts the parameters and forwards them to the application handler. Well… here’s a snippet from Nginx configuration[5]:

    location ~ ^/api/1/resource/(?P<resource_id>\d+)/?$ {
        uwsgi_pass      app_server;
        include         /etc/nginx/uwsgi_params;
        uwsgi_param     SCRIPT_NAME /resource;
        uwsgi_param     PARAM_RESOURCE_ID $resource_id;
        uwsgi_modifier1 30;
    }

And a handler:

    import uwsgi

    def resource(request, response):
        response('200 OK', [('Content-Type', 'text/plain')])
        yield "Resource Id: {resource_id}".format(
            resource_id=request['PARAM_RESOURCE_ID']
        )

    uwsgi.applications = {
        '/resource': resource,
    }

Notice how resource_id is captured7 and passed along to uWsgi handler (upstream).

Now the framework/application is doing what it actually is supposed to be doing: given input - generate output.

4. Url patterns are inherently static.

Emphasis on the word patterns.

Any time a new website view (function) or resource (for API) is created - url pattern to access it is born, and it shouldn’t[6] change. In essence, meaning it is a static content. Gentlemans rule of thumb in web development is that serving static content (js, css, images) should never reach the execution path of your application and so - most of the static files on the internet are served directly by web servers[7].

Wouldn’t it make more sense to somehow deploy the mapping of patterns-handlers to webserver, rather than process them on every request?

5. Url routing becomes reduced to to a well defined interface.

I think this is the most impactful thing that comes out of the previous 4 points.

The single black-box component, that ties together all different pieces of many web frameworks together, is gone. A component, which more often than not forces the developer to a specific mindset or code architecture, is gone and suddenly - nothing stands in between a request and application code.

Please, give yourself a minute for this idea to sink in - a web framework is reduced to a well defined, standardized and transparent interface[8] that is plugged into a webserver. To me this sounds like liberating.

Cons

Thinking further, I tried to come up with countering ideas - what immediate problems this practice would cause. Disclaimer: I am very high about this idea right now, so most likely I am doing very bad job at countering it. More ideas are welcome.

First, url patterns are used two ways - to resolve request into a handler, and also a reverse - given parameters generate url. Wouldn’t this immediately force into duplication - in application and in web server? A dubious answer is - it depends. Obviously, if application needs them and web server needs them and you type them in both by hand - it is of course a duplication.

But then again, if one place is promoted as authority - let’s say application level - a tool that generates web server url pattern configuration file, that can be deployed together with code, would reduce the effect of duplication by automating it.

Second - testing. Some of the frameworks provide a testing facility, that allows to mimick url requests in unit tests or integration tests. At first thought - this would require to implement a url dispatcher nonetheless, just to make it work, but…

On second thought, does that testing facility actually test the url? No way - there’s a webserver in production which can drastically change the behaviour of any url, as often is the case. What that facility is doing - is just testing web frameworks url routing component. Does this test belong to unit or integration tests? No. More like acceptance or smoke tests.

So with urls in webserver, fake testing client becomes redundant and can (must!) be exchanged with a real http requests library. Further, acceptance tests are in their correct place and test the actual thing, rather than faking it.

Third - what’s the point? Well, with this I can not argue, but I can express my position, which is: I am a huge fan of clean and simple interfaces. I am a huge fan of being able to control the structure of the code as it fits the application and not a framework or a library.

Some might say, that there’s no point even thinking about it, because url resolving takes very little time wherever it happens. True! But… I am a huge fan of less code - if this allows me to shave a huge chunk (all) of code standing between my code and the web server - for the better!

Also, I am curious how one would benchmark url router under heavy load? I think benchmarks are hard and I haven’t done anything more except for a simple proof of concept that such setup could work, but my gut tells me that under stress - performance of non-webserver url dispatcher would diverge to the worse. (This is my speculation.)

History

As to why would it seem so ubiquitous for everyone to use a framework/library embedded url dispatcher I came up with two hypotheses:

1. It is historical legacy.

Long time ago web servers were configured to use one directory with multiple .html files as an entry point to a website. In fact, people were lazy to always enter index.html at the end of url, so there were options like DirectoryIndex, which would tell web server which file to load if none was specified. Then dynamic and template languages emerged and suddenly websites became a bunch of .py, .php, .cgi script files with HTML embedded between bits of logic, representing a page of a website.

2. Convenience.

Before long someone noticed that with the help of dynamic languages it is much easier to process all requests in one index.php file, rather than repeat a lot of bootstrapping code throughout the files. Also, web server management ten years ago was really cumbersome process, that was mostly frowned upon and thought in terms of “yuck, let’s finish this quickly and move back to the real deal.”

Fast forward to this day and adding a new url to application level code seems normal, because hey - all the frameworks and libraries do that.

What now?

Honestly, I do not know, but I will continue to tinker this idea further. What do you think? (Discussion on HackerNews).

Footnotes:

methods are supported and not Authorization or Authentication.

even then it is cached on webserver for performance reasons.

An Essay For Aspiring Engineers

2014-02-12T00:00:00+00:00

This article was originally posted in my Lithuanian (native tongue) blog and was received quite well among my colleague software engineers as well as non-technical friends. It was meant as an answer to a different blog post complaining about: aspiring programmers asking wrong questions about career bootstrapping, businesses and universities marketing constant lack of talent (encouraging too much people to study IT related fields) and all the worthless stuff the universities are teaching. Some of the bits apply only to situation in Lithuania, but I think the main point I tried to make is universal.

It all started with another blog post written by a person, who had studied computer science, does not work as a programmer, but has career in IT. I thought that I could walk through similar points she makes, providing my perspective, because I also work in IT and as they say - “I have seen some things.”

Questions

Seldom I receive questions on how to start a programming career. Where to start? What technologies to learn? Which ones generate the most revenue? It doesn’t happen very often, but when it happens I never give a straight answer, I give directions where to look for answers - books, blog posts, other people. Much like a solitary monk would say to his first apprentice: “You must find your path yourself, young one.”

Some could think that I am evasive, but I disagree. I do not know the correct answer to those and other questions: is my decision to be a developer - good? What are the qualities of a good programmer? How does a single day at work looks like? Honestly, I do not believe anyone could answer them - not even the last one. The difference between how my day looks like now and how it looked for someone, when I took my first steps learning a programming language, is like day and night.

However, I can give you one question with a constant right answer - who is a programmer? (software engineer & developer too - I’m using them interchangeably here).

A programmer is a person, efficiently solving someone else’s problem(s).*

* With each solved problem programmer is creating at least one for him/her-self.

When you look through this perspective - it is not important any more whether web or desktop programming jobs are better, but rather what problem a person wants and is able to solve efficiently. If you’re just starting out - internet is full of helpful resources, e.g. look for engineering blogs of tech companies[1]. Blogs full of articles about problems they were having and how they solved them. Read and learn! Get some perspective. If you are able to solve the same problem more effectively - you are guaranteed a job. If multiple companies have the same problem - maybe it is time to think about a startup.

Any technology you can think of is only a tool in developer hands, thus a question, like “which one to learn to maximize my income”, is obviously from absurd-land. The more tools you have - the bigger solution space you create for yourself to choose from. The bigger solution space you have - the more chances you will solve the problem effectively. Even this myth of “natural lifelong learning skill”, which engineers and developers are praised for, is nothing more than comprehending: “if I will always dig a well with a shovel, it is only a question of time until someone with an excavator will come to change me”. I know few people, who are developers but are far from enthusiastic about it, yet they have enough grit to follow this truth and keep their skillset up-to-date.

Sometimes, however, even importance of all available tools fade away in the light of someone’s ability to squeeze a maximum out of a single one. There are people[2], who are solving problems with Excel and/or Visual Basic (stereotypically an inferior language). I know at least one in person. And you know what… no one could beat them at what they do. Why? Switching would be too costly or breaking a human habit (users) - too hard[3].

Marketing

This was about various companies PR agenda too often being about lack of talent and boasting the size of income, which is usually couple of times than average monthly wage in Lithuania. Also, about universities agreeing on this agenda to boost number of student applications, because unis in Lithuania benefit from something called “student basket” - an amount of money they get for a single student. This, supposedly, had to increase the quality of studies. Don’t ask to explain…

Marketing is good and I will repeat the message - there is a lack of software engineers. Not any engineers, but those who are able to understand the scope and context of a problem, come up with a sensible solution and write source code to make computers (even thousands of them) solve that problem in an automated and efficient manner.

Companies, who are not picky enough about their engineers, often do not understand the cost[4].

On the contrary, companies that value their product quality are really harsh at picking talent. And I mean really harsh. Consider this: output of my university class was roughly 70-80%. Best case, a good company will hire only 1 out of 10. Best case.

In Silicon Milkroundabout[5] I had a chat with a colleague working at StackExchange (Disclaimer: I do not work for SE, but I call all engineers my colleagues). He said they hire less than ten out of hundred applicants. I couldn’t believe it at first, but the process is fine tuned to filter out only the best: a short call to acquaint with a candidate, a longer technical call, if there are doubts - another one, finally you’re invited on-site for even more technical interviews, sometimes lasting throughout a day.

On the other hand - universities are educating knowledge synthesizers and (ideally) - innovators. Before applying to university, a person must understand that there is not enough demand for that many people with higher education. (There is this absurd belief in my country, arguably a soviet relict embedded in my parents generation and they are pushing it further, that if you do not have diploma - you are worthless and won’t get a job.) For instance, some of my friends do not have a formal higher education in anything, but they are developing amazing things. I also had a chance to do an interview with a guy who, from the looks of it, dropped out of school to pursue a programming career. While I might not approve such approach, I was impressed with his ability to come up with optimal solutions to my questions and follow-ups, almost immediately. On the other end of the spectrum - some friends knew nothing about programming before uni and now are on a successful at software engineering career track.

Even better, when you are not a programmer by education, but know how to do it. I first heard this from a programming enthusiast in Copenhagen - he had quoted Zed Shaw[6]. Later, I had to hold my jaw closed when I found out who he was: general counsel at the worlds biggest shipping company - by day, Clojure hacker by night.

Studies

Disclaimer: this part is heavily based on quotes from that other article, but I’ll try my best to give context.

“Universities lag behind business requirements.” (This phrase is almost ubiquitous between students and businesses in Lithuania.) But following this logic it would mean that universities are training engineers almost for any business - be it template website “bakers” or banking systems providers. NO. Like, this NO!

Look:

Universities must prepare knowledge synthesizers who are, able to solve unique business problems.

Following this heuristic - it is the businesses that lag behind. It isn’t bad - some of them do not need brain-picking bespoke solutions. It is absolutely natural and normal. But imagine when after four years of ingesting all the important knowledge about software engineering and ready to work on the next big thing, one finally is tasked to mash together a website using a template and a CMS. That is why I felt cheated, didn’t write my thesis and dropped out after four(!) years. I completely lacked motivation and often sang the quote I started this section with. Took me couple of years, and career steps, until I understood what I missed, and I am fixing this mistake at my own expense now.

This next quote (about curriculum) also resonated with me:

However, you are forced to write a compiler, because “what if you’ll need to write one.” Maybe you could add aircraft flying to the curriculum? In case the whole crew passes out during a flight. You know, just in case.

It resonated with me, because once I was singing the same song - why the heck would I need to “write an operating system”, dig into “computer architecture” or how “computer networks” at a low level work? All operating systems are written too hard to comprehend for newbie, no new mainstream computers were invented recently (except[7]) and there are networking libraries ready to be used.

In the last 6 months I had to roll my sleeves and dive into Linux Kernel source code to understand why packets sent over network to rsyslog were being lost, when all the documentation I could find and my understanding of the domain said they should go through. Do you feel the irony? All three of them - low level code, operating system and network. I could’ve brushed it away with an incompetent “this is not my domain anymore, I don’t know why this is happening” and go after another, more rewarding, task. However, my present self was super happy that my past self wasn’t that dumb and didn’t skip all the classes, and those skipped - were compensated by self-educating. And I found the root cause and I fixed the problem: there is a limit to UDP packet size, which you can trivially increase, but only to as much memory as kmalloc can allocate, which, indeed, had a cap of 128k bytes.

There is a saying - ignorance is a bliss, and it fits here perfectly: there’s a programming language and compiler or interpreter, nothing out of ordinary, but if something is not working - it becomes a language limitation, it is compiler fault or interpreter performance issue. This is where all the problems hide from ignorance and incompetence - this is the bliss.

Software engineer efficiency is limited to the depth of understanding his/her stack.

Sounds intuitive enough - if a developer only knows how to use a framework, his efficiency will be limited by that framework. If an engineer knows only his language, all optimizations will be limited by the language features. If, after everything else, they also know language internals - they are competent enough to consciously write the most efficient code.

If you ever dreamt of working at companies like Facebook (or build one), think of the problems they were having - Facebook website code was too slow. Rewriting 9.2mln[8] lines of code without stalling every day progress - is an abysmal step[9]. What else? Either buy a thousand more servers or rewrite the underlying language interpreter into a compiler. They successfully managed to do exactly that[10]. For the same purpose, they redefined how data centers are built - hardware and software[11]. I am doubtless - engineers who pulled all this off are heroes among their peers. And - they knew how to write a compiler.

That indistinguishable half a second saved, between you clicking “Like” and seeing a response, accounts for millions of dollars when we are talking about tens or hundreds of thousands of servers, generating Facebook wall for all of us.

Finally

I think I used different arguments to tell the same idea as in the other article. It doesn’t rain gold bricks for a software engineer. On the contrary - work is full of intensive brain picking and, often, very little reward, because with every problem solved you have a new one for yourself - maintain the code. It all depends on problem solving skills and understanding of the toolkit, whether you’re going to do boring tasks or push the boundaries, wherever they might be. So if you are an aspiring software engineer - start by finding a problem to solve. It doesn’t need to be grandiose - solve one for yourself, your family or friends. And if you got hooked - continue, you are on the right path.

Hope you liked this article, please do comment or upvote on Hacker News.

References

Some additional reading, explaining where my thoughts are coming from. If you trust me - no need to follow them, but it’s recommended if you want to check the facts.

The original article this was intended to reply (in Lithuanian) - here.

[1]: Facebook Engineering, Twitter Engineering, LinkedIn Engineering, Netflix Tech Blog, Code as Craft.

[2]: My friends call me a scumbag because I automate my work when I was hired to do it manually. Am I?

[3]: We are not normal people

[4]: Bad Indian Programmers

[5]: Silicon Milkroundabout

[6]: Zed Shaw on Programming

[7]: D-Wave Systems

[8]: How many lines of code is Facebook?

[9]: Things You Should Never Do, Part I

[10]: Facebook Speeds Development With “HipHop Virtual Machine”, A 60% Faster PHP Executor

[11]: Facebook Saved Over A Billion Dollars By Building Open Sourced Servers

There Is A Problem With Web Frameworks

2012-02-27T00:00:00+00:00

This is an essay about a new way to build websites, a way completely different from what currently web frameworks (at least PHP) offer. I will dare to say that they all are flawed and then present you one way that to fix the problem, and will boast a little, that this way was presented at one of W3C workshops for guys from W3C, IBM, Nokia, Oracle and others. The way is also available for you to try out, as an open source project. Although this post is long, I hope you won’t be frightened by that and will enjoy reading it as much as I enjoyed writing it!

However…

TL;DR. Frameworks are bad because of object-relational impedance mismatch and leaky abstractions. We made Graphity - an open-source project. Read our position paper for W3C workshop here.

First things first. In the summer of 2011 I was working, among couple of other things, on a website called Wulffmorgenthaler. There is a different story behind the english version (just in case you have questions), but today I will concentrate on a Danish website and its successor - HeltNormalt. So, around the middle 2011, the owner of the website decided to do rebranding - new website with a LOT of new kinds of content. Its old codebase was terrible piece of craftsmanship, part of which, sadly, involves me too, but the rebranding meant a green light to a fresh start on the codebase. Oh boy, was I happy about it! No more whining about Other People’s Code and also I was anxious to not repeat my past mistakes and make my code even better this time.

At the same time there happened to be Symfony2 Beta. I had not used Symfony, but decided to take a look - especially as some of my good developer friends were buzzing about it. I downloaded the sandbox, read the documentation and some blog posts. Symfony2 looked as a fresh breeze after years with Zend Framework. I liked the structure of the code, I liked their approach to development/production environments and asset management. I thought, “I shall push to use it for our fresh start!”. Early adoption all the way.

But then again, I was not in this alone. We were three coders and while I happened not to be a lead, I did have a voice. Oh man, did I preach “Symfony2! Symfony2!”, but the final word was in other hands.

The lead - Martynas, had a long time and never ending interest in a technology called the Semantic Web. He deserves credit, for what happened in the following months. So the decision was made, to use his approach and make any tools neccessary along the way (there was not much of them for PHP anyway). Then - all hell break loose. I and another colleague, Aleksandras, both loudly objected, argued, discussed, SHOUTED IN CAPS LOCK and so on. But the decision was immovable and as the future showed - it was to the benefit of our own experience, the code quality and a different perspective about things.

The coding began… And during those months of development I saw that there is a drastically different approach to do things. I do not lie - it took maybe three months to grasp all this myself. Sometimes I didn’t even know what I was coding! Luckily, Martynas had a clear vision and lead us through, though there were still some discussions up until the end.

So, what did I grasp?

There is a problem with frameworks, because

They try to help me too much. I started to hate the idea of using any MVC Based framework. Why? is a good question. Here’s an answer:

Contrary to the “easy to learn” slogans, shiny documentation and easy examples, frameworks do not let me prototype things fast, unless I know them inside-out. And even if I am expert, I am constrained.

Consider this: a framework is missing a feature. There are two ways to solve this. I can search for a piece of code written for that mutual purpose by another great developer, and use it. But, 9 out of 10 times that piece of code does not fit my exact needs - I will need to “polish” it. Or, it fits my requirements, but the code structure is totally different. Or, it has no tests, but seems to work. I might end up writing it myself, hopefuly if I understand the problem domain good enough, the code will be pretty good too. However, even in this case I am constrained. I am bound to structure my code in the way the framework authors intended the framework to be extended. What if I want to be a free spirit and do things my way if I know I will do them better?

P.S. As I am writing this - “Linus Torvalds on C++” appeared on Hacker News. And I totally relate to his ideas, in a different domain though.

models are a fifth foot on a dog, and

I have long since heard that ORM is an anti-pattern and I agree with that. But I had always thought the problem is with implementations - no one created that right one yet. Now i have come to the horrifying conclusion, that Models in MVC are totally worthless piece of code. <irony>They are needed only for bugs to hide somewhere.</irony> This comes from realizing, that mostly I opened Model code to fix something.

Consider this mindflow: Requirements of (especially web) software are constantly changing and these changes mirror directly onto the data structure. When an unavoidable change to the data and logic arrives, you update them and in the end open up Model class, add/remove methods/parameters, change various queries to use those methods/parameters. It feels so unnatural, that when you change your data and your queries, you have to change something more. This is a constraint on data. There is a better way.

Your data should be your model - it should be self contained. Your code shouldn’t bother about internal representation, rather it should care about data transformation. Which brings me to my next idea.

views are impostors!

Yes, Views do not do what they should. Views are representing your data, rather than presenting. This is so subtle difference, that didn’t come to me quickly, but when I grasped this - I was shocked again.

I’d expected view to be, in one way or another, the same data I have in my database, except that it is presented in the way the human eye or a browser script could make sense of it. However, your commonly known View mocks itself by _re_presenting the data using a spaghetti of moustaches! View is also static and tightly knit to the underlying Model it tries to _re_present.

Here’s an example: a requirement change happens for some theoretical website. Some theoretical engineer opens up Model code, updates properties and queries in accordance to new requirements and then goes on to change the View.

It’s quite alright to update the views with information on how to present new properties, but why should a View care if those properties are available in the data or not? It shouldn’t.

The alternative is The Transformation - a thorough dramatic change in form or appearance.

Think of it like this: The Transformation thingy knows what properties our data might have and how to present them. Yes yes, tiny snippets of code. Then you query your data, retrieve some properties and push them through The Transformation. What comes out? Data is presented according to these rules.

If you change the query, add or remove properties, the data is still presented without any change to the transformation code!

I know, the difference might seem very subtle, but the implications are significant.

You can also consider this metaphor: a View here is like looking through a window in a picture - you will see the same thing until you make another image of the same window from a different perspective. A transformation, on the other hand, is like looking directly through the window and seeing a different view as you smoothly change your perspective.

How it should work.

What do I want from a framework, then? Primarily, healthy abstractions of the low level stuff that I need for a webapp. For example, I now argue (I wouldn’t have believed a year ago), that the simplest abstractions needed for a website are:

Request - something that comes from the client.
Response - something that goes back to the client.
Resource - the data.
Repository - a place to store, retrieve and update Resources.

Do you see the difference? Instead of hiding the low-level stuff that modern frameworks tend to hide, we embraced it! Indeed, last year, after months of work, when we finished the rebranded Danish entertainment website called HeltNormalt, there are just those four things behind the scenes. Yet, believe it or not, the new website holds more than ten different types of content compared to just two in the old one. Here are some statistics about code:

Controller code in the old website, LOC (Lines-of-Code): 7625 vs Resource code in new one, LOC: 652.
View code in the old website, LOC: 2528 vs Transformations code in new one, LOC: 1898.
Model code in the old website, LOC: 19125 vs Query code in the new one, LOC: 614.
Zend Framework behind old website, LOC: tens of thousands vs Our framework (Graphity), LOC: ~5000.

Less code - fewer bugs.

The Model/Query difference comes mainly from our ORM. We used Propel, which generated a lot of code. You might ask, what’s the Query thing? Well, we don’t have models - but we do query the data. The point is, because our data is autonomous, we need only to query for the stuff (properties) that we need. We need not describe the data as Models.

Putting it all together (IRL).

Let me explain how it all works in real life, on the HeltNormalt website, without diving deeply into what RDF and SPARQL (the Semantic Web technologies, behind the scenes)

Resources

Every resource in our datastore is comprises a number of triples, each of which is a Resource URI, a property name, and a value. For simplicity sake, a heavily striped down version of a resource looks like this:

    @base <http://heltnormalt.dk> .

    @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
    @prefix dct: <http://purl.org/dc/terms/> .
    @prefix sioc: <http://rdfs.org/sioc/ns#> .
    @prefix foaf: <http://xmlns.com/foaf/0.1/> .

    </striben/2012/02/28> rdf:type Post .
    </striben/2012/02/28> dct:issued "2012-02-28T00:00:00"^^dateTime .
    </striben/2012/02/28> sioc:has_container </striben> .
    </striben/2012/02/28> foaf:thumbnail </img/strip/thumb/2012/02/28.jpg> .
    </striben/2012/02/28> foaf:depiction </img/strip/2012/02/28.jpg> .

It is pretty straightforward and natural - every resource has an URL. Each URL (thus - resource too) can have named properties, where each property also has a value. A value can be another URL (thus - link to another Resource) or a string.

Remember, how I wrote that we don’t need Models, because our data is self-contained? This is what I meant.

P.S. The snippet above is a very helpful thing called Turtle syntax, though simplified here. Actual data is in RDF/XML.

Queries

Now, as we don’t have Models and ORM as such, we still need to get our data somehow. So imagine above triples as a graph - in the center there is a resource with edges going out (properties). On the other part of the edge there is a value - a string, or (surprise surprise) another resource linked to this one. Now imagine hundreds of resources linked this way. A web of linked data.

How do you retrieve information from this graph? By a thing similar to pattern matching! In the query you say that you want to get some triples with some properties and values, and leave some blanks that should be filled up in results. Sounds vague, but here’s an example:

    PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
    PREFIX foaf: <http://xmlns.com/foaf/0.1>
    PREFIX sioc: <http://rdfs.org/sioc/ns#>

    CONSTRUCT {
        ?uri rdf:type sioc:Post .
        ?uri foaf:thumbnail ?thumbUrl .
    } WHERE {
        ?uri rdf:type sioc:Post .
        ?uri sioc:has_container <http://heltnormalt.dk/striben> .
        ?uri foaf:thumbnail ?thumbUrl .
    }

A very similar query is executed when you type in address: http://heltnormalt.dk/striben - and you get the list of strips. In simplified form results look like this:

    <?xml version="1.0" encoding="utf-8"?>
    <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:foaf="http://xmlns.com/foaf/0.1">
        <rdf:Description rdf:about="http://heltnormalt.dk/striben/2012/02/28">
            <rdf:type rdf:resource="http://rdfs.com/sioc/ns#Post"/>
            <foaf:thumbnail rdf:resource="http://heltnormalt.dk/img/strip/thumb/2012/02/28.jpg"/>
        </rdf:Description>
        <rdf:Description rdf:about="http://heltnormalt.dk/striben/2012/02/29">
            <rdf:type rdf:resource="http://rdfs.com/sioc/ns#Post"/>
            <foaf:thumbnail rdf:resource="http://heltnormalt.dk/img/strip/thumb/2012/02/29.jpg"/>
        </rdf:Description>
    </rdf:RDF>

Sorry for the XML, but I promise - it is important. I hope you see how the pattern matching worked here. Just in case: query said “find me all resources (?uri) and their thumbnails (?thumbUrl), where resources have type Post, container </striben> and a property thumbnail.”

Transformation

Time to live up the promise - why XML is important? Well, because for transformations we used the most natural transformation tool available for XML. The XSLT. I won’t dive into XSLT, the Wikipedia article has some examples, but suffice to say that we can represent our data in any way we want - HTML, XML, JSON, Plain Text, etc. just by using different XSL stylesheets. We could even generate a valid SQL dump to import into MySQL database, but seriously - we don’t want to do that. :-) (But we did an exact opposite! We had to import old data.)

That’s one of the greatest outcomes of all this - logic is stripped down (but still there is some logic), what’s left for you is XML transformations. The thing is, you greatly reduce a chance of bug - your data can be incorrect but can not contain bugs or be invalid (as long as validation in datastore works correctly). And when we did have had issues when some properties were missing in resource, nothing broke, we had our XSLTs set up in way that just the part where that property value should be shown - it was not shown. No ifs, no template logic. And you get this pretty much by default if you use XSLT correctly.

Embrace the Open Source version of this!

Early in the beginning, Martynas said that after we finish with the website, we should extract the back bone system and present it as an open-source framework. Behold fellow hackers - The Graphity.

At the end of the process, we felt that we did not invent anything new

we just reused what was always there, but hidden. So sometimes we like to call Graphity not a Framework, but instead - an Architecture! In theory this approach should work in any language that is widely used in web development today. Martynas has a working Java version of the same thing, slightly more sophisticated because Java already has some packages to work with linked data / semantic web, so he did not need to write everything from scratch as in PHP version. Python? Ruby? I hope there will be a version for those languages, and others, too!

Oh, and by the way, you might be puzzled - where to store the data if you decide to play with Graphity? Well, I happen to know one SaaS company, Dydra - just register and use it. In fact we did and, oh boy!, how friendly and helpful they were through the process of developing HeltNormalt! A perfect example for me how customer care should look like. Seriously, check them out.

Adventures at M.I.T. and a paper about Graphity.

In fall 2011 Martynas found a call for participation in a Linked Enterprise Data Patterns Workshop and said we should try to enter this event, by writing a short paper about what we did and how and how this could benefit the Semantic Web movement.

We did write the paper, we were accepted, and in early December flew over The Pond to Boston, MA to do a presentation! Actually, then I felt that our presentation looked a little bit off (a Danish entertainment company website) among enterprise grands like: IBM, Nokia, Oracle, just to name a few! But then again, who cares about being a little bit off, when sitting next to Tim Berners-Lee and listening to other bunch of great people talking about this great technology and drafting the guidelines for it’s future.

If you are interested in the paper we wrote, it has more comparative information how this approach differs from todays common practice in web development. Read it here: “Graphity - A Generic Linked Data Framework”.

We invite you to collaborate!

I hope someone endured up until here :-) This is the most important part actually!

We truly believe in Graphity, but as our ways with HeltNormalt have parted - we can not spend a significant amount of time on it.

Although we do spend some hours per week improving it - more hands and minds are always better, so if you feel interested - don’t hesitate! Try it out and contribute, we will be there for you on our Github account, I also will try to write more about it on this new and shiny blog and you can always drop an email for me directly or info@graphity.org

What are your thoughts on this - let’s talk in comments!

P.S. This essay wouldn’t be real without some help and feedback from: Martynas, Aleksandras, Aurelijus, James and Adomas. Huge thanks, guys!

Julius Seporaitis

On Large Language Models

Hype Cycle

Real Life Trends Flatten

Training Data

Context Window

Experience And Tribal Knowledge

Groundhog Day

Natural Language Is Not A Programming Language

Prompt Engineering

Translators & Dictionaries

Career In Software Engineering

Fetishisation of AI

Banning LLM Development

Threshold For Concern

The Future Is Bright

2022 Year In Review

Family

Hobbies & Projects

Career

Books

Travel

Sports

Gigs

What’s Next

Let's Subscribe To Code Anthropology

The Reason Kubernetes Exist

There Is Always Someone Walking The Same Path

Copy And Learn, Don't Paste

My Personal Path

Final Thoughts

What Can 75,000 Pull Requests Tell?

Data Acquisition

Comparative Analysis

Change Review Frequency (CP2)

Review Speed (CP2)

Change Size (CP3)

Number Of Reviewers (CP4)

Knowledge Sharing (CP1, CP5)

Interesting Questions

How long does it take for a new engineer to open their first pull request?

What is the median number of files in the first pull request?

What is the median number of lines of the first pull request?

How many weeks does it take for an engineer to reach median and peak performance?

Which day of the week is the most active?

What impact to the number of pull requests opened did the first lockdown have?

Conclusions

References

On The Soft Aspects Of Technical Interviews

1. No Questions

2. Answering the Wrong Question

3. Failure to Admit a Lack of Knowledge

4. Inability to Accept Help

Conclusion

On Fundamentals

Requiem for Phabricator

All You Need Is Trunk

Write, Review, Merge, Publish

The Serendipity Of Relationships

The Power Of Integrated Application Suite

Multi-Repo What?

Scratching Below The Surface

Database Design

A Sad Loss

Merits of Mentoring

First Steps In Volunteering

So Why Do It?

Book Review Of 2020

Books I Learnt The Most From

Books I Most Enjoyed

Honorable Mentions

On Continuous Automation

Commands

Local Automation

Remote Automation

Isn’t this obvious?

What you get for free

When does this not work?

Conclusion

On Continuous Improvement Of Code