Copasetic goals can be broken down into steps. Steps can be thought of as problems to solve. For example, if a goal is to launch a product, then there are indubitably a number of problems to solve in building the product. Invariably, examining the various problems in need of solving will highlight that there are at least two groups.

One group is made up of easy problems. These are the steps we've done before; they are familiar problems we've solved previously. There's also a group of complicated problems; they are known unknowns. These types of problems are new challenges; they are hard. It's critical to differentiate the two types of problems and prioritize them appropriately.

It's human nature

Unfortunately, while leaders can categorize steps correctly, they oftentimes prioritize them incorrectly. There’s a natural tendency for people to inadvertently prioritize easy work ahead of harder work. Afterall, solving hard problems usually takes a lot longer and oftentimes it’s quite satisfying to accomplish something (even if it’s easy). Nevertheless, when we go after the easy work before doing the hard work, we do ourselves a disservice.

Hard problems are usually the ones that when solved, open up doors and create new and hip opportunities. What's more, the hard problems are usually the ones that throw a wrench in things, resulting in a project that's perpetually 80% complete. Occasionally, hard problems can't be solved either! So it's prudent to attack this class of problems first. The reluctance to go after the hard problem is oftentimes the difference between success and failure.

Monkeys and pedestals

There’s a notion from Google’s moonshot company, X regarding monkeys and pedestals that captures the tension of easy problems versus hard problems quite nicely. As Annie Duke, a world champion poker player and author notes in her blog:

“imagine that you’re trying to teach a monkey to juggle flaming torches while it stands on a pedestal in the town square. Two tasks are competing for your money, time, and attention: training the monkey and building the pedestal.”

Of course, one task is easy and the other is hard. It’s tempting to solve the easy problem and build the pedestal first. It's a quick win and in doing so, you can show positive progres! But as she aptly states:

“there is no point in building pedestals if you can’t solve for the monkey.”

Solving the hard problem unlocks the entire equation! Get the monkey to juggle flaming torches and the world is your oyster.

One thing at a time

When thinking about monkeys and pedestals, it’s additionally important to recognize that in any set of choices, there’s usually a monkey and quite a few pedestals; moreover, once you solve for the monkey, there will undoubtedly be future choices that present a tension between the two. That is, there are always monkeys in the equation but you only have to solve one of them at a time.

Putting monkeys into perspective

Throughout my career, I've certainly prioritized pedestals at the expense of monkeys. Unfortunately, I've usually come to this conclusion after the fact.

Monkeys present themselves in various forms! They can be incredibly hard technical problems, but in practice, they're more often found in seemingly mundane activities such as finding customers or getting people to actually use your product.

Indeed, building a product is oftentimes the pedestal and getting people to use it is the monkey. The software industry is awash with tomes describing how to find product-market-fit, the essence of building minimal viable products, and avoiding a solution that's looking for a problem; however, time and time again, leaders overlook this class of hard problems and solve for the pedestal. The technology startup graveyard is littered with headstones etched with "they failed to solve for the monkey."

Leadership is the art of influencing people to achieve a goal. Moreover, an effective leader is capable of breaking a goal into steps. Those steps must be grouped into easy problems and hard problems. Successful leaders attack the hard problems first, because those are the problems when solved, unlock the entire equation. Don't fall into the trap of solving for the pedestal because it could mean the difference between success and failure.

Can you dig it?

"Two Truths and a Lie" is a groovy icebreaker game that encourages players to share interesting details about themselves where two statements are true and one is a fib. In the game, the group guesses which of the remarks are true and which one of them is a lie. I love the game; unfortunately, I’m pretty bad at it as people quickly pick up on my fabrication. Nevertheless, I’m going to play with you and let’s see if you can find the lie:  

1. Developers spend more time reading code than writing code. 
2. More time is spent maintaining code than in initially writing it. 
3. Management accounts for the carrying cost of existing code when planning.

Which one of these statements is the falsehood? I imagine the lie is obvious, but strangely, as an industry, we consistently overlook a fundamental truth of software development and proceed to mislead ourselves, our customers, and stakeholders. We routinely overlook the costs of maintaining code. I categorize software maintenance as a carrying cost and if you do not appreciate its effect on a development team's capacity, you will routinely overpromise and underdeliver. 

Carrying costs are recurring

We rarely account for how much time and effort is required to keep existing code running healthily. Healthy running code is not a point in time but rather a span of time. It's a recurring cost that starts once software is in the field and used by people and continues until that code is retired. Thus, because we don’t think in terms of carrying costs and fool ourselves into thinking maintenance-like efforts are one-time events, we overpromise. Once code is in the wild, time is required for the care and feeding of it. What’s more, the time required is proportional to the surface area of the software. The more code that exists, the more time will be spent keeping it running. 

I say this with confidence because I’ve fallen into this trap! I’m guilty of underestimating how much work is required to keep the lights on. In doing so, I’ve overpromised and underdelivered to stakeholders. It’s an easy trap to fall into too: it’s far more fun to write new code and add new features. Delivering new functionality to stakeholders intuitively feels more exciting than telling them you fixed a series of bugs. Code maintenance has never been a fashionable activity one brags about.  

Accounting for carrying costs

If planning and resource allocation were copasetic and easy in software development, projects wouldn't be late and stakeholders would never be disappointed. Figuring out how much time to allocate to software care and feeding is a mixture of science and art. In my experience, the time required to keep the lights on is far more than you initially think. Accordingly, my advice is to err on the side of conservativity.

The art aspect of this process is in recognizing a team's capacity encompasses more than new feature development. Their capacity must include the care and feeding of running software. From a scientific standpoint, there’s years of knowledge around the various techniques for ascertaining a software team’s velocity; suffice it to say, as a leader, you absolutely must get a feel for how much work a team can reasonably accommodate in a span of time. Once you realize that the vast majority of stakeholders believe work means new features, then you can address the problem by clarifying additional time is required for maintenance.

One way to reason about capacity planning and feature work is to think in terms of surface area. I tend to logically think of a software project as made up of domains. The more domains an initiative has, the larger the surface area. Surface area is at tension with capacity. Capacity is the maximum amount of work your team can do. If you have a lot of surface area, you need a lot of capacity. Capacity is almost always your limiting factor; accordingly, carrying costs tend to accumulate as domains proliferate. 

Focus, focus, and more focus

Anyone who’s successfully built a software product from scratch intuitively appreciates the value of focus. A laser focus on creating value for specific users is another way of limiting surface area. As a product’s focus widens to accommodate more features, users, and use cases so too does the surface area of a product. You can hire to keep the pace of surface area, but this relationship is rarely in lock step. Hence, lean towards conservativity when accounting for your capacity to maintain what's going into the field.

There are certainly times when it pays to lean into racing out ahead (i.e. building this new thing right now will yield more users, etc). A bias to action can increase a product's surface area. In those instances, by all means, do it. Recognize that you'll have to pay down that veritable debt in some way either by temporarily slowing down, hiring, or cutting back on promises. Taking the initiative to pull head of the competition or capture a market in need is definitely part of the game! Understanding how and when to race ahead or pull back is how the game is won.

Acknowledge carrying costs and plan accordingly

I hope you enjoyed playing two truths and a lie! Indeed, the fib is that management properly accounts for carrying costs. Frequently this cost goes accounted in favor of new feature development. Sadly, by ignoring carrying costs, we continually overpromise and underdeliver. Acknowledging carrying costs will immediately make you a more effective planner.

An essential aspect of leadership is influencing and inspiring a group of people to achieve challenging goals. Tactically, you will absolutely need to understand your team's capacity and master stakeholder expectations. Software needs care and feeding. It needs maintenance to properly function. A healthy appreciation for the carrying costs of code in production will make you a more realistic and effective leader.

Dig it?

I was recently a guest on the copasetic Ship It! podcast, which is one of my favorite shows from the amazingly prolific folks at Changelog! Autumn, Justin, and I discussed the value of Platform Engineering, various lessons learned from my experience at Netflix ranging from the challenge of technology evolution, infrastructure migrations, saying no, and the importance of empathy when working with stakeholders. Plus, we talked about Productivity Engineering, how one measures productivity, and much, much more. Give it a listen!

Ship It! 96: Productivity engineering at Netflix – Listen on Changelog.com

It was groovy chatting with Justin and Autumn; what’s more, I’m super excited that the Ship It! podcast has spun back up. Autumn and Justin bring fresh new perspectives and a renewed energy to the discussion about all the things that happen after git push. Be sure to subscribe to Ship It! along with Changelog's other podcasts like Practical AI and, of course, the stalwart podcast that started it all: The Changelog.

I'd love for you to listen to the show and, of course, if you have any feedback, please let me know! Dig it?

As a leader, your job is to make decisions. There are a few copasetic principles I’ve used over the years that help me with decision-making. Two particular principles help in determining how long you should spend contemplating a decision and who should make the decision. Not all decisions are created equally; what's more, leaders don't need to make all the decisions. These two principles are essential and as you'll see, there's an equally important third principle that's frequently overlooked.

What are the consequences of the decision?

Some decisions carry more weight than other decisions. Accordingly, there’s the notion of Type 1 and Type 2 decisions, famously embraced by Amazon's Jeff Bezos. In this framework, Type 1 decisions are one-way doors requiring thorough analysis. You cannot undo these decisions, so it’s important not to rush into them. Type 2 decisions, on the other hand, are two-way doors. These decisions can be undone. The risk of making a bad Type 2 decision isn’t catastrophic thus the diligence required shouldn’t be arduous. You can make Type 2 decisions faster than Type 1 decisions. 

Differentiating between the two types of decisions is key to Amazon’s ability to move fast. As Jeff Bezos wrote in a letter to shareholders in 1997, he felt it important for Amazon leadership to appreciate Type 2 decisions for what they are and not mistake them for crucial Type 1 decisions. Confusing the two means:   

“The end result of this is slowness, unthoughtful risk aversion, failure to experiment sufficiently, and consequently diminished invention. We'll have to figure out how to fight that tendency.”

Clearly, this groovy framework works for Amazon and has certainly influenced me in my decision-making. Understanding a decision's consequence underscores how much time I should spend on making it. Higher risk Type 1 decisions demand more of your attention and careful thought.

Who should make the decision?

Determining the consequence of a decision is important, but it's equally imperative to determine who should make the decision. A leader who has built an amazing organization and who provides excellent context shouldn’t necessarily need to make a lot of decisions. If you have a capable team, there are a bevy of decisions you can effectively delegate. For example, as Reed Hastings once opined: 

“I take pride in making as few decisions as possible, as opposed to making as many as possible.”

Make no mistake, Reed certainly made decisions in his role as CEO of Netflix and those decisions were most definitely critical to the business. But many other decisions were made by his team of trusted lieutenants. And many of their decisions were delegated to their trusted teams and so forth on down. The idea espoused by Netflix leadership is to ensure everyone has amazing context and thus push decision-making to the appropriate level.

I certainly began my leadership career thinking I should make all the decisions; however, I've learned to embrace Reed's philosophy. I've come to realize that I can empower my teams to make far more informed decisions if I've provided great context.

The more important question to figure out

Distinguishing who should make a decision and the consequences of that decision are certainly important principles in decision-making. But for me, the first rule of decision-making isn’t either of these. Instead, the first rule is to determine when a particular decision needs to be made.  

There’s a tendency for leaders to demonstrate confidence and poise under pressure. The need to prove oneself as a component leader manifests itself as fast decision-making. But I’ve come to realize that many seemingly important decisions don’t, in fact, need to be made in the moment. In my experience, determining when to make a decision often helps uncover the consequences of it and accordingly who should properly make the decision. 

Deciding when a decision needs to be made doesn’t mean putting off decision-making! Situational awareness is key; bias to action is important. Additionally, analysis paralysis or waiting too long is obviously an ineffective practice. Consequently, deciding when a decision needs to be made may yield an answer of right now! Obviously, if that’s the case, then make sure you’ve gathered the skinny and make a call. 

By giving yourself a few moments to consider when a particular decision needs deciding, you reduce the chances of rushing into it. If you determine that an answer isn’t immediately required, you certainly need to figure out when it must be decided. In the course of determining the answer to that equation, you have time to reason through the consequences of the decision and can furthermore figure out who should make the decision.  

Decision-making principles

Leadership requires making decisions and I've found it helpful to apply principles. It's important to first figure out when a decision needs to be made. Moreover, determining the consequence of a decision will point out how long to spend making it. Finally, many decisions can be made by people closer to the problem as they're likely more informed than you.

You don't need to make snap decisions all that often. You also don't need to make every decision. Take the time to apply principled decision-making and you'll become a more effective leader.

Can you dig it?

I’ve been diggin' Code Llama lately. It's free, easy to set up, and using Code Llama via Ollama can be a lot of fun. Code Llama has already proved itself useful out-of-the-box; nevertheless, I was curious if I could boost my productivity by using retrieval-augmented generation (or RAG for short). I decided on pointing Code Llama directly at specific code for question answering. I wanted to see if Code Llama could help me understand an underlying code base and help me develop new features.

If you're willing to roll up your sleeves, this is a fairly easy process! All you have to do is write a little Python code.

What's RAG?

Briefly, RAG is technique that combines something specific (like code, documents, images, etc) with a more general LLM. In this way, as Kim Martineau explains, RAG improves the:

quality of LLM-generated responses by grounding the model on external sources of knowledge to supplement the LLM’s internal representation of information.

For instance, if I asked an LLM, such as OpenAI's ChatGPT, to summarize everything I've recently written about leadership, I'd be out of luck. ChatGPT has no knowledge of my recent blog entries as the model was trained before I wrote my articles. Nevertheless, I could provide the model with some relevant information, such as everything amazing I've recently written and then ask the LLM to summarize my leadership articles. Depending on the quality and quantity of the specific information provided, the LLM could surprise you!

In essence, via RAG, you provide the LLM with some specific information, and then the corresponding answers to your questions are more informed through the use of that specific data.

Code Llama and RAG

Leveraging RAG with Code Llama takes about 60 lines of Python code. What's more, I leveraged Ollama along with LangChain. You'll need to ensure that you've both pulled and run the Code Llava LLM via Ollama beforehand. The steps to create a RAG with Code Llama are:

1. Create an instance of an LLM.
2. Create a prompt (i.e. context for the LLM).
3. Load a directory full of code.
4. Split the various files from step 3 into chunks.
5. Store those chunks in a vector database.
6. Create a chain from all the above which generates answers to questions.

I created a simple class named ChatCode that has three methods: an initialization one (i.e. Python'sinit), ingest, and an ask. The init method creates an instance of ChatOllama using the Code Llama LLM and creates a prompt context.

def __init__(self):
    self.model = ChatOllama(model="codellama")
    self.prompt = PromptTemplate.from_template(
      """
      <s> [INST] You are an expert programmer for 
      question-answering tasks. Use the following pieces of retrieved
      context to answer the question. If you don't know the answer, 
      just say that you don't know. Use three sentences maximum and 
      keep the answer concise. [/INST] </s> 
      [INST] Question: {question} 
      Context: {context} 
      Answer: [/INST]
      """
    )

The key aspect of init is the creation of a PromptTemplate, which seeds the LLM with specific context (i.e the LLM should act as an expert programmer). The context provided connects the interaction between the LLM and a user. As Harpreet Sahota eloquently states, LangChain's PromptTemplate class enables

the refined language through which we converse with sophisticated AI systems, ensuring precision, clarity, and adaptability in every interaction.

The magic of RAG happens in the ingest method where a number of important operations occur. Firstly, a file system path is provided and all Ruby files are loaded into memory via the GenericLoader class. Those Ruby files are then spit into chunks for efficient indexing into a vector store. The logic here is obviously specific to Ruby and you can easily do the same for Python, Go, Java, or a handful of other languages.

def ingest(self, path: str):
        
    loader = GenericLoader.from_filesystem(
        path, glob="**/[!.]*", suffixes=[".rb"],
        parser=LanguageParser(),
    )

    text_splitter = RecursiveCharacterTextSplitter.from_language(
        language=Language.RUBY, chunk_size=1024, chunk_overlap=100)

    chunks = text_splitter.split_documents(loader.load())
    chunks = filter_complex_metadata(chunks)

    vector_store = Chroma.from_documents(
      documents=chunks, embedding=FastEmbedEmbeddings())
    
    retriever = vector_store.as_retriever(
        search_type="similarity_score_threshold",
        search_kwargs={"k": 3,"score_threshold": 0.5}
    )

    self.chain = ({"context": retriever, 
                   "question": RunnablePassthrough()}
                  | self.prompt
                  | self.model
                  | StrOutputParser())

The vector store in this case is Chroma, which by default stores vectors (i.e. documents) in memory. A LangChain Retriever instance is created that returns results from Chroma given a specific query. In this case, a similarity score threshold retrieval is used, which returns a maximum of three documents above a threshold of 0.5. Finally, a LangChain chain is created that links everything together. This final step is where RAG's power is leveraged as an existing model, like Code Llama, is combined with specific context, such as Ruby code.

With RAG set up, asking a question of a code base couldn't be any easier! The ask method simply invokes the aforementioned chain like so:

def ask(self, query: str):
    if not self.chain:
        return "Please, add a directory first."

    return self.chain.invoke(query)

As you can see, there's not a lot of complex code. It's quite simple! Specific context is loaded into memory and combined with an LLM for a more precise, context-aware answer. In this case, asked questions will be answered by what's found in the underlying indexed code.

Show me the money, baby!

As I have a few different Rails application on hand, I fired up the Python interpreter on my local machine and ingested a simple dictionary application named Locution.

>>> from chat_code import ChatCode
>>> code = ChatCode()
>>> code.ingest("/Users/ajglover/Development/projects/github/locution")

The ingest method might take a few moments depending on how many files are loaded. Once ingest returns, you can then invoke the ask method. I decided to start simple and give CodeChat a softball question:

>>> code.ask("what language was this code base written in?")

While the question is an easy one, it also demonstrates that CodeChat is focused on my Locution code base!

'The code base was written in Ruby.'

With CodeChat providing a specific answer, I'm confident that my RAG setup with Code Llama is working correctly. Next, I'll ask a more relevant question for what would be a typical use case for a question-answer style application pointed at a code base. In my case, I want to know about the underlying database.

>>> code.ask("what database tables does this application work with?")

CodeChat dutifully tells me that two tables are defined (did I mention this is a simple Rails application?).

'The application works with two database tables, "definitions" and "words". The "definitions" table has columns for the definition and example sentence, as well as a foreign key referencing the "words" table. The "words" table has columns for the word and part of speech, as well as timestamps for creation and update.'

The generated response is quite helpful as it describes the details of the two tables words and definitions. But I'm not quite satisfied as I'd like to see the DDL. It's no problem though – all I have to do is ask!

>>> code.ask("can you show me the DDL for the two tables defined?")

CodeChat gives me what appears as a cogent answer:

Sure, here are the DDL statements for the two tables defined in the provided context:

1. `definitions` table:
    
    CREATE TABLE definitions (
    definition VARCHAR(255),
    example_sentence VARCHAR(255),
    word_id BIGINT NOT NULL,
    created_at DATETIME NOT NULL,
    updated_at DATETIME NOT NULL,
    PRIMARY KEY (word_id));
    
2. `words` table:
    
    CREATE TABLE words (
    word VARCHAR(255),
    part_of_speech VARCHAR(255),
    created_at DATETIME NOT NULL,
    updated_at DATETIME NOT NULL,
    PRIMARY KEY (word));
    
Note that the `definitions` table has a foreign key relationship with the `words` table on the `word_id` column.

I was immediately suspicious of a few odd aspects of the answer, however. While there are indeed two tables, word_id isn't the primary key of the definitions table nor is word the primary key of the words table. Skepticism towards what an LLM will confidently tell you is a key skill! The good news is that with code, you can usually spot any issues quickly.

Perhaps asking low level database questions isn't necessarily the right level of abstraction required at this point. Afterall, it's a Rails application, so let me see if I can figure out how to use application code to find a specific word:

>>> code.ask("is there an API call to find a specific word in the application?")

My RAG-ified Code Llama application produced a detailed answer to the above question, containing some great examples and explanations of the corresponding code.

There is no specific API call to find a specific word in 
the application, but there are several ways to retrieve 
words based on their attributes or associations. 

For example, you can use the `Word.find_by` method to find a 
word by its ID, or use the `Word.where` method to find words 
that match certain criteria.
 
You can also use the `include` method to eager load associations, 
such as definitions, for a specific word. For example:

@word = Word.find_by(id: 1).includes(:definitions)

This will retrieve the word with an ID of 1 and include its 
associated definitions in the response.
 
If you need to find a specific word based on multiple 
attributes, such as the word itself and the part of speech, 
you can use the `where` method to create a more complex query. 
 
For example:

@word = Word.where(word: 'hello', part_of_speech: 'noun').first

This will retrieve the first word that matches the word "hello" and 
the part of speech "noun".
 
You can also use the `order` method to sort the results based 
on a specific attribute, such as the ID. 

For example:

@words = Word.order(:id).page params[:page]
 
This will retrieve the words in ascending order by their IDs 
and paginate the results based on the value of 
the `params[:page]` parameter.
 
Overall, there are several ways to find specific words in an 
application based on various attributes or associations. The best 
approach will depend on your specific use case and requirements.

This type of detailed response with code examples tailored to my specific code base is incredibly helpful. It's akin to reading a tutorial specific to an application. I've easily spent a few hours asking ChatCode various questions about my simple Rails application and I'd say that 80% of the generated answers are helpful. I suspect that I can extract better answers by providing better questions!

I found that asking questions like "how do I create a controller method that removes a word" and "how do I add a new attribute to the word model" yielded fairly competent responses with detailed elaborate steps. In this way, I can see how an AI coding assistant could be helpful when becoming acclimated with a new code base. Rather than spending a lot of context switching time between my IDE and Googling for answers, this workflow is intuitively more productive.

The legacy of code as a use case

Throughout much of my career, I've had to work on ancient code. This sort of code is often labeled as legacy code. It's frequently mystifying and definitely frustrating when getting started. The above use case of leveraging a RAG that combines a specific code base with an LLM, like Code Llama, could certainly make the experience of familiarizing yourself with legacy code way more easy. What's more, what I've demonstrated here is a command line interface. This experience is less optimal than a snazzy UI or even an integrated IDE experience (like Github's Copilot). Both UI and IDE integration options are entirely possible nevertheless!

AI enhanced productivity is here

Throughout my career, I've often heard complaints about new technologies. Java was too slow in 1997. Mobile devices weren't powerful enough to do anything interesting in the early 2000s. And AI isn't too sophisticated – it hallucinates, frequently making mistakes (as I've shown above). But the history of technology has clearly shown that useful technologies improve. Today's LLMs are the worst they'll be. LLMs can only get better as we learn how to both leverage them and importantly, improve them.

The skeptics of today are right to point out deficiencies! But remember that the deficiencies of today will be solved tomorrow provided the underlying technology is promising. Don't overlook AI because of its shortcomings. It's here to stay and will only become more pervasive and pragmatic!

Want to learn more?

I created a Github repository, aptly named windbag, which has the above code. Moreover, Duy Huynh's article on Hackernoon was extremely helpful in demonstrating the various aspects of RAG. It certainly helped me get started. Furthermore, while I used Code Llama, you can just as easily use OpenAI's ChatGPT LLM using ChainLang's API. It would be interesting to compare and contrast the two!

Can you dig it?

Because it’s my bag, I’ve been diving deep into the incredibly exciting and rapidly evolving world of AI. The innovations in the open source arena are astonishing; indeed, you can now run a diverse set of LLMs from the convenience of your laptop. If you are willing to roll up your sleeves, there are plenty of OSS models and frameworks that cost you only the time it takes to set things up. 

One popular tool for running open source LLMs locally is Ollama. I started using it recently and I’ve been blown away with how useful, productive, and enlightening common workflows can become if you take the time to explore, understand, and ultimately leverage AI. With Ollama, you can run various LLMs such as Meta's Llama2, Mistral, LLaVA, and my new favorite, Meta's Code Llama.

Meta's Code Llama

Code Llama is an LLM that can use text prompts to generate and discuss code. With Code Llama, you can generate Python, Java, Typescript, and more. As Meta's documentation for Code Llama states:

It has the potential to make workflows faster and more efficient for developers and lower the barrier to entry for people who are learning to code. 

Indeed, the evolution of AI enhanced productivity will come in the form of boosted workflows. These workflows will have fewer speed bumps resulting from context switching and accordingly increase productive capacity.

I saw AI's boosting of productivity potential firsthand recently. It’s been quite a long time since I’ve coded in Python (circa 2005!) and in an effort to refresh my memory on the finer aspects of Python, I decided to play some ping-pong. But in this case, I decided that I'd use Code Llama as my coding partner.

After installing Ollama, you need to download the Code Llama LLM like so:

$ ollama run codellama

After downloading the LLM, which could take a bit, you’ll be presented with an in-terminal chat-like interface. You prompt Code Llama and it'll magically respond with (mostly) cogent answers.

Ping-pong pair programming

In ping-pong, your partner writes a failing test case. Your job is to write the least amount of code that makes that failing test case pass. Once you’ve done that, you then write a failing test case and your partner in turn writes the least amount of code to make your test case pass. You go back and forth and in the process of playing ping-pong, you'll undoubtedly learn new techniques, styles, and ways of approaching problem solving. Over the years, I've learned valuable lessons when pairing with folks! Ping pong has certainly made me a better programmer. I've even employed ping-pong in the context of interviewing.

I figured I’d get right into AI pair programming by building a simple stack implementation in Python. Stacks commonly have push and pop methods which follow a last in, first out (or LIFO) algorithm. push adds items onto the stack and pop removes the last item added. I was also aware that out-of-the-box, Python comes with a unit testing framework dubbed unittest (akin to Java's JUnit).

Kicking things off, I began with the following prompt:

>>>> write me a test using python unittest for a pop method of a stack

As requested, Code Llama promptly returned some code using Python’s unittest framework:

import unittest
from collections import deque

class TestStack(unittest.TestCase):
    def setUp(self) -> None:
        self.stack = deque()

    def test_pop(self) -> None:
        self.assertEqual(self.stack.pop(), None)
        for i in range(5):
            self.stack.append(i)
        self.assertEqual(self.stack.pop(), 4)
        self.assertEqual(self.stack.pop(), 3)
        self.assertEqual(self.stack.pop(), 2)
        self.assertEqual(self.stack.pop(), 1)
        with self.assertRaises(IndexError):
            self.stack.pop()

Code Llama additionally described the usage of unittest’s setup method as well as the usage of assertEqual and assertRaises. 

Of course, I was provided with way more implementation code than what I really wanted. Nevertheless, the example provided was enough to get me started as the example code quickly got me up to speed on unittest and also informed me that Python has an implementation of a stack data structure via the deque object. 

I copied the code into a file dubbed test_stack.py and then ran it. Unfortunately, nothing happened. Then I remembered from my Python days of yore that I needed the equivalent of amain method. But I had forgotten the syntax. No problem, I just asked Code Llama and it told me to insert this:

if __name__ == '__main__':
   unittest.main()

I added the above code and reran the test. Unsurprisingly, the test errors out as the first assert of self.stack.pop() raises an IndexError because the underlying list is initially empty. You can see that the test case, in fact, verifies this case correctly in the last part of the test_pop method. 

Testing, testing, 1, 2, 3

As it turns out, AI (inadvertently) wrote me a failing test! Now all I needed to do was make the test pass with as little code as possible. I figured that I would trim down the test case and simply test for an empty pop yielding None. 

import unittest
from collections import deque

class TestStack(unittest.TestCase):
    def setUp(self) -> None:
        self.stack = deque()

    def test_pop(self) -> None:
        self.assertEqual(self.stack.pop(), None)

if __name__ == '__main__':
    unittest.main()

Next, I wanted to write my own stack implementation and remove deque. Looking at the TestStack class, I remembered some important aspects of coding in Python: methods are defined with def, whitespace is a delimiter, and don't forget about colons. But I couldn't remember exactly how imports work. No problem! All I had to do was ask Code Llama:

>>> how do I import a python class in my local directory?

I was informed via a nice description that I could create a file, using snake case, and inside of the file define a class. Then in another file, I can import that class using the from file import class syntax. Excellent! Next, I needed to define a minimal stack – ideally, with only a pop method. Might as well ask my groovy fried Code Llama how to define a custom stack in Python:

>>> how do I implement a custom stack class in python?

Code Llama dutifully spat out the following code:

class CustomStack:
    def __init__(self):
        self.stack = []

    def push(self, item):
        self.stack.append(item)

    def pop(self):
        if len(self.stack) == 0:
            raise IndexError("Stack is empty")
        return self.stack.pop()

    def peek(self):
        if len(self.stack) == 0:
            raise IndexError("Stack is empty")
        return self.stack[-1]

    def size(self):
        return len(self.stack)

Not only did I get what appears to be working pop and push methods, but I also got a peek and size method to boot! This is obviously more code than I needed; accordingly, I simplified the code with the intent of making the aforementioned test pass:

class Stack:
    def __init__(self):
        self.stack = []

    def pop(self):
        None

When I rerun the test case, things pass! Of course, I've merely written the least amount of code necessary to make the test pass. It's now my friend Code Llama's turn to write a failing test case. This wasn't too hard of a prompt to fashion:

>>> write me a test that fails for the pop method in this class: class Stack:
...     def pop(self):
...         return None

Impressively, Code Llama did exactly what I wanted. It generated a simple test case that will force a failure:

import unittest
from my_stack import Stack

class TestStack(unittest.TestCase):
    def test_pop(self):
        stack = Stack()
        self.assertEqual(stack.pop(), 1)

I added another test case to my existing TestStack, ran it, and sure enough, I've got a failing test. It's time to make it pass. My brutally simple pop method needs to do some real logic as opposed to returning None.

Can't pop without a push

Satisfying my friend Code Llama's new test case means I need to flesh out a push method so I can test popping a value.

class Stack:
    def __init__(self):
        self.stack = []

    def pop(self):
        if len(self.stack) > 0:
            return self.stack[-1]
        else:
            return None

    def push(self, value):
        self.stack.append(value)

My Stack now has a push method that adds an item to an internal list (self.stack) and the pop method simply returns the top item or None if the list is empty.

I've refactored my test case slightly and have two passing test cases now.

import unittest
from stack import Stack

class TestStack(unittest.TestCase):
    def setUp(self):
        self.stack = Stack()

    def test_pop(self):
        self.assertEqual(self.stack.pop(), None)

    def test_pop_again(self):
        self.stack.push(1)
        self.assertEqual(self.stack.pop(), 1)

if __name__ == '__main__':
    unittest.main()

After a few back and forth interactions with Code Llama and my Python confidence is increased. I can see how this technique would work equally well with a language I wasn't familiar with too.

What a match!

The game of ping-pong need not go on forever. In fact, a few rounds is often enough to get into the flow of things and learn a little from your partner. In my case, I had a lot of fun getting to know how Code Llama approaches problem solving and along the way, it helped re-acquainted me with Python.

I dig ping-pong pairing with an AI and will continue to do it going forward. As with all things AI related, prompting is a crucial skill as well as having a keen eye for code correctness. LLMs will confidently give you an incorrect answer (i.e. code that doesn't work).

In my case, I was using a terminal; nevertheless, there are other in-IDE options ranging from GitHub's Copilot to other innovative tools. You can also take advantage of Ollama and Code Llama via CodeGPT and have yourself some sweet VSCode integration.

Can you dig it, man?

I’ve never been a fan of DORA metrics. I appreciate the copasetic thought leadership behind them and their intended purpose of tracking a company's ability to reliably move fast. But it’s always struck me as odd that one of the metrics is Change Failure Rate. This is an unnecessary metric that works against the other metrics.

If you track Change Failure Rate, it really means that you’re afraid of change. Moreover, if you’re afraid of change, you can’t move fast. If you can't move fast, your business will lose to one which can.

What are DORA metrics?

Briefly, DORA metrics are four key performance indicators (KPIs) that attempt to measure the effectiveness of a software delivery process. Those KPIs are:

1. Deployment Frequency: how often an organization deploys code into production.
2. Lead Time for Changes: the average time it takes for a code change to go from commit into production.
3. Change Failure Rate: the percentage of deployments that fail.
4. Mean Time to Restore: the average time it takes to recover from a failed deployment. This is often abbreviated as MTTR.

The folks behind DORA metrics are respected luminaries and their recommendations come from an incredibly large set of studies and surveys from a diverse audience of global companies. Sadly, while their recommendations are quite popular, it's my experience that only one of the four metrics really matters. What's more, Change Failure Rate creates a tension with Deployment Frequency and results in a perverse incentive that ultimately can reduce Lead Time for Changes. 

Competitive advantages

Speed is the primary competitive advantage a company has in any market. Therefore, the only metric that matters to all parties in a business is Lead Time for Changes (commonly shortened to Lead Time). This metric is broadly the time it takes for a business initiative to reach customers.

Lead Time for Changes can be reduced by making it fast and safe to deploy (Deployment Frequency) along with having the safeguards in place to rapidly detect and fix any issues that arise (MTTR). When Deployment Frequency is high and MTTR is low, the business is happy.

MTTR, however, is easier said than done. Indeed, if you want to move fast with some semblance of reliability, you’ll need to invest in three disciplines: observability, failure domains, and fast fixes via rollbacks or via rolling forward. If you can detect issues quickly (often called Mean Time to Detect), have thought through how to limit the blast radius of errors (like avoiding global changes), and can rapidly fix those issues by either reverting the change or deploying a fix, you’ll have agency with MTTR. If you want to be groovy and move fast, you need confidence that you can rapidly resolve issues. Consequently, you have to know when an issue arises, have the ability to limit its scope, and quickly rectify it.

Fear based metrics

If you're tracking Change Failure Rate, you’re afraid. Tracking this metric means your MTTD and MTTR are high. It means you have poor observability. It probably also means you lack automation and the necessary machinery to fix things without a lot of fire fighting. It means your business panics when things break because there's a lack of confidence engineering can fix the issue quickly. It means the business doesn't trust engineering. 

Here’s the thing: complex systems break. They break all the time too. All high speed companies with massive scale (like Google, Netflix, Meta) have figured out a way to find and fix those issues quickly. Indeed, failure is inevitable in complex distributed systems; therefore, you can’t reduce all failures. It’s far more efficient to accept failures as an outcome of moving fast and put resources towards figuring out how to deal with them quickly. Tracking Change Failure Rate is a distraction.

When the measure becomes a target

Tracking Change Failure Rate is a perverse incentive. What's more, the word “failure” is massively negative. There are few outcomes in the context of software processes that end well when the word failure is used. No one wants to have their name associated with a metric that uses the word failure. 

If you judge people for failing, it’s a certainty that they’ll game the system to avoid the stigma associated with the metric. In this case, they’ll hide failures or slow deployment velocity so as to limit the growth of this nefarious metric. This is Goodhart's law in action, baby. The metric ceases to be a good measure because as soon as folks are afraid to report failures, you don’t learn. And you don't move fast.

Failures happen. Great engineering cultures celebrate and learn from failures while awful cultures punish for failures. Focus on what matters: moving fast with reliability.

Measure what really matters

DORA metrics represent a framework for optimizing a company's software delivery process and boosting its overall performance. Every business wants to move fast as it's the only way to win. Accordingly, the only metric that matters is Lead Time.

If you want to understand this metric, ask the business if it feels Engineering is moving fast enough. It's every business leader's bag to want to move faster, so listen carefully to the answer. If the answer is a frustrating "no" then you have a lot of work to do. You'll need to improve things in short order if want to survive. If the answer is a "yes" then you might want to also worry. Complacency leads to stagnation. Anything in between is the sweet spot for increasing overall engineering transparency and working to ensure that your company can reliably move fast.

Measure what matters by thinking through incentives. Moreover, acknowledge and accept that failures will occur. Invest in detecting failures and rapidly rectifying them. What's more, invest in learning from failures. Get behind the rallying cry of “move fast with reliability” and stop wasting time measuring failure.

Can you dig it?

I never really thought about values for much of my career. I probably assumed that everyone more or less shared my values. It wasn’t until a few years ago that I started to reflect on my values and in doing so, I’ve learned some important lessons. First, most people share the same values, but we often disagree on how values manifest themselves. We also tend to disagree on their relative priority. Second, it’s crucial to recognize that a disagreement arising from conflicting values is practically impossible to rectify. 

What do you mean by values? 

Values are your guiding principles. They are the core beliefs that shape your decisions and actions. Values represent what you consider important and desirable in life. Indeed, values give your life meaning and purpose. 

When faced with choices, big or small, values serve as internal compasses. They help you navigate complex situations by suggesting paths that align with what you believe is right, copasetic, or important. Aligning your decisions with your values often leads to a greater sense of satisfaction, fulfillment, and general grooviness. 

As a leader, it’s important to take stock of your values from time-to-time as they influence the decisions you make and the consequent actions you take. If someone has ever asked you the question “what kind of leader are you?” your answer is likely in terms of values. Your values shape your identity and understanding them helps you better understand you.

Words aren’t the same as actions

If you spend time reflecting on the values that you value, you’ll likely come up with a series of nouns such as accountability, love, loyalty, or empathy, for instance. These words are meaningful, but it’s important to recognize they aren’t unique to you. Most people value love.

What’s more, the context of a particular value can be nuanced: loyalty to family could be different than loyalty to a company. Consequently, when you think through your values, it’s helpful to articulate how those values manifest themselves through your actions or how you judge those values in other people’s actions. Moreover, it’s helpful to prioritize your values, especially if some values are in tension with each other. Empathy and accountability are a few of my values. Those two can be at tension with each other; accordingly, I place a higher priority on accountability. 

If you haven't done so before, take some time and list your values. Describe what they mean to you. Order your values by importance. Periodically revisit your values too. My values have certainly evolved over time, especially their relative order. 

A values clash

Most companies espouse a set of values as part of their corporate culture. Ideally, you’ve aligned with your particular company’s values; indeed, it’s probably a driving force to work for the company. When you and your company share values, things are copasetic. Nevertheless, there are occasions where values clash. Specifically, the values of your company, or more realistically, your boss, occasionally will fall into conflict with your values. This is why it’s important to know your values and your expectations of how those values manifest themselves. 

If you find yourself unhappy or frustrated with corporate decisions, direction, or leadership, take an accounting of the values you see reflected. For instance, if you value transparency, but struggle with the lack of it from your company, this is likely a cause of tension and unhappiness. Unfortunately, as I’ve learned over time, this tension is rarely solvable. That is, when your values clash with the corporation, the corporation usually wins. 

Values define you. If you value curiosity and find yourself working for a manager who's a know-it-all, chances are you won’t be happy. If you value fairness but feel the decisions corporate leadership makes are explicitly unfair, you’re going to feel conflicted. You can certainly try to change the values of those around you, but recognize that it's an uphill battle. Afterall, their values define them.

Values represent what you consider important and desirable in life. If you want to find meaning and purpose, then know your values and be true to them.  

As I’ve written about previously, Postgres is an incredibly rock solid database. What’s more, when the good vibes go your way and you need to scale Postgres, its august ecosystem answers the call quite nicely. In particular, PgBouncer enables you to both vertically scale Postgres and more efficiently take advantage of Postgres horizontal scaling. 

PgBouncer is primarily a connection pooler and router for Postgres. It acts as middleware between applications and one or more Postgres instances, managing both routes to individual databases as well as a pool of connections. Leveraging PgBouncer can significantly improve the performance of an application by reducing the overhead of opening and closing new connections to the database as well as enabling you to easily route traffic to multiple Postgres instances (such as read replicas, for instance). 

With PgBouncer managing database connections, you free up valuable resources for your database instances thereby enabling you to more efficiently vertically scale. Moreover, PgBouncer's routing features make it copacetically easily to route queries to multiple instances. In this way, PgBouncer is conceptually akin to a load balancer enabling efficient horizontal scaling.

Getting started with PgBouncer

Getting started with PgBouncer couldn't be any easier; in fact, I've set up a Github repository, dubbed Recoil, that'll enable you to familiarize yourself with both PgBouncer's connection pooling and routing features from the comfort of your laptop. All you need is Docker Engine and you'll be good to go, baby!

The connection pooling boogie

Connections in Postgres are expensive because Postgres creates a new backend process for each connection, which involves memory allocation and initialization tasks. A connection pooler manages a set of connections thereby freeing up a database's resources to focus on other aspects, such as rapidly returning data.

To see how PgBouncer works as a connection pool, you can use use pgbench, which is a simple command line utility that runs benchmark tests on a Postgres instance. It's an efficient way to exhaust database resources, which is easy do if you set a high value for pgbench's client connections (i.e. via the -c flag) . You can, of course, increase Postgres's max_connections; however, you'll potentially do this at the cost of other important resources.

Using the handy docker-compose.yml file found in the Recoil repository and following the detailed directions in the README, you can fire up a single instance of Postgres and do two things: initialize the database for benchmarking and then running a benchmark test.

Initializing the database for benchmarking is easy. Run the following command in your favorite terminal:

$ pgbench -i -s 5 -h localhost -p 49149 -U postgres recoil

The -i flag initializes a benchmark with a scale factor of 5 via the -s flag. This will create 500,000 rows in a ppgbench_accounts table.

By default, Postgres's max_connections is set at 100. You can quickly exhaust this resource by running a benchmark test that exceeds this limit like so:

$ pgbench -c 101 -j 2 -t 100000 -h localhost -p 49149 -U postgres -S recoil

In this case, 101 clients, configured via the -c flag were thrown at the local Postgres via 2 threads (i.e. -j). Each client attempted to issue 100,000 transactions via the -t flag. Unfortunately, this test fails as Postgres quickly exhausts valuable connections. The database will error out with a message along the lines of FATAL: sorry, too many clients already.

This is a great use case for employing a connection pool like PgBouncer! Following the groovy directions I outlined in Recoil's README you can stand up PgBouncer in front of Postgres and let it manage database connections; what's more, if you set PgBouncer's connection pool to a large number, such as 200, you can get through the above pgbench test without failure.

Accordingly, you can reissue the pgbench benchmark test pointing to a different port, since you're going directly to PgBouncer (instead of Postgres) like so:

$ pgbench -c 101 -j 2 -t 100000 -h localhost -p 5432 -U postgres -S recoil

Connection pooling isn't a panacea; indeed, you can play around with pgbench and quickly exhaust PgBouncer as well. What's important is that you've essentially offloaded connection management to a specialized process which is more efficient. Moreover, PgBouncer adds a few other handy features for your scaling needs including routing.

Routin' SQL requests

Horizontally scaling Postgres is fairly simple, especially if you read my blog post on read replicas. Of course, when you stand up more than one database, you'll want an easy way to route SQL requests. A simple load balancer would do this automatically for you; however, with PgBouncer, you can easily route via a name. In the case of read replicas, you'll want to route writes to the primary instance and reads to one or more replicas. Conceptually, the architecture of using PgBouncer as a combination pooler and router looks like so:

With PgBouncer, routes are configured like connection strings, where the database name signifies a specific database. For instance, in my Recoil example, the database is named recoil. In a normal read replica environment, read replicas would have the same name; nevertheless, with PgBouncer, you can give each read replica a unique identifier, which clients can leverage for specific routing.

You can see routing in action following the directions in the Recoil Github repository. If you examine the docker-compose.yml file, you'll note that the PgBouncer instance, pgbouncer, defines two databases. Defining the first database is done like so:

- POSTGRESQL_HOST=postgres-primary
- POSTGRESQL_PASSWORD=recoil
- POSTGRESQL_DATABASE=recoil
- PGBOUNCER_DATABASE=recoil

Subsequent databases, however, are defined using a somewhat esoteric string like so:

- PGBOUNCER_DSN_0=recoil-ro=host=postgres-replica port=5432 dbname=recoil

The crucial part of this string is the DSN_0 aspect, which gives this route the name recoil-ro. This logical name points to the postgres-replica service listening on port 5432.

You can see routing (and replication too!) in action by using psql to log into the primary and issuing two SQL queries. First, shell into the primary using the following command:

  $ psql postgresql://postgres:recoil@localhost:5432/recoil

If you're following the directions in Recoil's repository and ran bunzip2 on the employee_data.sql.bz2 file and then started things up with docker compose up -d, then when you run this SQL command, you should see a count of 300024.

select count(*) from employees.employee;

Next, insert an additional employee record like so:

insert into employees.employee values (9999999999, '1987-03-21', 'Joe', 'Smith', 'M', '2022-01-22');

With Postgres replication, behind the scenes this record is automatically added to the configured read replica (i.e. the postgres-replica service found in the Docker compose file). You can verify this new row was added by using psql to shell into the replica instance by referencing the route name elaborated earlier, which is recoil-ro.

$ psql postgresql://postgres:recoil@localhost:5432/recoil-ro

Run the aforementioned count query and you should see the answer is 300025, which reflects the newly added employee record inserted into the primary instance and replicated to the replica.

As you can see from the code above, PgBouncer is acting like a proxy, routing queries to appropriate database instances via name. In fact, conceptually, PgBouncer is akin to a load balancer. It does represent a single point of failure, but if you zoom out slightly, you'll recall that PgBouncer also is a connection pool.

Keep on truckin' with PgBouncer

If you're using Postgres and are looking for a way to improve the performance and stability of your application, then PgBouncer is a groovy option. It's easy to set up and use, and it can provide a significant improvement in performance. Can you dig it?

As a leader it’s tempting to think your ideas and decisions are fantastic. Most likely, your track record of previous ideas and decisions got you to where you are. Moreover, it’s human nature to avoid the person who tends to disagree with you. After all, you’re the leader. You’re the one who’s accountable. 

When I was a new leader, I disliked it when individuals would disrupt the harmony of meetings by disagreeing with my ideas, decisions, or game plans. My instincts were to avoid including them or ignoring their opinion. Oh how I wish I could reach back in time and shake some sense into that version of me! My instincts were awfully incorrect. Dissenting opinions are a veritable gold mine of alternative paths, variables, and, importantly, clarity. Mining for dissent, rather than seeking a lazy harmony, always makes decisions even better. 

Make it safe

Naturally, you have to make it safe for disagreement. You have to put your guard down and let your ego go. It’s hard at first. But if you want folks to critically think through and challenge ideas, you need to make it easy. Be welcoming of different opinions; what's more, when folks do speak up and dissent, celebrate it. It'll become a virtuous cycle as people will see it's safe and try it for themselves.

When someone disagrees with you, it’s important to take the position that they are disagreeing with your idea, not you. Disagreement isn’t a personal attack of your intelligence, but simply a different point of view. Your job as a leader is to understand why there’s a disagreement. Understanding how someone arrived at a different position is vital. This process takes practice and a lot of patience. Nevertheless, in these disagreeing situations, encourage dissenters to elaborate their reasons. Make sure you understand their reasons. Did this person uncover a blind spot in your analysis? Did you overlook something? Don't dismiss this dissent no matter how tempting it is!

Avoid the laziness trap

Mining for dissent is an effective means to avoid groupthink. Sometimes people get lazy – it’s been a long day and they aren’t in the mood to think critically or engage in some sort of disagreement. This definitely happens to me! Zoom fatigue is real. Don’t take the easy path though! You have to encourage debate and disagreement. You have to pull it out of people sometimes. Mining for dissent is as easy as asking "what am I missing?"

It no doubt feels good to be in a room of yes-people, but it’s a trap! Those harmoniously pleasant interactions without any deep thoughtful debate are weak. Chances are there's a hole somewhere. Be brave and seek if people disagree with you and then find out why. If you have a high powered team that has a foundation of trust, you can put away your ego!

Great leaders embrace dissent

Allow folks to poke holes in your ideas and through that process, you’ll end up with a better idea. I guarantee it. An idea which has gone through dissent mining will be one that’s been vetted and critically analyzed. And in the process, you’ve built buy-in and ownership from your team. By actively mining for dissent, your initially weak idea, decision, or position becomes your team’s well throughout strategy or plan. Moreover, when surprises inevitably pop up, you’ll all be in the trenches together solving whatever problems arose. By actively engaging a debate, you've built commitment.

It took me some hard lessons to realize that when people disagreed with me, they cared. If someone disagrees with your idea, it means they took the time to think about your idea! Embrace dissent. Mine for it. It'll result in better decisions, buy-in from your team, and make you a better leader.

Postgres is an incredibly mature and battle hardened database that powers many of the applications you rely on daily. From Instacart, Instagram, various Apple services, and Spotify, chances are that data related to you resides in a handful of well scaled and highly tuned Postgres instances across the globe. 

Out-of-the-box, Postgres works well enough. If you’re lucky, and you find yourself with a popular application (which generates a lot of load through reads and/or writes), you’ll likely find yourself needing to scale Postgres. Fortunately, you have a few options, starting with throwing more hardware at the problem by adding faster disks, additional CPUs, and more memory. You can think of these options as vertically scaling Postgres. Vertically scaling Postgres can increase throughput and reduce latencies; however, these options don’t address resilience.  

Addressing resilience ultimately requires making copies of the underlying data, which can be achieved through shared storage, read replicas, or via distributed versions of Postgres like AWS’s Aurora, Yugabyte, and Google’s AlloyDB (just to name a few!). These tactics enable you to horizontally scale Postgres and of these options, one of the easiest to spin up are read replicas.

There are certainly trade offs when using read replicas. First and foremost, read replicas are asynchronous copies of your primary Postgres database. Changes made on the primary are streamed (via the WAL or write ahead log) to the replicas with a slight delay. What’s more, read replicas are read-only, meaning you can't write data directly to them. You can only read from them, which reduces resource demands and avoids data conflicts. It’s also easy enough to have multiple read replicas spread across different servers or regions, distributing read traffic and improving responsiveness for user queries. 

In essence, read replicas offer a backup of your database and in a worst case scenario, you can turn a read replica into the primary should your primary database go down. Of course, there’s some probable downtime if this were to happen because, at a minimum, you’ll need to reroute writes to the newly christened primary. 

Show me the code, man

Setting up a read replica doesn't take a lot of effort! For instance, if you want to see how they work locally and you're using Docker, Bitnami's Postgres Docker image couldn't make it any easier. I created a Github repository, named Hoodoo, which contains a Docker Compose file demonstrating read replicas in action.

If you clone the repository and run docker compose up -d, you'll find that two Postgres instances have started: one primary and a replica. What's more, you can decompress a provided SQL script and import this data into the primary, enabling you to work with an employees database. You'll notice after you import the employees database into the primary, the replica will reflect the same (i.e. replication in action, baby!).

As I mentioned, Bitnami has packaged Postgres making creating a replica effortless. The key to configuring the primary and one or more replicas is via Docker Compose individual services environment settings. For instance, the primary is configured with the following environment variables:

 environment:
    - POSTGRES_PASSWORD=hoodoo
    - POSTGRESQL_REPLICATION_MODE=master
    - POSTGRESQL_REPLICATION_USER=repl_user
    - POSTGRESQL_REPLICATION_PASSWORD=repl_user
    - POSTGRESQL_DATABASE=hoodoo

As you can see above, the primary is configured as master and a special username and password are configured. The replica's environment is configured as follows:

environment:
   - POSTGRES_PASSWORD=hoodoo
   - POSTGRESQL_MASTER_HOST=postgres-primary
   - POSTGRESQL_MASTER_PORT_NUMBER=5432
   - POSTGRESQL_REPLICATION_MODE=slave
   - POSTGRESQL_REPLICATION_USER=repl_user
   - POSTGRESQL_REPLICATION_PASSWORD=repl_user

Note how the replica is configured as a slave; moreover, this service is configured to point at the primary via a host and port number.

You can connect to either database via the psql command - the primary is exposed on port 5432 and the replica on 5434. To get a true feel for how things work in a read replica environment, you can issue a write to the primary and then in another terminal window, issue a read on a replica (i.e. a select) and the data will be there.

For instance, first verify the count of employee records in the database via a select:

select count(*) from employees.employee;

The answer should be 300,024. Verify this on both the primary and the replica. Next, insert a new employee into the primary via the below SQL statement:

insert into employees.employee values (99999999999, '1957-03-20', 'Robert', 'Smith', 'M', '2024-01-02');

Finally, you can run the same select count(*) command on the replica and you'll see the count has increased by 1. Magic! My hope is that the Hoodoo repository makes it super easy to see Postgres read replicas in action and enable you to get a feel for how to employ this strategy for horizontally scaling Postgres.

Horizontal scaling tradeoffs

For a resilient backup of your database via read replicas, you do make a few tradeoffs. There's a slight delay in replication between a primary and a replica. This delay is a function of the speed of light and processing power. You can use read replicas to offload traffic from a primary, but it’s only reads. Depending on your load, you’ll also need to pay special attention to the primary’s WAL settings as if they aren’t tuned correctly, your primary’s performance could suffer. Finally, spinning up one or more read replicas will naturally cost you more money, most likely above and beyond what you’ve already spent on vertically scaling your database (although you’ll most likely reap some savings by lowering the primary’s load). Read replica instances are an additional asset that’ll need monitoring and ongoing maintenance (i.e. upgrades, etc) too. 

If your Postgres database is reaching the limits of vertical scaling and you’re looking to increase reliability, a read replica strategy can be beneficial. Read replicas can increase read performance because you can offload read-heavy workloads from the primary server. Moreover, read replicas improve scalability: you can add more of them to handle increasing read traffic without impacting the primary server. You can also geo-distribute read replicas to reduce latencies. What’s more, read replicas enhance availability. If needed, you can promote a replica to become the primary if the primary fails.

Can you dig it?

Before joining Netflix 10 years ago, I had held various leadership positions. In all cases, I was very much a command-and-control style leader. I made what I thought were cogent decisions and expected my teams to unquestionably execute on them. When things didn’t go as I’d planned, I naturally became frustrated and blamed the people on my teams. It never occurred to me that there was a better way to lead. 

Then I joined Netflix as a leader and I learned about the notion of context over control. Instead of trying to control people, leaders were encouraged to provide teams with context. I learned that great leaders provide the insight and understanding that enables sound decisions through context.

Inspire through context

Context is about articulating goals and the corresponding strategy to achieve them. Context is generically the what and not the how. The folks closest to the problem oftentimes have a much better understanding of the challenges and opportunities in front of them and can determine a much better solution (i.e. the how). Importantly, their solution aligns with the goals set forth by effective leaders who have articulated the what. 

Accordingly, encouraging context over control means that leadership must clearly articulate important concepts like business goals and the strategy to achieve those goals. Importantly, leaders must explain why those goals are important and what they mean to the business. Moreover, leaders should strive to provide transparent decision making, eschewing bureaucratic concepts like committees, management approvals, complex processes that become more valued than goals, and the notion of top-down decision making. 

Failure is on you

Once I embraced the notion of context not control, my viewpoint on failure flipped as well. When teams fail to execute according to all the context you’ve provided, rather than blaming them (like I used to do) instead, try to determine what context you failed to provide. In my case, I almost always have been able to determine my ability to convey context needed improving. That is, my teams were acting with the information they had. As I invested in transparency of decision making, I noticed my teams became more effective. They made better decisions. 

If you're hiring amazing people, they will do better work if they understand the context behind decisions, goals, and strategy. It’s easier said than done and requires a large investment from leadership! Communication is hard; leaders have to deliberately invest in it. Recognize that some context may make folks feel uncomfortable. But that’s ok if you’ve built a team of adults. Transparent context is not a panacea either: there are certainly times when control is appropriate. Emergencies and a lack of mature team members are examples of those times.   

Relinquish control

The best leaders facilitate great outcomes by setting appropriate context for their teams as opposed to trying to control them. For years, I drummed up developers giving orders to do the work I felt was important. Once I switched to inspiring developers to yearn for lofty, but achievable goals, I saw new levels of impact and success. If only I had learned this lesson earlier in my career.