NIGEL: Don’t touch it!
MARTY: I wasn’t going to touch it, I was just pointing at it.
NIGEL: Well, don’t point, even.
—“This is Spinal Tap”

We saw in Racing with disaster that when two goroutines go to war, a data race is all that you can score. That is, when your program concurrently reads and writes some piece of shared data, the results are non-deterministic, and that’s generally unacceptable in distributed systems.

Here’s the problematic program:

func main() {
    copies := 1
    go func() {
        if copies > 0 {
            copies--
            fmt.Println("Customer A got the book")
        }
    }()
    go func() {
        if copies > 0 {
            copies--
            fmt.Println("Customer B got the book")
        }
    }()
    time.Sleep(time.Second)
}

(Listing copies/1)

Goroutines A and B both read and try to decrement the shared copies variable, and the result is that we can’t predict which one will get there first, and thus which customer will get the book. In a distributed system where the copies variable is held on a remote computer, we could easily end up with a situation where both customers think they’ve got the last copy of the book.

For example, suppose that “goroutine A” is actually Amina, a Happy Fun Books customer happiness operative who’s trying to put through a purchase at her sales terminal. Concurrently, the part of “goroutine B” is played by Bhavana, who’s also trying to process purchasing the same book at her own terminal.

How can Amina and Bhavana co-ordinate to avoid a data race?

One very simple, and low-tech, idea would be to have Amina shout out “Hey, y’all, I’m about to modify book abc. Don’t do anything with abc until I say it’s okay!” And then later: “All good, I’m done with abc now”.

An audio-based protocol like this would work, sort of. It would mean a lot of shouting, though, and that’s not really ideal: in the best bookstores, a quiet, studious atmosphere should prevail.

A better scheme might be to have a big whiteboard in the middle of the store where everyone can see it. When Amina wants to make a change to book abc, she looks at the board to see if “abc” is already up there. If not, she writes it on the board. Then she makes her changes, and when she’s done, she rubs “abc” off the board so other users know it’s once again safe to modify abc.

What if Bhavana wants to make a change to abc while Amina still has it “locked” like this? Well, she has to wait. Yes, it might be slightly annoying for other users to wait until Amina releases the lock, but it’s better than the catalog data getting corrupted: that would annoy everybody.

And under normal circumstances, no one will need to lock a book for very long, so most books will spend most of the time unlocked. It’s unlikely that anyone will have to wait for the lock on a given book, and even if they do, the waiting time won’t be significant. We don’t expect much contention for these locks, in the jargon.

It’s a workable scheme, with some caveats. For example, Amina has to be careful to remember to obtain the lock—that is, to check the board to see if the book’s ID is up there, and add it herself if not. If she forgets to do this before modifying the book information, there’ll be a data race.

Also, she has to remember to release the lock afterwards: to rub the ID off the board. Otherwise, Bhavana will be stuck waiting for the lock forever, and more and more users will gradually get stuck waiting on the same lock over time, until eventually nobody can do anything at all (this is called a deadlock).

Despite these potential problems, locking protocols like this are the simplest way to prevent data races in real-life concurrent environments. They don’t usually involve a big whiteboard, but the principle is the same.

In my book The Deeper Love of Go, we’ll use Happy Fun Books as an example application to help us learn Go programming from scratch. Starting with fundamental concepts like functions, data, and variables, we’ll gradually build up a suite of Go software to store and manage book information. Along the way we’ll talk about writing command-line tools, HTTP servers and clients, designing REST APIs, storing and retrieving data as JSON, and all sorts of other exciting topics. I hope you’ll take a look!

Mutexes in Go

A lock system like the one we’ve described ensures mutually exclusive access to some data value, such as book abc. In other words, if I have access to it, that excludes you having access to it, and vice versa. In the jargon, then, the lock object that ensures this mutual exclusion is called a mutex.

Go’s standard library includes one of these, as part of the sync package: it’s called sync.Mutex. Here’s an example of how it’s commonly used:

var mutex sync.Mutex
mutex.Lock()
// do something with data
mutex.Unlock()

We needn’t worry at all about how the mutex works internally. We can treat it as a black box with two methods: Lock and Unlock. We call Lock to get the lock, we do whatever we need to do with the protected data, and then we call Unlock to release the lock when we’re finished.

So what happens if someone else already has the lock when we call Lock? The answer is that Lock will wait until the lock becomes available, and then obtain it for us. This can take an arbitrary amount of time (indeed, the lock might never be released), but if or when Lock does return, it guarantees that we now have the lock.

When someone holds the lock on the mutex, it blocks everybody else from accessing the protected data: that’s a mutex’s job, after all. So we should only acquire the lock right before we need it, and release it as soon as possible afterwards. We should also avoid doing anything too time-consuming during this period, to avoid unnecessarily blocking the lock for other users.

Both reads and writes need to be lock-aware

It sounds like a mutex will help us to ensure safe concurrent access to the catalog, but before we do that, there’s one more point we need to talk about. When do we need to acquire the mutex lock? Is it only when we want to change the data, or is it also when we’re reading it?

You might be tempted to say that it’s only when we’re changing the data. After all, if we’re just reading, we’re just reading! Why would we need to get a lock on something that we’re not changing?

But that idea doesn’t work, because even though we’re just reading, someone else might concurrently be writing. While someone has the “write lock” on the data, it’s not safe for anyone else to read it. By “not safe” I mean that the data we’re looking at it might be in the middle of being changed, and we could see partial or invalid results because of this.

In other words, a data race exists when two or more users (or different parts of the same program) are accessing the same piece of data concurrently, and at least one of those accesses is a write. So all accesses to mutex-protected data need to be lock-aware, even reads.

Here’s an updated version of our example program, this time with a sync.Mutex which we’ll use to protect the copies variable from concurrent access:

func main() {
    var mu sync.Mutex
    copies := 1
    go func() {
        mu.Lock()
        defer mu.Unlock()
        if copies > 0 {
            copies--
            fmt.Println("Customer A got the book")
        }
    }()
    go func() {
        mu.Lock()
        defer mu.Unlock()
        if copies > 0 {
            copies--
            fmt.Println("Customer B got the book")
        }
    }()
    time.Sleep(time.Second)
}

(Listing copies/2)

The “lock, defer unlock” pattern

Everything is the same as before except that now we have an extra variable mu, the mutex. Here’s the customer A goroutine (customer B’s is identical except for the printed message):

mu.Lock()
defer mu.Unlock()
if copies > 0 {
    copies--
    fmt.Println("Customer A got the book")
}

If the mutex scheme is going to work, then each goroutine has to obtain the lock before trying to read the copies variable. That’s what mu.Lock() does.

If the lock is available, great: we’ll obtain it and proceed straight away. If it’s in use by someone else, the call to mu.Lock() will wait until it’s released (that is, the calling goroutine will move to the blocked queue until the lock becomes available).

Once the goroutine has got the lock (and made it back to the front of the ready queue, if it had to wait in the meantime), it can proceed. We then defer the call to mu.Unlock(), ensuring that whatever happens in this goroutine, we release the lock when the function exits.

Taking the token

The sequence of events for each goroutine will be:

1. Obtain lock (blocking if necessary).
2. Read copies.
3. If the book is still available, set copies to zero.
4. Release lock.

This time, at step 3, the book can only be available to exactly one of the goroutines. The two critical sections (steps 2-3 in each goroutine) can no longer overlap: that’s what the mutex prevents.

It’s a little like a single-track railway line, where only one train can safely occupy a given section of line at a time. If two trains try to share it, especially when they’re travelling in opposite directions, it’s usually a bad day for everyone concerned. To prevent this, we need a way to enforce mutual exclusion of trains in each block (the equivalent of our critical section).

One typical scheme relies on a physical token that the train driver has to obtain before entering the block. On reaching the other end of the block, the driver hands the token back to a signaller, so that it can be used by the next train, and so on. Mechanical interlocking prevents the points and signals being set for any train movement until the corresponding token is in place.

The mutex in our example program is the digital equivalent of the token: each goroutine must wait until the token is available, signalling that it can enter the critical section. Once it leaves, it gives up the token so that other goroutines can use it to enter their own critical sections.

A race-free program with locking

In our book example, the critical section starts where the goroutine reads copies, and ends after it (optionally) writes to the same variable. So, before the goroutine enters this section, it must obtain the “token”, by calling mu.Lock(). After it leaves, it gives up the token by calling mu.Unlock() (via a defer statement).

One of two things has to happen, then: either the customer A routine gets the lock first and updates copies, meaning that customer B misses out, or the customer B goroutine gets the lock first, and customer A misses out.

Access to the copies variable is guaranteed by the lock to be mutually exclusive, eliminating the data race. Let’s see if the race detector agrees:

go run -race main.go

Customer B got the book

Well, that’s great news for customer B, and it’s good news for us, too: we no longer have a data race. Sadly, this is little comfort to customer A, who missed out. Maybe they’ll come back tomorrow.

I’m thrilled to reveal the cover for my new Rust romance novel “Zero Cost to Love”, on sale April 1st.

She’s an internationally famous high-performance computing architect. He’s a rumpled, handsome, yet socially awkward basement coder. When they meet-cute by collaborating on a PR to the Rust compiler, they fall in love—blazing fast.

But despite their shared passion for strict typing and zero-cost abstractions, it seems that their love can never be: he favours tabs while she’s a spaces girl.

💖💖💖💖💖 “Readers will cargo run to buy this tempestuous tale of lust and longing frustrated by irreconcilable differences over whitespace. You’ll be moved, not copied.” (Publishers Weekly)

In a world of undefined behaviour, their hearts refuse to segfault. Enjoy the emotional beats:

• The meet-cute: their pull request review spirals into a 47-comment debate about lifetime annotations.

• The first spark: she whispers, “Your borrow checker insight… it’s beautiful.”

• The dark moment: she opens his editor and sees the tabs. The horror. The betrayal.

“How long have you been hiding this from me?”
“It’s configurable per project, Clara!”
“Don’t gaslight me with config, Daniel!”

Then the reconciliation at the end when they realise rustfmt can save them. True love, auto-formatted.

Don’t miss my other, marginally less romantic Rust books.

Your ideal form of influence is first to help people see their world more clearly, and then to let them decide what to do next.
—Jerry Weinberg, “Secrets of Consulting”

Let’s suppose you’re a self-driving person, and you want to run your own independent business. You’ve practised the secret consultant’s mindset as an employee, and now you’re ready to consultate for real. What does that look like?

What is consulting?

As the name suggests, being a consultant means your primary role is to give advice. You may also do some hands-on work for the client, such as programming, but that’s an optional extra. What you’re there for is to know things, not necessarily to do things.

This distinguishes a consultant from a contractor, whose primary task is to do hands-on work. Generally speaking, the client tells a contractor what to do, whereas a consultant tells the client what to do.

But it’s a blurry line, and you may find yourself on both sides of it with the same client. Indeed, what you think a consultant does is much less relevant than what the client expects a consultant to do.

Four ways to help

As I write in my new guide to consulting and independent working, Master of Your Domain, clients may approach you looking for one or more of the following kinds of help:

• System builder. The client has a requirement for a specific system, application, or service. They want you to build it, train them on how to use it, and perhaps maintain and support it over time.

• Problem solver. The client has a specific issue that they don’t have the knowledge or resources to fix, and that’s where you come in. You may be fixing the problem yourself, or training other people to fix it, or both.

• Problem finder. The client knows they have problems (that’s always true), but they don’t know exactly what they are. Your job is to find out what’s wrong, and help fix it.

• Hired hand. The client knows what needs to be done, they just don’t have the time or resources to do it. Your role is to execute the tasks they give you.

Understanding exactly which of these the client is looking for is important, but it doesn’t mean that your engagement will be limited to just that one aspect of consulting.

Instead, you’ll probably find that what most clients need is some combination of these four things, whether they’re aware of the fact or not. Whatever you’ve been asked to do, your work as a consultant is partly to figure out what else the client really needs, and do that, too.

Three things you know

Still, it’s worth remembering that your primary value to clients is that you know something they don’t. If that’s not true, then you can’t do much for them. Your valuable knowledge falls into three main categories:

• What to do. It’s very unlikely that the client has a problem that no one’s ever seen before. It’s probably been seen and solved many times, and you can guide them to the most appropriate solution.

• What not to do. The client knows they are in dangerous waters, but they don’t know what specifically to beware of. With your help, the client can steer clear of the worst hazards.

• Unknown unknowns. You can spot problems the client may not even know they have, or trouble they might be inadvertently storing up for the future.

One important thing you don’t always know, however, is what the client’s problem actually is. That’s because the problem they initially present to you may not be the one that really needs solving.

Always respond, not with an answer, but with a question: “What problem are you really trying to solve?” If the answer still sounds like a solution, instead of a problem, then keep asking.

The consultant’s job is not merely to give the client what they want, but to convince them to want the right thing. You can only know what that is once you understand what problem they’re really trying to solve.

Listening

The mistake most novice consultants make is to start by fixing the problem. Anxious to show value, they give the client a huge info-dump about everything they need to change, fix, discard, introduce, or understand.

Your job, though, is to not to be the hero of the hour, bravely pulling victims from the flaming wreckage. That might feel very satisfying in the short term, but it won’t bring you much repeat business.

Instead, aim to be the client’s undercover success engineer, discreetly assisting them in just the right ways, and at the right moments, and letting them take all the credit for the resulting turnaround.

After all, they were smart enough to engage you in the first place, so in a way they do deserve the credit. It takes a wise person to know when to ask for help.

And there’s another reason to begin every engagement by listening more than you talk: you might learn something. The client has a much clearer idea of the problem than you do. After all, they’ve been thinking about it for a lot longer.

If you try to start by explaining the problem to them, then, the danger is that you’ll come across at best vague and woolly, and at worst completely wrong.

Instead, be open and receptive, invite them to give you as much context as possible about the situation, and turn your ears up to eleven for as long as it takes for them to tell you what you need to know.

At this point the client will probably be asking “What should we do?” Again, you’ll be tempted to jump right in and start dispensing your hard-won wisdom. But a more thoughtful consultant would ask instead “What have you tried up to now? And what happened?”

If the obvious answers had done the trick, you wouldn’t be here, so it’ll save time to eliminate these in advance. The client has probably already done a lot of work on the issue themselves, so use that as a head start.

Let them get there

When you do present a solution, don’t drop it on the client fully-formed like tablets of heaven. Show them the general direction, and then they can come with you on the journey.

The best way to get someone to buy into an idea is to let them have it themselves. In other words, lead the discussion gently up to a point where what the client needs to do becomes so obvious, they don’t need you to say it for them.

And when you let them be the one to put the idea into words, they’re already halfway on board. You can move on to developing the idea along the right sort of lines: “That sounds great, and then we could…”

Even if someone is against an idea to start with, if you can get them to contribute a tiny bit to it, you may be able to turn them into an enthusiastic advocate.

Underpromise and overdeliver

Be pessimistic about timescales. In this business, “pessimist” is simply what an optimist calls a realist. Everything takes longer than expected, even when we expect it to, because of Hofstadter’s Law. That’s straightforward, but the danger is that the client wants you to lie to them about how long the work will take.

I mean, they don’t want you to lie to them, but they’re extremely motivated to hear an encouraging answer. If you say “three weeks”, they might argue with you until you come up with something else. If you say “a week”, they’ll probably accept it.

Like every other consultant, I grappled with this problem too. When I told clients the work they wanted would take ten hours, they were grumpy, but when I delivered it in only nine hours all was forgiven. When I told them it would take five hours, but delivered it in nine, they were furious—and didn’t use me again.

Telling people what they want to hear is easier than telling them the truth, but it’s just setting yourself up for failure. Letting clients talk you into unachievable timescales is like giving sugar to cranky kids: it might keep them quiet for a little while, but sooner or later a full-scale meltdown is inevitable.

The Batman trap

Things are the way they are because they got that way.
—Jerry Weinberg, “Secrets of Consulting”

There’s a saying that amateurs fix the problem, while professionals fix the system. Solving the immediate pain points is nice, but they’ll probably recur in one form or another. Your real work is helping the client figure out why these issues arose in the first place, and prevent them from happening again.

If you find yourself fixing the same problem for the same client twice in a row, then you’re evidently not addressing the underlying issues. Figure out what the client is doing wrong that means they keep having this problem, and help them build a better system.

The “system” you’re fixing includes the people who work there. A consultant’s natural and creditable desire to be helpful can lead them into the “Batman trap”. This is when people don’t ever actually acquire the skills and knowledge they need to fix things that go wrong. Why would they need to? The consultant always fixes everything!

If you find that your client’s staff regularly send up the bat-signal to get your help dealing with low-level problems, this may also be a signal that you’ve become Batman. A city that has a caped crusader standing ready to deal with crime and disorder is a city that doesn’t have the skills to fix itself. Don’t train your clients to become Gotham.

Of course you want to be a useful consultant, and you will be, but sometimes the highest form of consultancy is “helping by not helping”. Give people the tools they need and show them what to do, but don’t keep turning the wrench for them.

When things break and someone shouts for help, resist the strong temptation to swoop in and save them. If Batman stops showing up for every little incident, people will eventually figure out how to handle them without him. And Batman can move to another city where they haven’t figured that out yet.

Consulting has its limits

Even when we trust and value someone very highly, we don’t always agree with everything they say. There’s only so much influence you can have with any given client, and only so much change that you can achieve for them. It’s important to know when you’ve reached that limit.

One way or another, most consulting engagements come to an end eventually, even when you’ve done everything right. There are always useful lessons to learn, and it’s wise to reflect on what went well and what you’d do differently if you had the chance. Indeed, we learn more from our failures than from our successes. “When you fall down,” as the saying goes, ”pick something up”.

If you’ve enjoyed this series, you might like my new book Master of Your Domain. It’s all about how to build and run your own business, whether that’s consulting, writing, teaching, or whatever you do best. You’ll learn how to market yourself, how to make and sell products, and how to find and delight customers. Happy Independence Day!

Esta página tiene enlaces y recursos en español para aprender el lenguaje de programación Rust.

Libros

• El Lenguaje de Programación Rust — por Steve Klabnik, Carol Nichols y Chris Krycho, con contribuciones de la comunidad Rust
• Rust Embebido — un libro introductorio sobre del uso de Rust en sistemas embebidos
• Rust para Go devs — una guía minimalista de Rust para desarrolladores de Go
• Rust para C#/.NET devs — una guía para desarrolladores de C# y .NET

Artículos

• Rust vs Go: ¿cuál elegir? — Rust y Go comparten mucho, pero sus diferencias importan. ¿Cuál deberías elegir?
• Rust: El lenguaje de programación del futuro — examina qué hace que Rust sea tan especial, sus ventajas y usos
• Rust vs Python — compara la sintaxis, el rendimiento y gestión de la memoria

Cursos

• Comprehensive Rust — un curso gratuito de Rust desarrollado por el equipo de Android en Google

Mentoría y capacitación

• Perfecciona Rust conmigo — mentoría personalizada de Rust
• Entrenamiento Rust de grupos — capacitación transformativa para su equipo

Vídeos

• Rust en español — aprende Rust en español y a tu ritmo con estos vídeos de YouTube de Roberto Huertas
• Rusty Full Stack — un canal en el que aprenderemos a programar en el lenguaje de programación Rust

Software engineers are ingenious people, and we’re particularly clever at avoiding real work by fiddling with our tools, trying new tools, bikeshedding other people’s tools, and having internet flame wars about whether using the wrong editor makes you a bad person.

I’m a Rust teacher and author. Over 45 years of programming, I’ve used a lot of different tools and done an incredible amount of bikeshedding about them. Now, a little older, slightly wiser, and considerably greyer, I don’t care so much about tools: I’m more interested in actually building stuff. Whatever gets the job done is fine by me.

That said, here’s what I mostly use for Rust development today, so that you can berate me about my choices. I am unafraid.

1. Zed

I don’t want to spend a load of time configuring my editor, installing plugins, tweaking settings, and obsessively maintaining a config file. I just want something that looks nice, has great Rust support, and works the way I want it to, right out of the box. That’s Zed.

If you’re familiar with VS Code or Cursor (and it’s hard to imagine that you aren’t), then Zed looks and behaves exactly the way you’d expect. Even the keyboard shortcuts are pretty much the same. It’s just faster, lighter, less bloated, and more reliable.

Why Zed works for me:

• AI integration. I use AI as an occasional assistant or librarian, so it’s nice that Zed has full editor integration: I can highlight some lines and prompt “Refactor this to a method”, or “What’s wrong here?”. There’s also a chat panel to the side, for more in-depth conversations (“Should I be thinking about par_iter here, or just threads?”)

  If AI’s not your jam, Zed also works perfectly well without it—a single switch turns off all AI trappings. For me, though, it’s a handy external brain pack.

• Rust support. With rust-analyzer integration via LSP, I get syntax colouring, Rust-aware renaming and refactoring, inline annotations and type hints, inline errors and warnings, and useful code actions like auto-imports. There’s also an integrated debugger, and though I rarely use it, I appreciate that it’s there.

• Fast and responsive. Zed is a native Rust app using GPU-accelerated rendering, so it feels snappy compared to JS-based tools. Zed runs fine on low-end hardware, and great on anything better.

2. ChatGPT

It’s fashionable in some quarters these days to cry “AI is the devil’s work!” and refuse to adopt any kind of LLM-powered tools. “My code is a holy temple that shall not be profaned by the taint of AI”, say the Butlerian jihadists. That’s okay, and I’m neither a booster nor a doomer—but what I am is pragmatic. If there’s a tool that helps me get my work done better and quicker, I’ll use it.

I almost never ask AI to generate code; that’s the fun part, so I like to reserve it for my own enjoyment. Instead, I’ll talk to ChatGPT about the problem I’m solving, possible lines of approach, things to consider, and how to refine my initial ideas. Then I write the code, and we’ll review and polish it together.

The key ways I use AI for Rust development:

• Helpful pair programmer. As I work, I’ll run into problems and compile errors. Usually, a little thought is enough to resolve these, but when it’s not, I’ll ask ChatGPT to explain to me what’s really going on. Only when I’m confident I completely understand the problem will I start changing code to try to solve it.

• Forensic code reviewer. When I think I’m done, I’ll ask AI to review the code for possible bugs, issues, and edge cases I might have overlooked. ChatGPT usually has some valuable improvements to suggest; I would have figured these out myself eventually, perhaps, but why wait?

• Patient, supportive teacher. I work with ChatGPT all the time to deepen and extend my learning—about Rust and everything else. I’ll ask it to generate quiz questions to test my understanding of some concept, and we’ll review and refine my answers until I’m completely happy that I’ve mastered the topic.

  AI never gets tired or bored, it never gets impatient, and it never belittles me for asking really dumb or obvious questions. It’s not infallible, but who is? I’m learning a lot from AI about how to be a better teacher.

I use the ChatGPT desktop app, which is faster and more functional than the web version, and I almost exclusively use the voice input mode (little microphone icon at the right of the prompt box), which is super responsive and accurate. It’s also way quicker than typing.

Speaking of which…

3. Wispr Flow

Wispr Flow is “voice input everywhere”. It’s like the voice input in ChatGPT, and it uses the same LLM-based speech-to-text API, but it works in any editor, app, text box, or input area. Anywhere you can type, you can talk instead. It feels weird at first, but it’s so much faster it quickly becomes second nature.

I was surprised at how much faster using Wispr Flow is than typing on the Mac, but it’s a real game-changer on mobile. Instead of pecking at that horrible little iPhone keyboard and wrestling with autocorrect, I can just switch to the Wispr Flow “keyboard” and speak the text instead. This is how iOS should have been from day one.

I use Wispr Flow all day, every day, for:

• Emails, chats, texts, WhatsApp, Slack, and socials. I can put in Airpods, step away from the desk, and wander round the house or garden while I catch up on all my daily comms and admin work. Not only is it much quicker than typing, I actually find I’m communicating better and more frequently because it’s easier.

• Writing. I write books for a living, so I write hundreds of thousands of words a year. That’s a lot of time and keystrokes I can save by just saying what I want to say. As a result, my writing is easier, more fluent, and more conversational. I can capture all my rambling thoughts on a subject with Wispr Flow, then use ChatGPT to help me refine and organise them into a working draft.

• AI interaction. I use ChatGPT’s built-in speech-to-text feature a lot, but whenever I’m interacting with AI somewhere that doesn’t have that, such as the Zed agent panel, I can use Wispr Flow to speak my prompts. Now I wince when I see someone painfully and slowly typing out “w-h-a-t-d-o-e-s-t-h-i-s-e-r-r-o-r-m-e-s-s-a-g-e-m-e-a-n-?”. Just say it, bro.

Wispr Flow is the most impressive bit of software I’ve seen in years: they’ve really gone to a lot of trouble to make it smooth, fast, responsive and almost invisible. So much so that I’ve registered to be an affiliate—full disclosure—and if you use my link to sign up, I’ll get a small commission on your first year’s subscription. ’Preciate it.

4. Nushell

Nu is the shell that Rust would be. Calling it a “shell” buries the lede: it’s a thinking language based on pipelines, structured data, and interactive queries.

It’s like SQLite and Powershell had a baby, and then implanted JQ into it at birth, and that baby was a rich, modern programming language inspired by Rust (and the baby was written in Rust). And the baby was a shell.

I can wrangle, slice, dice, and query data from JSON, TOML, CSV, and HTTP APIs, write scripts and programs to automate my most common workflows, and manage all my files using a shell that actually understands data types, instead of just raw text.

Combined with Atuin, the magical shell history manager, so that I need never re-type a command line I’ve used before, and running it all inside Zed’s embedded terminal, I’ve got a dev environment that’s completely in the spirit of Unix, but also very 2026.

5. That’s it

I don’t need much else. Rust already comes with a great set of built-in tools:

• rustup takes care of Rust installation and updates.

• Cargo handles all my build and dependency management tasks

• rust-analyzer gives me Rust power tools in whatever editor I’m using

Other than that, my Mac is pretty stock. I don’t even use Homebrew, because I don’t want a million out-of-date and out-of-band packages cluttering up my hard drive.

A few other bits and pieces that improve my quality of life:

• Brave is my browser of choice, because I don’t like ads (the ability to watch YouTube videos without the ads is life-changing). It supports Chrome extensions, so I can drive it entirely from the keyboard with Vimium.

• Pixelmator Pro is the one-stop graphics, layout, drawing, and photo editing tool that Photoshop used to be. I appreciate being able to pay for a piece of software once, and own it, rather than renting it by the month.

• OmniFocus is the all-singing, all-dancing task manager I’ve used for decades. It syncs across all my devices. I run my life, business, books, and software projects on it. It’s one of the best-value things I’ve ever bought. Not affiliated; I just like it.

Ever wondered if there’s a software engineer, somewhere, who actually knows what they’re doing? Well, I finally found the one serene, omnicompetent guru who writes perfect code. I can’t disclose the location of her mountain hermitage, but I can share her ten mantras of Go excellence. Let’s meditate on them together.

Warning: This is not medical advice. Side effects may include higher code quality, reduced stress levels, and increased salary.

1. Write packages, not programs

What if I told you there was a library of over a million Go packages to do just about anything you could ever want? How much would it speed up your development if you could just import the package that solves your problem, instead of painfully re-inventing it from scratch every time?

The standard library is great, but the universal library of free, open-source software is Go’s biggest asset. Return the favour by writing not just programs, but packages that others can use too.

Apparently we can have nice things, so it’s only fair that if you develop some useful Go code yourself, you should contribute it back to the universal library, right? That means writing your code not merely as a one-off program for your own use case, but as a reliable, reusable, importable software component published with an open-source licence.

Writing packages, not programs, has some design implications, too. Keep your main function minimal: its only job is to process flags and arguments, figure out what the user asked for, and call into your “engine” package to do the actual work.

Your package shouldn’t print anything; instead, it should return the data. Leave it up to the consumer of your package to decide what to do with it. Similarly, don’t call panic or os.Exit in your package; return errors instead. Don’t recover panics from your dependencies either: this can mask problems your consumer needs to know about.

Keep your module structure simple: ideally, a single package. Complex trees of sub-packages make it difficult for users to find what they need, and you’ll give yourself import cycle headaches too. Instead, keep the structure flat, and limit your package to just two files: one for the implementation, and one for the tests.

2. Test everything

Speaking of tests, my Go guru assures me that they’re the only true path to saintly software. When I mentor new Go programmers myself, I sometimes sense their hearts sinking a bit at the mention of tests. There’s a perception that they’re like healthy exercise: undoubtedly a good habit, but one that we all struggle to maintain. “Write tests” is like “Go to the gym“, in other words: good advice, but hard to act on.

On the contrary, I think it’s more like saying “Eat chocolate!” That’s the kind of advice we all love to hear, and it doesn’t take much willpower to apply.

Tests are great, and when you approach them the right way they can be fun to write. They’re a useful design tool, because writing a test makes you the first user of your own function. If it’s awkwardly named, or has too many dependencies, or returns the wrong kind of result, you’ll notice right away. If the thing you’re testing is easy to write a test for, it’ll be easy to use in real programs—and if it’s not, fix it so it is.

Writing tests helps you dogfood your packages: awkward names and inconvenient APIs are obvious when you use them yourself.

Make your tests small and granular, focused on one small piece of logic—maybe a single method or function—and use your package’s public API instead of sneaking behind the curtain to look at implementation details. Those might change, whereas the behaviours your users care about shouldn’t. Check your test coverage to make sure you’ve tested all the code that matters (and it all matters).

• Test names should be sentences

• Focus tests on small units of user-visible behaviour

• Add integration tests for end-to-end checks

• Test binaries with testscript

3. Write code for reading

The best way to make sure you’re writing readable code is to read it. Put yourself in the mindset of someone who doesn’t already know what the code does, and go through it line by line. Is it easy to follow what’s happening? Is the purpose clear? Are the names of functions and variables well-chosen to convey what they represent? How much cognitive load are you asking them to lift?

Read other people’s programs too; as soon as you spot something you don’t understand, ask yourself why not. If the meaning of a certain line is not obvious, ask what change would make it obvious? Don’t rely on comments; these are often wrong, out of date, or merely unhelpful.

In your own code, use comments sparingly, and as a last resort. Focus on explaining why this code is here, not what it does—if that needs explanation, refactor the code to clarify it.

Good names make code read naturally. Design the architecture, name the components, document the details. Simplify wordy functions by moving low-level “paperwork” into smaller functions with informative names (createRequest, parseResponse). Keep each function at roughly the same level of abstraction.

Use consistent naming to maximise glanceability:

• err for errors
• data for arbitrary []bytes
• buf for buffers
• file for *os.File pointers
• path for pathnames
• i for index values
• req for requests
• resp for responses
• ctx for contexts

Don’t be afraid to refactor and re-work your programs ruthlessly until they’re as clear and simple and focused as you can possibly make them. Check this by showing the code to someone else and asking them to talk you through it, line by line. Watching where they stumble will show you the speed-bumps in your code: keep refactoring until you’ve flattened them out.

4. Be safe by default

Use “always valid values” in your programs, and design types so that users can’t accidentally create values that won’t work. Make the zero value useful; this lets users create literals of your type with minimal paperwork. For example, bool fields will default to false, so make that make sense for your type.

Don’t create useless “config” structs; use fields on the object itself to configure its behaviour. If these fields can be invalid, don’t let users write them directly: instead, make them unexported, and provide validating methods to get and set their values.

If your object has sensible defaults, write a constructor method that returns a valid, default object ready to use. Add configuration using “WithX” methods:

widget := NewWidget().WithTimeout(time.Second)

Use named constants instead of magic values. http.StatusOK is self-explanatory; 200 isn’t. Define your own constants so IDEs like GoLand can auto-complete them, preventing typos. Use iota to auto-assign arbitrary values:

const (
    Planet = iota // 0
    Star          // 1
    Comet         // 2
    // ...
)

Prevent security holes by using os.Root instead of os.Open, eliminating path traversal attacks:

root, err := os.OpenRoot("/var/www/assets")
if err != nil {
  return err
}
defer root.Close()
file, err := root.Open("../../../etc/passwd")
// Error: 'openat ../../../etc/passwd: path escapes from parent'

Don’t require your program to run as root or in setuid mode; let users configure the minimal permissions and capabilities they need.

5. Wrap errors, don’t flatten

Don’t type-assert errors or compare error values directly with ==, define named “sentinel” values that users can match errors against:

var ErrOutOfCheese = errors.New("++?????++ Out of Cheese Error. Redo From Start.")

Don’t inspect the string values of errors to find out what they are; this is fragile. Instead, use errors.Is:

if errors.Is(err, ErrOutOfCheese) {

To add run-time information or context to an error, don’t flatten it into a string. Use the %w verb with fmt.Errorf to create a wrapped error:

return fmt.Errorf("GNU Terry Pratchett: %w", ErrOutOfCheese)

This way, errors.Is can still match the wrapped error against your sentinel value, even though it contains extra information.

6. Avoid mutable global state

Even if your package doesn’t create goroutines, your users might use it concurrently. Package-level variables can cause data races: reading a variable from one goroutine while writing it from another can crash your program.

Instead, use a sync.Mutex to prevent concurrent access, or allow access to the data only in a single “guard” goroutine that takes read or write requests via a channel.

Don’t use global objects like http.DefaultServeMux or DefaultClient; packages you import might invisibly change these objects, maliciously or otherwise.

Instead, create a new instance with http.NewServeMux (for example) so that you own it exclusively, and then configure it how you want.

7. Use (structured) concurrency sparingly

Concurrent programming is a minefield: it’s easy to trigger crashes or race conditions. Don’t introduce concurrency to a program unless it’s unavoidable.

When you do use goroutines and channels, keep them strictly confined: once they escape the scope where they’re created, it’s hard to follow the flow of control. “Global” goroutines, like global variables, can lead to hard-to-find bugs.

Make sure any goroutines you create will terminate before the enclosing function exits, using a context or waitgroup:

var wg sync.WaitGroup
wg.Go(task1)
wg.Go(task2)
wg.Wait()

The Wait call ensures that both tasks have completed before we move on, making control flow easy to understand, and preventing resource leaks.

Use errgroups to catch the first error from a number of parallel tasks, and terminate all the others:

var eg errgroup.Group
eg.Go(task1)
eg.Go(task2)
err := eg.Wait()
if err != nil {
    fmt.Printf("error %v: all other tasks cancelled", err)
} else {
    fmt.Println("all tasks completed successfully")
}

When you take a channel as the parameter to a function, take either its send or receive aspect, but not both. This prevents a common kind of deadlock where the function tries to send and receive on the same channel concurrently.

func produce(ch chan<- Event) {
    // can send on `ch` but not receive
}

func consume(ch <-chan Event) {
    // can receive on `ch` but not send
}

8. Decouple code from environment

We all know that good software avoids excessive coupling between packages or components, but many programs are too tightly coupled to the operating system or environment where they run.

Don’t depend on OS or environment-specific details. Don’t use os.Getenv or os.Args deep in your package: only main should access environment variables or command-line arguments.

Instead of taking choices away from users of your package, let them configure it however they want. Be agnostic about how you’re configured. Let users decide whether they want to inject settings via the environment, flags, config files, API calls, or some other way.

Single binaries are easier for users to install, update, and manage; don’t distribute config files. If necessary, create your config file at run time using defaults.

Use go:embed to bundle static data, such as images or certificates, into your binary:

import _ "embed"

//go:embed hello.txt
var s string
fmt.Println(s) // `s` now has the contents of 'hello.txt'

Use xdg instead of hard-coding paths. Don’t assume $HOME exists. Don’t assume any disk storage exists, or is writable.

Go is popular in constrained environments, so be frugal with memory. Don’t read all your data at once; handle one chunk at a time, re-using the same buffer. This will keep your memory footprint small and reduce garbage collection cycles.

9. Design for errors

Always check errors, and handle them if possible, retrying where appropriate. Report run-time errors to the user and exit gracefully, reserving panic for internal program errors. Don’t ignore errors using _: this leads to obscure bugs. Assume that anything that can error will error, and handle it appropriately.

Retry on transient errors if that makes sense. Don’t let the program panic on predictable run-time errors, such as failing to read a file: a stack trace won’t help the user figure out what’s wrong. Reserve panic for unrecoverable internal program bugs only.

Don’t make the user rely on documentation to be able to run your program. Make their first-run experience a pleasant one: instead of nasty error messages, show usage hints and examples. Don’t try to interact with users by prompting them via the console or dialog boxes. Let them automate your program and run it headlessly, using flags or config files to customise its behaviour.

10. Log only actionable information

If you use logging, don’t spam the user with pointless info messages: if nothing needs saying, say nothing. Logorrhea is irritating, so don’t spam the user with trivia.

If you log at all, log only actionable errors that someone needs to fix. Don’t use fancy loggers, just print to the console, and let users redirect that output where they need it. Never log secrets or personal data.

Use slog to generate machine-readable JSON:

logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
logger.Error("oh no", "user", os.Getenv("USER"))
// Output:
// {"time":"...","level":"ERROR","msg":"oh no",
// "user":"bitfield"}

Logging is not for request-scoped troubleshooting: use tracing instead. Don’t log performance data or statistics: that’s what metrics are for.

Guru meditation

Let’s be real: no program is perfect, and the same applies to programmers. We won’t always achieve all the goals set out here, or not at first. It’s more important to get the program working first, get it in front of users early, and only once it does what it’s supposed to should we worry about making the code nicer.

My mountain-dwelling guru also says, “Make it work first, then make it right. Draft a quick walking skeleton, using shameless green, and try it out on real users. Solve their problems first, and only then focus on code quality.”

Equally, though, it’s a mistake not to care about code quality. You never know whether your Go package will be used in something like a medical X-ray machine or a spacecraft control system. All software is critical to somebody.

Software takes more time to maintain than it does to write, so invest an extra 10% effort in refactoring, simplifying, and improving code while you still remember how it works.

Rust y Go comparten mucho, pero sus diferencias importan. ¿Cuál deberías elegir?

Rust es un lenguaje potente, rico y gratificante que prioriza la seguridad y la corrección, sin sacrificar el poder ni la eficiencia. Es ideal para crear software que no solo funcione, sino que siga siendo fiable con el tiempo.

Go es un lenguaje pequeño, fácil de aprender y rápido de escribir. Con Go, los desarrolladores pueden crear software con rapidez y adaptarse con facilidad.

En un eslogan:

Rust para alto riesgo, Go para bajo costo.

Aquí está un resumen de ambos lenguajes que destaca sus ventajas, señala lo que comparten y aclara sus diferencias clave.

Metas compartidas

Rust y Go son lenguajes relativamente modernos que se benefician de las muchas lecciones que hemos aprendido a lo largo de las últimas décadas de la ingeniería de software. Priorizan:

• Seguridad de memoria. Rust y Go eliminan errores graves de seguridad presentes en lenguajes como C y C++, incluidos los buffer overflows (desbordamientos de búfer), los null pointers (punteros nulos) y las data races (carreras de datos).

• Herramientas modernas. Ambos tienen toolchains (cadenas de herramientas) rápidas, potentes y unificadas; con ellos, los desarrolladores pueden escribir, probar y implementar el software con facilidad.

• Rendimiento y escalabilidad. Go y Rust se dirigen explícitamente a los proyectos de gran envergadura, desarrollados por equipos grandes, que tienen que funcionar de forma fiable a gran escala y con alto rendimiento.

Diferencias importantes

Rust y Go tienen mucho en común, particularmente en comparación con lenguajes heredados como C/C++, Java, Python y Ruby. Pero tienen también filosofías de diseño radicalmente diferentes.

Las prioridades clave de diseño de Go son:

• Simplicidad. Es un lenguaje pequeño, fácil de aprender y rápido de escribir, particularmente en proyectos complejos. Deja fuera o simplifica muchas funciones avanzadas de otros lenguajes.

• Abstracción. Go proporciona servicios de alto nivel incluso de concurrencia y garbage collection (recolección de basura), para que el desarrollador pueda enfocarse en resolver el problema en vez de estar ahogado en papeleo.

• Estabilidad. Go enfatiza fuertemente la retrocompatibilidad, lo que facilita el mantenimiento del software a largo plazo. Rara vez introduce funciones nuevas; prefiere realizar mejoras pequeñas e incrementales en rendimiento y calidad de vida.

Rust, por el contrario, se trata de:

• Corrección. Rust se orienta a aplicaciones críticas para la seguridad—industriales, médicas o aeroespaciales—mediante un análisis estático avanzado que elimina muchos errores en tiempo de compilación.

• Rendimiento. El compilador de Rust genera código optimizado que se ejecuta tan rápido como lo permite el hardware, con un rendimiento comparable al de C++ o incluso al lenguaje ensamblador.

• Control. Para obtener el máximo de las CPU modernos, hay que programar a bajo nivel. Rust ofrece control estricto y interoperabilidad excelente con bibliotecas de C/C++.

Cuándo elegir Go

Go es un lenguaje ideal cuando la situación exige:

• Incorporación rápida. Go tiene muy poca sintaxis, pocas palabras clave y solo las funciones esenciales. Los desarrolladores pueden aprender los fundamentos y ser productivos rápidamente.

• Creación rápido de prototipos. Go se adapta a desarrollo ágil porque los equipos pueden crear y evaluar rápidamente prototipos.

• Costes mínimos de desarrollo. Go es ideal para proyectos grandes y equipos porque es más fácil formar a los desarrolladores, y requieran menos experiencia.

Los dominios ideales de Go

Gracias a su enfoque en simplicidad y velocidad, Go es una elección popular para aplicaciones como las siguientes:

• Servicios web, redes y nube. Go está orientado a microservicios pequeños y ligeros, backends web y sistemas basados en contenedores, diseñados para operar a gran escala.

• Software de línea de negocio. Herramientas y flujos de trabajo a medida, gestión de datos, aplicaciones CRM / ERP y automatismos de negocio funcionan bien con el estilo de desarrollo rápido de Go.

• Infraestructura, SRE y monitoreo. Go es ideal para la ingeniería de plataformas y facilita la creación de software confiable y escalable para monitoreo, automatización, despliegue y gestión de configuración

Cuándo elegir Rust

Rust es una buena elección cuando sus prioridades son:

• Fiabilidad y resiliencia. Las funciones de seguridad de Rust son ideales cuando el código debe funcionar correctamente siempre y seguir haciéndolo durante años o incluso décadas.

• Alto rendimiento. Para operating systems, sistemas en tiempo real, aplicaciones de baja latencia, videojuegos y cargas de trabajo de procesamiento intensivo, Rust se ejecuta con gran rapidez y aprovecha al máximo el hardware subyacente.

• Uso eficiente de recursos. La frugalidad y eficiencia de Rust se adaptan bien a hardware limitado o de bajo consumo o bajo coste como dispositivos integrados, IoT, satélites, vehículos y aeronaves autónomas y sistemas militares.

Los dominios ideales de Rust

Los puntos fuertes de Rust—seguridad, corrección y fiabilidad—lo hacen la mejor opción para:

• Automatización industrial y robótica. Rust es ideal para sistemas en tiempo real y telemática, como el control de plantas industriales, máquinas herramienta y la gestión de dispositivos concurrentes.

• Automotriz, aeroespacial y militar. Estos entornos exigentes aprovechan las funciones críticas de seguridad y el control de bajo nivel del hardware que ofrece Rust.

• Medicina y salud. Las cualidades de rendimiento y seguridad de Rust son muy adecuadas para aplicaciones intensivas en datos, como imagen médica y diagnóstico, herramientas quirúrgicas robóticas, dispositivos conectados—incluidos marcapasos y monitores de salud—y la automatización de laboratorios.

Conclusión

Rust y Go son ambos elecciones excelentes para el desarrollo de software de propósito general y ofrecen una combinación de funciones y rendimiento que los sitúa por encima de competidores tradicionales como Java, C/C++, Python, Ruby, JavaScript / TypeScript, C#, PHP, Scala y Swift.

Go prioriza la simplicidad, la escalabilidad y la velocidad de desarrollo, por lo que es la mejor opción para equipos y aplicaciones que necesitan lanzar rápidamente y mantener los costes al mínimo.

Rust, por otro lado, está optimizado para el software crítico para la seguridad que exige el máximo rendimiento; es la elección lógica cuando la fiabilidad prima sobre otros factores.

Bitfield Consulting ofrece capacitación y recursos de aprendizaje efectivos y de gran calidad para Rust, Go y habilidades generales de desarrollo de software:

Vea también Aprende Rust: recursos en español.

Versión en español

Rust and Go have many similarities, but also important differences. Which is right for you?

Rust is a powerful, rich, and rewarding language that prioritises safety and correctness without sacrificing power or efficiency. It’s ideal for building software that has to work and keep working.

Go is a small, elegant language that’s easy to learn and quick to write. Go lets developers move fast, while staying flexible.

In a bumper sticker:

Rust for high stakes, Go for low costs.

Here’s a quick overview of both languages, highlighting the important advantages of each, pointing out what they have in common, and showing where they differ in important ways.

If you’re a developer wondering which language would be better for you to learn, or a business considering which language to adopt from a strategic point of view, read on.

Common goals

Both Rust and Go are relatively modern languages, incorporating many of the lessons we’ve learned from decades of software engineering. They both prioritise:

• Memory safety. Rust and Go are both designed to eliminate the kind of correctness and security bugs that plague older languages such as C and C++, including buffer overflows, null pointers, and data races.

• Modern tooling. Both languages have fast, powerful, unified toolchains that make it easy for developers to build, test, and deploy their software.

• Performance and scale. Go and Rust both explicitly target large projects, developed by large teams, that must operate reliably at global scale in high-throughput applications.

Important differences

While Rust and Go have much in common, especially when compared to legacy languages such as C/C++, Java, Python, and Ruby, they also have radically different design philosophies.

The key design priorities for Go are:

• Simplicity. It’s a small language, intended to be easy to learn and quick to compile, especially in large and complex projects. It leaves out or simplifies many advanced features found in other languages.

• Abstraction. Go provides high-level facilities such as concurrency and garbage collection, letting the programmer focus on solving their problem instead of getting bogged down with admin.

• Stability. Go has a strong emphasis on backwards compatibility, making software easier to maintain over the long term. It rarely introduces new features, preferring small, incremental performance and quality-of-life improvements.

Rust, on the other hand, is all about:

• Correctness. Rust targets safety-critical applications such as industrial, medical, and aerospace, using state-of-the-art static analysis that eliminates many common bugs at compile time.

• Performance. Rust’s clever compiler produces optimised code that runs as fast as the underlying hardware will allow, equalling the performance of C++ or assembly language programs without sacrificing memory safety.

• Control. To get the most out of modern CPUs, programmers need to get “close to the metal”, and Rust offers low-level control and excellent interoperability with C/C++ libraries.

When to choose Go

Go is an ideal language when the situation demands:

• Rapid onboarding. Go has very little syntax, few keywords, and the bare minimum of features. Programmers can learn the fundamentals and be productive in the language very quickly.

• Fast prototyping. Agile development suits Go, because teams can get a skeleton product up and running quickly for evaluation.

• Minimal development costs. Because programmers are easier to train, and need less experience to be effective, Go works well with bigger projects and teams that need to control cost.

Go’s ideal domains

Because of its focus on simplicity and speed, Go is a popular choice for applications such as:

• Web, cloud, or network services. Small, lightweight, high-scale microservices, web application backends, and container-based systems all favour Go.

• Line-of-business tools. Bespoke, site-specific tools and workflows, data management, CRM/ERP applications, and business automations work well with Go’s rapid development style.

• Infrastructure, SRE, and monitoring. Go is an ideal platform engineering language, making it easy to build reliable and scalable monitoring, automation, deployment, and configuration management software.

When to choose Rust

Picking Rust makes sense when the priorities are:

• Reliability and resilience. The safety features of Rust are ideal when code has to work correctly every time and stay working for years or even decades with minimal maintenance.

• High performance. For system kernels and drivers, real-time systems, low-latency applications, gaming, and compute-intensive workloads like AI, Rust code runs fast and takes maximum advantage of the available hardware.

• Efficient use of resources. Rust’s frugality and efficiency suits small, low-power, or low-cost hardware targets such as embedded devices, IoT, satellites, autonomous vehicles and aircraft, and military systems.

Rust’s ideal domains

Rust’s core strengths of safety, correctness, and reliability make it a top choice for:

• Industrial automation and robotics. Rust is ideal for real-time systems and telematics, controlling complex industrial plant or machine tools, and concurrent handling of many devices at once.

• Automotive, aerospace, and military. These challenging environments make the most of Rust’s safety-critical features and low-level hardware control.

• Medical and healthcare. Rust’s performance and safety qualities suit data-intensive imaging and diagnostic applications, robotic surgical tools, connected devices such as pacemakers and home health monitors, and lab automation.

Conclusion

Both Rust and Go are excellent choices for general-purpose software development, and offer a combination of features and performance that lift them above traditional competitors such as Java, C/C++, Python, Ruby, JavaScript / TypeScript, C#, PHP, Scala, and Swift.

Go prioritises simplicity, scale, and speed of development, so it’s the best option for teams and applications that need to ship fast, yet keep costs to a minimum.

Rust, on the other hand, is optimised for safety-critical software demanding maximum performance; it’s the logical choice when reliability trumps all other concerns.

Bitfield Consulting offers high-quality, effective training and learning resources for Rust, Go, and general software development skills:

What’s so great about Rust, then? It’s a fair question, and one that can be surprisingly difficult to answer—at least, when someone puts you on the spot. It’s a bit like suddenly being asked “Why do you love your wife?”. If you feel a little tongue-tied, it’s not because you can’t think of any good answers. Rather, so many reasons may come to mind that you’re not sure what to say first.

And when people ask me “Why Rust?” I know it’s not out of skepticism, but from genuine curiosity. They haven’t yet heard a clear signal through all the noise. They’re really asking “What makes Rust special, and why should I care?”

I’ll try to answer that without tripping over my tongue, or sounding like a shiny-eyed cult member. And when we’re done, if you’re not interested in accepting Rust as your personal saviour, I want you to know that I am okay with that. Go in peace, my friend. Be well.

But first, hear me out.

The core value proposition

Rust is a truly excellent programming language. Perhaps the best we have, by any reasonable definition: powerful, portable, and productive.

It’s modern enough to incorporate the lessons we’ve learned from eighty-odd years of software engineering. Yet it doesn’t spend too many innovation tokens: Rust is founded on solid, well-proven principles that have worked in other languages for decades.

Correctness and safety

Rust leans hard into helping us write correct programs, catching as many problems as possible at compile time, and catching any that remain at run time.

Not all software is safety-critical, but when lives really are at stake—in medical, industrial, military, automotive, and aerospace applications—we need a language that puts reliability front and centre.

Here’s what that looks like in practice:

• Memory safety. Rust automatically eliminates many of the correctness and security problems that plague other languages, such as buffer overflows.

• Data safety. Rust prevents accidental corruption or mutation of data by making sure that only one program thread or task at a time can have write access to values, preventing data races and other mutability issues.

• Referential safety. Rust guarantees that references to data will always be valid, with extensive compile-time checking: null pointers can’t exist in Rust, removing a common source of bugs and crashes.

If this doesn’t mean much to you yet, don’t worry. You don’t need to be a computer science guru to get the benefit of Rust’s safety features. Long story short, Rust protects the programmer by making invalid states unrepresentable.

Low-level control and performance

Safety is great, but it’s nothing without control. Critical software such as OS kernels and drivers, or real-time and high-frequency systems, need unfettered access to the underlying hardware.

This is where Rust really shines, providing:

• Hardware-level access to memory and registers. Rust supports you, but it doesn’t get in your way. When every byte and nanosecond counts, you have full control of the machine.

• Ability to target CPU and GPU-specific features. With broad architecture support, conditional compilation, and inline assembly language, your Rust programs can get the most from modern hardware.

• Controlled bypass of safety checks. Rust lets you temporarily disable the guardrails for only those sections of code where it’s necessary. A special keyword identifies these “manual override” sections, reducing the attack surface for potential bugs and vulnerabilities.

• Fast, predictable performance. Rust compiles to native machine code, giving you the speed and efficiency of C++ or hand-tuned assembler. Rust doesn’t need a garbage collector, so there’s no memory reclamation overhead and no stop-the-world pauses.

• Interoperability with C/C++. Rust programs can talk directly to kernel or SDK code written in C, C++, or any other language, using Rust’s excellent Foreign Function Interface (FFI) support. You can call C libraries from Rust, or mix and match C++ components with Rust packages.

Overall, Rust sits neatly in the “best of both worlds” space between garbage-collected memory-safe languages like Go, and high-performance, close-to-the-metal languages like C++.

Happy, productive developers

Okay, Rust sounds good on the technical side, but what’s it like to use in practice? Well, Rust routinely tops polls of the world’s most-loved languages, so I think the answer must be “pretty great”. Here are a few reasons why:

• Super-helpful compiler. The Rust compiler is not only smart and efficient, it has amazing error messages: not just what’s wrong, but why it’s wrong, and what you need to do to fix it.

• Powerful build system. The Cargo tool manages your dependencies, runs your tests, builds your code, generates your documentation, and publishes your package to Rust’s open-source repository.

• Smart code advisor. Clippy, the Rust linter, not only warns you about problematic code, it even suggests safer, faster, or more idiomatic alternatives.

• Rich, expressive language. Rust gives programmers powerful, state-of-the-art tools for problem-solving: iterators, threads, closures, pattern-matching, sum types, generics, async, and traits.

• Peace of mind. If your Rust program compiles, it probably works. No more stressing about hidden bugs, or whether your code will blow up in production.

Fast-growing, friendly community

Rust adoption is growing rapidly across the entire tech industry, making Rust a logical pick for businesses today:

• Future-proof investment. Rust is a smart and forward-looking choice: there are millions of Rust users and the community is growing all the time. Rust is stable and mature today, and it’ll be around for decades to come; that makes it a reliable long-term bet for your technology stack.

• Gradual migration. You don’t have to throw away your existing codebases. Rust has great interoperability with C, C++, and Java-based languages, as well as Python, meaning that you can adopt Rust gradually into your products, component by component, service by service. Rust supports all modern architectures and operating systems, including WebAssembly for browser-based apps.

For developers looking to build a secure, rewarding career, Rust’s advantages make it a safe bet:

• Valuable skill. Even though demand is growing fast, there aren’t yet all that many experienced Rust engineers out there. New jobs are opening up all the time; you’ll have less competition for those vacancies, a wider choice of employers, and better compensation and job security wherever you decide to land.

• Covers all the bases. Rust is versatile and used across all kinds of industry sectors and applications. You won’t find yourself siloed by your choice of language; instead, Rust keeps your options open.

Happy developers make for a happy community, and the Rust community has a friendly and welcoming atmosphere. Newbies feel supported, and there’s a creative diversity of opinions and backgrounds. Governance is democratic and by consent; Rust is owned by its users, and no big-tech company can take it over or slap a licence fee on it.

Rust programmers take documentation seriously: the language has great facilities for writing detailed hypertext docs with live, interactive examples, Rust itself has all the docs you’ll ever need, and most Rust crates are incredibly well-documented. After all, Rustaceans know that it doesn’t matter how great your software is if no one can figure out how to use it.

Meets you where you are

Rust is attracting developers from all kinds of different backgrounds and language communities, including complete beginners. Here’s what Rust looks like from each of these perspectives:

• For beginners, it’s a great first language. Not only will Rust teach you most of the important software concepts that apply to all languages, it’ll also help you build a safe, thoughtful, and disciplined approach to programming.

• For C/C++ programmers, Rust offers safety plus interoperability. You get the high-level abstractions and low-level control you’re used to, but without the paperwork. Rust takes over the burden of memory management and correctness checks, and provides seamless integration with existing C/C++ libraries and apps.

• For Python users, Rust brings amazing performance. There’s no interpreter or VM overhead, so Rust programs run at the native speed of the hardware. Give your Python codebase a speed boost by using Rust to implement Python modules, or build on your existing investment by calling into Python libraries from your Rust programs.

• For embedded developers, Rust provides low-level control with zero dependencies. It’s no surprise, given its strong focus on correctness, that Rust is now a key component of Linux, Windows, iOS, macOS, Android, and other operating systems. It’s also ideal for constrained environments like microcontrollers and embedded devices: Rust can produce bare-metal binaries needing no runtime, no operating system, and no SDK.

Learn once, use everywhere

Good engineers are T-shaped, as the saying goes: they have a broad knowledge of different technologies, but they also have deep expertise with at least one. So, which language should you spend your valuable time on really mastering?

Rust is:

• versatile enough to write every kind of software

• mature enough to have world-class tools and libraries available

• modern enough that your Rust skills will be valuable for decades to come

Yet it’s really not so radically different to the languages you already know. Take your existing software design and problem-solving skills and bring them to Rust; you’ll find everything works just the way you’d expect.

From Rust, with love

Ultimately, software engineering is not just a job, it’s a craft, where if you bring care and skill to what you do, you’ll get satisfaction and fulfilment in return.

For me, Rust is more than just a tool. Learning Rust is stimulating, writing Rust is liberating, and teaching Rust is rewarding. I’ll let the technical arguments speak for themselves, but in the end perhaps the most persuasive reason I can offer you for considering Rust is this:

• There must be something special about a language that so many people fall in love with. And maybe love doesn’t always need a reason.

You roll over lazily in bed and eye the clock. Somehow or other you don’t quite fancy going to work. How strange! “Hey, boss,” you murmur intimately. “Okay with you if I have the day off? I’m not sick, I just don’t feel like getting out of bed today.” “Sure thing,” your boss responds. “Sleep in, treat yourself to a leisurely late breakfast, and spend the day lazing on the sofa, pottering about, or taking a walk, with my blessing. Do a little light work later if you want to, but if you’re just not feeling it, use the time for self-care instead.”

Sounds good, doesn’t it? You’re not sleeping with the boss, of course; that would be most unethical. You’re the boss.

As we saw in Self-driving people, many of us dream of one day working for ourselves, but just aren’t exactly sure how to get started. The trouble with dreams of independence is that it’s easy for them to remain just that. Without a clear timetable and plan of action, you’re unlikely to achieve much. As with most things, the hardest part of building your independent career is taking the first step.

The best time to start, then, is before you start. Don’t quit your job yet: there’s some ground to prepare before that time comes. It’s wise to start planning and working towards independence long before you take any irrevocable steps:

• You can figure out how to start getting revenue for side hustles, and project when it’ll enable you to go to work for yourself full-time

• You can polish up the skills you’ll need to go solo before you quit, while you’re still drawing that sweet, sweet salary

• You can play a little game I like to call “the secret consultant”; it’s a new way to think about your work, by asking how what you’re doing actually creates value for the person who’s paying for it. Your employer won’t be mad. On the contrary, they’ll be delighted.

Let’s talk about how to start moving the needle towards freedom.

Setting a deadline

While you shouldn’t try to launch the rocket before you’ve built it, you do need some kind of time-bounded plan. Vague deadlines like “in a few years” are too nebulous to spur you into any definite action. Instead, they keep receding as time goes on, always remaining just a little way off, like the end of the rainbow.

So I think it’s important to pick some specific date as your Independence Day: the day you’ll be ready to support yourself entirely with the revenue from your own business. Once you have a deadline, things will start to fall into place.

You’ll be able to identify the specific skills and experience that you need to get between now and then. It’ll give you a head start on the networking, brand-building, and business knowledge you’ll want to have in place when the time comes. And you can figure out how much money you’ll need to be making for independence to become sustainable.

Once you have a reasonable estimate of what your business needs to bring in just to keep your head above water, start thinking about how long it might take you to reach that level of reliable monthly revenue.

Your salary is fairly predictable, but the same won’t be true of your independent business earnings, especially at the start. It may take you months to get your first customer. Even once you’re established, you might make a decent amount one month, but nothing at all the next.

So you can, and should, start working independently long before your official Independence Day, if at all possible. (One thing to check, though, is that your contract of employment actually allows you to do this: some may not.)

Sharpening your skills

You need to be pretty good at something to make a business out of doing that thing. You can’t learn on the job—or rather, you can and will learn a great deal, but you’ll also need to have decent skills before you start.

The best way to get them is on your employer’s dime, of course. So even the most independent-minded people should figure on spending at least a few years in employment before launching their own solo career.

And it makes sense to start with something you’re already good at, something that you can sell, and that most people can’t do for themselves. Remember, you don’t have to be the world’s expert at whatever it is. You only need to be able to do it.

Whatever the skill, ask yourself: are you good enough yet that paying customers should be happy with your work? If not, why not? What aspects do you need to improve to bring your skill up to commercial standard?

You know more than you think

Remember that everyone suffers from impostor syndrome to some extent. Actually, you know a lot more than you realise. That’s because before you know how to do something, it seems complicated and impossible, and after you’ve learned how to do it, it seems trivial and obvious.

A good way to compensate for this cognitive bias is to try teaching somebody else to do the skill. You’ll be amazed at how much you know that most people don’t know. This will bolster your self-confidence, and also help you deepen your own knowledge. When you understand something well enough to explain it to somebody else, you really understand it.

Bear in mind, though, that any specific knowledge, especially about technology, will inevitably go out of date. The more immediately marketable the skill, the more limited its shelf life. That means you’ll need to keep learning, even when you’re working independently.

All good companies invest in research and development, and yours should be no exception. When you’re not fully booked with paying work, use your spare time to read, study, and practise new skills. Even when you are fully booked, make sure you still regularly carve out time to top up your knowledge tank and keep abreast of new developments.

The most valuable skills, though, are those that never become obsolete: problem solving, time management, self-discipline. Patience, determination, methodical working, keeping good notes and records. Absorbing new information, listening to people and understanding their requirements, writing clearly and precisely, explaining complicated technical issues, communicating with non-technical people; all these will stand you in good stead throughout your independent career.

Make a list of these and all the other skills you can think of that would possibly be useful in your future solo business. Grade yourself on all of them, and identify where you think you need to improve. Start working on those areas right now.

Learn on your employer’s dime

While you’re on salary, you have more leeway for this kind of learning and experiment. If it takes you slightly longer to do something because you’re learning new techniques, that’s okay; if you screw it up and have to start again, it’s not the end of the world. Failure is an option.

Use every opportunity you have to bolster the skills and experience you’ve decided you’ll trade on when you go solo. Your employer can hardly object to you taking the initiative on self-training: it makes you more valuable to them as well as to yourself.

With that in mind, it’s a good idea to start thinking of yourself like an independent worker, even before you technically become one.

Let’s play a game called “the secret consultant”. In this game, you role-play as a consultant who’s been brought in to your company to solve some kind of specific problem.

The secret consultant

When you’re a consultant, even a pretend one, it fundamentally shifts the way you think about what you’re doing. Ask yourself questions like “What does the client actually need me to do here? Am I listening and understanding their requirements properly? What am I doing specifically that delivers value to the client? Am I communicating that value to the right people, in the right way?”

When you’re asking these questions, and getting the right answers, you’re winning the secret consultant game. This will build valuable experience and confidence, and you may even find that it makes you better at your job. Your employer may think so, too.

If you feel stressed, anxious, or under pressure in your current position, being a secret consultant can help. After all, a consultant’s engagement is time-limited: whether it works out well or not, they’ll eventually move on to new clients and new problems. It’s easier to maintain a healthy degree of emotional detachment when you know that.

One major source of work stress for employees is the knowledge that mistakes can have long-term consequences: if your boss doesn’t view you favourably, or you have a difficult relationship, it can seriously damage your career prospects. As a consultant, though, that’s not the case. If you screw up an engagement with one client, that’s not ideal, but don’t worry too much about it: there will always be other clients.

When you play the secret consultant, it helps you take a more relaxed attitude to your real job, too. Loosen up a bit, be more experimental, and get less involved in arguments about how to do things. The stakes might be high for the company, but they’re not so high for you, since you won’t be here forever. So chill out, and take a more objective view of things.

Diversifying your client base

One other interesting thing you’ll notice as a secret consultant is that your client base suddenly looks rather over-concentrated. One hundred percent of your revenue comes from a single company, and if anything happens to it, your income will drop to zero right away.

No sensible business owner would let themselves get into such a dangerous position. They wouldn’t want any single client to account for more than, say, a quarter of their total revenue. So they would diversify: they’d look for new clients, learn new skills, open up new areas of business, and spread their risk exposure.

But that’s the one thing you can’t do when you’re a full-time employee. So, if you’re not actively working towards independence already, you’re living on borrowed time. I don’t say this to scare you, but as soon as you start thinking about yourself as a business, full-time employment suddenly seems rather risky, doesn’t it?

On your independence day, and maybe well before it, start getting yourself into a safer and more sustainable position by taking on multiple clients. While the prospect of going it alone may seem scary, it’s actually less risky than the position you’re in right now. You never know when you’re going to lose a client; it could be tomorrow.

Are you ready to be… Master of Your Domain?

So start planning your independent future today, because it might arrive sooner than you expect. My new book Master of Your Domain is all about how to do that, and how to build the skills you’ll need when you’re ready to go to work for yourself.

You’ll learn how to monetize the knowledge and experience you’ve built up over a lifetime, and how to turn them into a real, functioning business with its own letterhead and IRS number.

I’ve packed the book full of everything I know about being a consultant, getting clients, making deals, marketing yourself, making and selling products, producing great online content, teaching, mastering your craft, and even writing and publishing your own books.

If you’re a self-driving person and you’ve been looking for a way to get more agency and autonomy, as well as a better work/life balance, then I wrote Master of Your Domain just for you. I hope you’ll enjoy it, and before too long, you’ll be sleeping with the boss every night. Sweet dreams!

Rust is a popular, modern, memory-safe programming language focused on safety, correctness, and expressiveness, without sacrificing performance.

Here are some of the best books for learning and mastering Rust, as recommended by the Rust community.

The Rust Programming Language (“The Rust Book”)

The Rust Programming Language is the consensus recommendation in the Rust community for the first book every new programmer should read—it’s colloquially known as “the book”. It’s the official introduction to Rust, and available for free online, or in a paper edition from No Starch Press.

The authors

Authors Steve Klabnik, Carol Nichols, and Chris Krycho, are well-known Rust experts and contributors. In addition, the book has received many contributions from the Rust community itself.

Readers say

Readers say it’s comprehensive, and explains fundamental concepts step by step. However, they also note that while The Rust Programming Language is a good reference work, it’s not necessary to read the entire book at first. Beginners are recommended to read the first few chapters to get an understanding of the basics, and save the more advanced material for later.

The first five or six chapters are enough to give learners a basic understanding of the language, and it’s recommended at that point that they try writing some Rust programs themselves, and perhaps move on to a more application-focused book such as The Secrets of Rust: Tools.

Compared to others

It’s more comprehensive than most books, though Programming Rust goes a little deeper on some topics such as the standard library. Unlike The Secrets of Rust: Tools, it doesn’t walk readers through developing realistic Rust projects, or present large chunks of working code like Zero to Production in Rust. However, it’s free, and widely regarded as the standard reference work.

The Secrets of Rust: Tools

Master the world’s favourite programming language with my friendly guide to building durable Rust software.

The Secrets of Rust: Tools is designed as a companion and follow-up to The Rust Programming Language. It focuses on teaching idiomatic Rust, best-practice design patterns, and test-driven development. Readers are guided through building a series of practical, useful tools in Rust, including a memo manager, a weather client, and a Cargo plugin.

The author

Author John Arundel is a respected Rust teacher and writer, with long and deep experience in the software industry and an impressive backlist of books on programming, including The Deeper Love of Go and Cloud Native DevOps with Kubernetes.

Readers say

Readers say The Secrets of Rust: Tools stands out from other Rust books for its beginner-friendly simplicity and focus on clean code. Others commented that the project-based style helps to show how Rust’s features come together into real programs. The clear, witty style met with approval from many, who found it more fun to read than some other books.

Compared to others

Unlike other more advanced books such as Rust for Rustaceans and Zero to Production in Rust, The Secrets of Rust: Tools assumes no previous knowledge of programming, making it ideal for someone learning Rust as a first language. It doesn’t attempt to rehash the material from The Rust Programming Language, focusing instead on how developers can apply their knowledge to solving practical problems.

Rust for Rustaceans

Rust for Rustaceans was published in 2021, meaning that it’s not necessarily up to date with the latest features of Rust, but it’s still a well-respected book that dives deep into the technicalities of the language, and gives readers a thorough analytical grounding in topics like foreign function interfaces, procedural macros, and object.

The author

Jon Gjengset is a well-known figure in the Rust community for his educational videos and livestreams, and he’s also a frequent conference speaker.

Readers say

Rust for Rustaceans is often described as a good choice for intermediate Rust developers who are reasonably confident in their skills but want to deepen their knowledge. Readers appreciated the technical focus, but some found it “confusing” and hard to follow, saying that in their opinion the book was really for experts rather than intermediate developers.

Compared to others

Rust for Rustaceans isn’t as beginner-friendly or as clearly written as something like The Secrets of Rust: Tools, but it’s deeper and more analytical, covering topics like unsafe code, memory ordering, and covariant types. Like Effective Rust, it offers many tips on idiomatic Rust usages, though it doesn’t have such broad coverage of the standard library as Programming Rust.

The Rust Spellbook

Deepen your wizardry with hundreds of tips, tricks, techniques, and tactics for writing better Rust programs.

The Rust Spellbook, published in 2026, is a compendium of hundreds of Rust tips, tricks, and techniques, including useful and little-known features of the language, the standard library, the Cargo tool, and the Rust compiler. It includes many obscure, advanced, or recently-stabilised Rust features of interest to any working programmer.

While many of the tips will be immediately useful, some are deeper, more specialised, and aimed at building a broad, thorough understanding of Rust’s power and features.

The author

John Arundel, whose other books include The Secrets of Rust: Tools, is a Rust teacher and author known for his clear, accessible writing, beginner-friendly approach, and emphasis on software engineering fundamentals.

Readers say

Some readers describe The Rust Spellbook as “well written”, and note that it contains much new information not found in other books (“How did I not know this?”). Others praised the clear explanations of technical topics such as performance analysis and memory alignment, and felt that overall, while aimed at intermediate and advanced developers, the book was still accessible and useful even to Rust beginners.

Compared to others

Unlike The Rust Programming Language or Programming Rust, The Rust Spellbook doesn’t aim to provide an introduction to the language or even a complete guide. Instead, it’s an anthology of Rust “spells”: lesser-known features, patterns, or techniques that improve the quality of life for Rust programmers. It’s longer, deeper, and more detailed than Effective Rust, though doesn’t go into as much depth on any one topic as Rust for Rustaceans. A useful companion to the other books on our list, with almost no overlap in content.

Effective Rust

Effective Rust is a short, information-dense book on idiomatic Rust patterns and APIs. It’s not a beginner book, but targets more experienced programmers who want to fine-tune their knowledge of Rust and learn about where the different language features can most effectively be used.

The author

Author David Drysdale is a staff software engineer at Google, and has many years experience in Rust development.

Readers say

Readers say that the book is “packed with good advice”, and that it offers good jumping-off points for further learning. However, some criticised it as “shallow”, and noted that the book’s use of many examples of bad code could be confusing.

Compared to others

It’s not an introduction to the language, like The Rust Programming Language, or a guide to designing real-world software like The Secrets of Rust: Tools. Instead, Effective Rust focuses on the micro-details: specific method and syntax choices that optimise Rust programs to be idiomatic and efficient.

Programming Rust

Programming Rust is a 2021 book which, like Rust for Rustaceans, is no longer current with modern Rust, but still contains much useful information. It’s especially notable for its comprehensive coverage of all Rust’s built-in types, standard library features, and lots of detailed code examples.

The authors

Jim Blandy is a software developer known for his work on the Firefox web browser and the Emacs editor. Leonora Tindall is a Rust engineer working in the healthcare field.

Readers say

Readers generally describe Programming Rust as “broad and deep”, praising its methodical approach to the nuts and bolts of Rust. Some noted it was not an easy read, but found the writing style relatively clear. Though there were a few complaints about typos and small mistakes, the consensus view is that Programming Rust is a solid work and, at over 700 pages, represents good value for money.

Compared to others

According to reviewers, there’s a fair amount of overlap between Programming Rust and similar books such as The Rust Programming Language and Rust for Rustaceans. While not the most up-to-date book, it has some unique material that’s still worth reading today. Unlike the more introductory works including The Secrets of Rust: Tools, it’s aimed at experienced programmers, especially those coming from a C++ background.

Rust in Action

Rust in Action aims to teach advanced Rust development through a number of technically-challenging projects, including CPU emulation, memory mapping, floating-point encoding, and raw network packet handling. It’s squarely pitched at those with a computer science and engineering background, rather than beginners, but would be suitable for those using Rust in the systems programming domain.

The author

Tim McNamara is a Rust consultant and YouTuber specialising in natural language processing and data engineering. Rust in Action is his first book.

Readers say

Readers described this book as “good for experienced C++ coders”, but warned that it’s “not for low-level language beginners”. Some found the heavily-annotated code samples confusing, but noted that the material was generally well explained.

Compared to others

Like many of the other titles on this list, Rust in Action is from 2021, so it’s not as up to date as more recent books like The Secrets of Rust: Tools. Unlike The Rust Programming Language or Programming Rust, it doesn’t attempt to be a complete guide to the language, instead focusing on important but niche topics such as systems programming and Windows. It’s strong on the technical detail of things like networking, but unlike Zero to Production in Rust, doesn’t cover popular topics such as web services.

Zero to Production in Rust

Zero to Production in Rust is a deliberately opinionated guide to backend web development, aiming to present a complete, realistic application and guide readers through understanding its intricacies step by step. It’s a little more recent than some books on this list, dating from 2022, though its chosen web framework (actix-web) has nowadays been largely displaced by the more modern and popular axum library.

The author

Luca Palmieri is a Rust developer and co-organiser of the Rust London User Group.

Readers say

Readers praised the book for its “clear, vivid explanations”, and appreciated its focus on a single web stack, rather than trying to help users choose between a bewildering variety of frameworks. However, some criticised the quality of the print book, citing “cheap, curly paper”, though they described the material itself as “comprehensive and well-structured”.

Compared to others

Zero to Production in Rust, unlike most other titles on this list, focuses on a single topic: web services. It covers that area well, but it’s neither a beginner-friendly introduction such as The Secrets of Rust: Tools, nor a deep technical dive like Rust for Rustaceans. It’s most suited to those who want to get web-based Rust applications up and running quickly, giving readers a set of pre-selected options that will work, without going into too much detail on why those particular choices were made.

The best Rust books compared

Here’s a quick overview of the Rust books on this list showing their various features, strengths, and weaknesses

Top picks

Taking into account factors like reviews, recommendations, and Rust community consensus, these are the top three books for Rust developers today.

Best for beginners: The Secrets of Rust: Tools

For those new to Rust, or to programming in general, The Secrets of Rust: Tools is highly praised as a readable, even fun introduction that builds readers’ confidence by guiding them through several practical projects in Rust. It’s the ideal jumping-off point for a new developer’s Rust journey.

Best for intermediate developers: The Rust Spellbook

John Arundel’s The Rust Spellbook is packed with hundreds of useful, practical tips for working Rust programmers, as well as a wealth of technical detail and deep background that will help intermediate developers build a solid understanding of the language.

Best for experienced developers: Rust for Rustaceans

Rust for Rustaceans can be dense and difficult reading, but the material repays careful study with a more thorough understanding of the why of Rust. Rust demands that programmers approach their code in a fundamentally different way from most other languages, and Jon Gjengset’s book explains how to think in Rust at a deep level.

BERNARD: I’m going to have to let you go.
MANNY: What? But I sold a lot of books. I got on well with all the customers.
BERNARD: It’s not that kind of operation.
—Black Books

Imagine you’re browsing in a bookstore: one of those delightful pokey old ones with endless shelves, rooms leading to other rooms, and books piled high on every available surface. Somewhere in the labyrinth, a well-fed ginger cat is sleeping precariously on a stack of Agatha Christies.

You spot something that looks interesting: a copy of Kip Thorne’s Black Holes and Time Warps. As you bend down to look closer, though, the title shimmers and changes. It’s now David Deutsch’s The Fabric of Reality. It’s priced at $10, but as you watch, the label distorts and the numbers flow before your eyes. It just went up to $15.

Well, that’s weird! You notice that, where there was only one copy before, there are now twelve huddled together on the shelf. You look round to see if anyone else is noticing this odd behaviour, and when you look back, you find the books are all gone. The one you were holding has morphed into Douglas Hofstadter’s Gödel, Escher, Bach.

We’d be puzzled indeed if this sort of thing happened even in the otherworldly surroundings of a vintage bookstore. But it’s the kind of problem that we run into all the time in concurrent computer systems, where seemingly stable things can change, fluctuate, or disappear when you’re not looking at them, and even—disconcertingly—when you are. It’s called a data race.

In Starving, sleeping, and yielding, we wrote a simple concurrent program in Go that prints interleaved “hello” messages from two different goroutines:

package main

import (
    "fmt"
    "time"
)

func goroutineB() {
    for i := range 10 {
        fmt.Println("Hello from goroutine B!", i)
        time.Sleep(10 * time.Millisecond)
    }
}

func main() {
    go goroutineB()
    for i := range 10 {
        fmt.Println("Hello from goroutine A!", i)
        time.Sleep(10 * time.Millisecond)
    }
}

(Listing hello/4)

And we saw that because of the non-deterministic behaviour of the scheduler (as explored in Go, go, goroutines), we can’t know in principle what the exact output of a program like this will be. We can predict that we’ll see messages from both goroutines, but we don’t know in exactly what order they’ll appear: this uncertainty is characteristic of concurrent programs.

It’s not really a problem with this program, though, because the goroutines are totally independent: they don’t share any data. Unfortunately, being able to share data between goroutines is precisely what makes Go’s concurrency so useful.

A shared variable

For example, let’s modify this program so that each goroutine prints the value of a shared package-level variable message:

var message = "Hello"

func goroutineB() {
    for range 10 {
        fmt.Println(message, "from goroutine B!")
        time.Sleep(10 * time.Millisecond)
    }
}

func main() {
    go goroutineB()
    for range 10 {
        fmt.Println(message, "from goroutine A!")
        time.Sleep(10 * time.Millisecond)
    }
}

(Listing hello/5)

Here’s the output:

Hello from goroutine A!
Hello from goroutine B!
Hello from goroutine B!
...
Hello from goroutine A!

Racing goroutines

Does a data race exist in this program? We can decide that by asking two questions. First, are multiple goroutines accessing the same variable? Second, is at least one of those accesses a write?

If the answer to both questions is yes, then we have a data race. It turns out that we’re safe for now, because even though the two goroutines are sharing message, all their accesses are reads; no writes.

Let’s live a little dangerously, then. If we have one of our goroutines modify the message variable, then we should have a data race, according to the rules. We’ll change goroutine B as follows:

func goroutineB() {
    message = "Goodbye"
    for range 10 {
        fmt.Println(message, "from goroutine B!")
        time.Sleep(10 * time.Millisecond)
    }
}

(Listing hello/6)

So, when goroutine B runs for the first time, it will set message to the string "Goodbye". Let’s see what the output looks like:

Hello from goroutine A!
Goodbye from goroutine B!
Goodbye from goroutine B!
Goodbye from goroutine A!
...
Goodbye from goroutine B!

Fair enough. For a given iteration of goroutine A, then, will it print “Hello” or “Goodbye”? The answer is that we can’t know in advance, and that’s the significance of a data race. The output of goroutine A depends in this case on whether or not goroutine B has ever executed, and that in turn depends on what goroutine A does.

Data races in concurrent systems

What are the consequences of this for a more realistic concurrent program? Suppose that our sleepy bookstore, Happy Fun Books, has two point-of-sale terminals, both accessing the same book catalog concurrently. And let’s suppose that two customers are both trying to buy the last copy of Black Holes and Time Warps.

We haven’t written a program to do this yet, but let’s imagine there’s a command called buy that, among other things, checks the stock level of the given book and decreases it by 1. If you try to buy abc when there are zero copies of book abc in stock, clearly that can’t succeed, so you get an error message saying “sorry, out of stock”.

So, if there’s initially one copy of abc in the catalog, then what happens when customers A and B both concurrently try to buy it? In theory, one of them must win the race: if customer A is a microsecond ahead of customer B in placing their order, then they’ll get the last copy, and the second attempt to buy abc will fail because there are now zero copies left.

But in practice, computer systems store temporary copies of data in cache memory. We can imagine that our bookstore terminals might use a similar kind of cache arrangement, and in that case these multiple copies of the book data can get out of sync, with alarming consequences.

In my book The Deeper Love of Go, we’ll actually write the buy program, and everything else required to run a bookstore, using a networked database system in Go. It’s not only an introduction to programming for complete beginners, but also a primer on concurrency, distributed systems, and locks for those who missed out on (or snoozed through) the relevant CompSci classes. If you’re enjoying this extract, check out the book!

Double access problems

Here’s what could happen with our duelling customers:

1. Customer A runs buy abc, setting the number of copies of abc in its local cache of the catalog to zero. But customer B also has a cached version of the catalog that now disagrees with customer A’s version, because it still says there is one copy of abc, instead of zero.

2. Concurrently, customer B runs buy abc. This succeeds because, as far as B’s cache is concerned, there’s still one copy of abc left in the catalog.

3. Both customer A and customer B’s caches now agree again. Specifically, they agree that there should be zero copies of abc in the catalog, so there’s no conflict, and the system doesn’t detect any problem.

4. But there is a problem! Only one of the two customers can walk out of the store with the book. Yet they’ve both bought and paid for it, and the bookstore system failed to prevent the same copy from being sold twice.

What can we do about this?

Detecting races

Let’s write a Go program that demonstrates a simple version of the duelling-customers problem, and then we’ll see if we can detect and fix it.

Clash of the customers

We’ll represent each customer by a goroutine, and each goroutine will attempt to “buy” the same book by decrementing a shared copies variable:

func main() {
    copies := 1
    go func() {
        if copies > 0 {
            copies--
            fmt.Println("Customer A got the book")
        }
    }()
    go func() {
        if copies > 0 {
            copies--
            fmt.Println("Customer B got the book")
        }
    }()
    time.Sleep(time.Second)
}

(Listing copies/1)

Here, we’re setting up a variable copies, representing the number of copies of the book in stock. It starts at 1.

Next, we create two goroutines, representing customers A and B, each of which does the same thing: it checks to see if copies is greater than zero, and if so, it “buys” the book by setting copies to zero and printing a message saying which customer got it.

In total, then, we have three goroutines now, including the main goroutine, which does nothing but call time.Sleep, to give the two “customer” goroutines a chance to run, and to make sure the program doesn’t end too soon.

Demonstrating the race

When this program runs, then, the two customer goroutines should both get a chance to run, and they will race each other for access to copies. Sometimes customer A will get there first, and at other times, customer B: the result should be more or less random.

Let’s see what happens when we run this program ten times in a row, then:

Customer B got the book
Customer A got the book
Customer A got the book
Customer B got the book
Customer A got the book
Customer B got the book
Customer B got the book
Customer B got the book
Customer A got the book
Customer A got the book

We already knew that a data race is present here in theory, because there are multiple goroutines accessing the same variable, and at least one of those accesses is a write. But now we’ve demonstrated the race in practice: we can’t tell in advance whether customer A or B will get the book. The outcome of the program is non-deterministic, in other words.

And we know that in a more realistic simulation of a multi-user system that includes cacheing, a data race means that we can get undesirable results such as both customers apparently purchasing the same copy.

Go has a useful tool to help us here, called the race detector. If we use the -race flag with the go command, the Go runtime will use the race detector to find and report any data races in the program.

We feel it should report one here, so let’s find out:

go run -race main.go

Customer B got the book
==================
WARNING: DATA RACE
Read at 0x00c000090038 by goroutine 6:
  main.main.func1()
      .../love/copies/1/main.go:11 +0x2c

Previous write at 0x00c000090038 by goroutine 7:
  main.main.func2()
      .../love/copies/1/main.go:18 +0x50

Goroutine 6 (running) created at:
  main.main()
      .../love/copies/1/main.go:10 +0xa0

Goroutine 7 (running) created at:
  main.main()
      .../love/copies/1/main.go:16 +0x104
==================
Found 1 data race(s)
exit status 66

The race detector output

We know from the definition of a data race that there must be at least one write to the variable and at least one other conflicting access, and indeed that’s the case. Here’s where the write occurs:

Previous write... by goroutine 7:
  main.main.func2()
      .../love/copies/1/main.go:18

That’s referring to this line in the customer B goroutine (“goroutine 7”, according to the arbitrary numbering scheme used by the runtime):

copies--

Customer B is trying to decrement copies, thus updating the value of the variable, but this is conflicting with a concurrent read by the customer A goroutine (“goroutine 6”):

Read... by goroutine 6:
  main.main.func1()
      .../love/copies/1/main.go:11

That refers to this line in the customer A goroutine:

if copies > 0 {

So, customer A is trying to read the variable, while customer B is concurrently trying to write to it. Go has successfully detected the data race in this program. It’s a good idea to always use the -race flag when you test programs with multiple goroutines, just to make sure you haven’t missed any potential races.

If the race detector is so super-terrific, you might ask, then why isn’t it just turned on all the time? The answer is that it works by adding instrumentation to every piece of data in your program to check if it’s being accessed concurrently. Those checks slow down the program a lot, so we wouldn’t want them enabled when running the program for real, but they remain very useful for testing.

Having proved a data race exists, though, how can we fix it? In the next post, we’ll see how to use Go’s sync.Mutex type to prevent conflicting reads and writes, eliminating the race condition.

What should you read first when you’re learning Go?

Go, also known as Golang, is a modern programming language designed to be easy to learn, quick to compile and run, and with concurrency as a first-class citizen. Go is widely used in all kinds of software, including network services, command-line tools, DevOps, and web applications.

While it’s perfectly possible to get up to speed with Go by simply following online tutorials and writing a few programs yourself, it can really help to have a well-written book at hand which will explain the fundamentals and what makes Go different.

You can’t learn to program from a book, to be sure, any more than you can learn to fly from reading about birds. The best way to learn is still by doing the thing (fortunately, writing software is substantially easier than flying).

A good book on Go will:

• Support your learning
• Help deepen your understanding
• Introduce you to concepts and practices that you weren’t aware of
• Suggest projects and directions for your future learning

Here are some of the best books to help you learn Go, as recommended by the Go community.

The Deeper Love of Go

Introduces the Go programming language for complete beginners, as well as those with experience programming in other languages.

The Deeper Love of Go is a new introduction to Go suitable for complete beginners to programming. If you have some experience with other languages, the book will also explain what’s unique about Go, and how to write idiomatic code.

The author

John Arundel is a well-known Go trainer and writer with many years of experience in Go and many decades of experience in software engineering. He is also the author of several books on Go for readers from beginner to advanced level. The Deeper Love of Go is his introductory book.

Readers say

Readers praised the book for its beginner-friendly qualities, saying that some other programming books have made them feel stuck early on, whereas The Deeper Love of Go helps unlock a lot of computer science fundamentals and programming ideas. Most felt it was well written, with some saying that Arundel’s writing was up there with the best O’Reilly authors.

Concepts such as variables, methods, and pointers are introduced naturally as part of a realistic application managing a small bookstore. Readers enjoyed the project-based approach and the opportunity to solve multiple code challenges throughout the book.

Reviewers praised the book for its gentle yet uncompromising approach to deeper topics such as concurrency and distributed systems. The consensus view is that it’s a solid introduction to programming in general and Go in particular.

Compared to others

The Deeper Love of Go is the most beginner-friendly book on our list. It doesn’t assume any previous knowledge of programming or of Go, and it’s often recommended to those wanting to learn Go as their first language.

It’s not as comprehensive as something like Learning Go since it doesn’t intend to cover all the features of the language. However, it does go into much more detail about the fundamentals it does cover. Like The Power of Go: Tools, it covers CLI tools, APIs, and user interface design, and like Concurrency in Go, it introduces readers to goroutines and the Go scheduler. Unlike advanced books such as Know Go, it doesn’t cover generics.

Learning Go

Learning Go is a single-volume textbook on Go that aims to cover every feature of the language, not necessarily in depth, but giving readers a good overview of the basics. While this one-page-per-concept approach sometimes results in rather terse coverage, especially of some advanced features, Learning Go is a useful reference for Go developers to have on their bookshelf. First published in 2021, it doesn’t cover all of Go’s latest features, but is still a solid introduction. Read our detailed review of Learning Go.

The author

Jon Bodner is a software engineer and Go developer. Learning Go is his only book.

Readers say

Readers generally praise this book, though noting it can be rather dry to read, perhaps because it’s so comprehensive. There were some complaints about confusing examples that referenced concepts which hadn’t yet been introduced.

Many reviewers recommended the book, though they highlighted a number of typos.

Compared to others

Learning Go is different from the other books on our list because it’s a single-volume reference guide to the whole language. As such, it doesn’t have space to go into great detail on any particular topic, but it does present a useful overview. It’s aimed at people with a programming background, so it’s not quite as beginner-friendly as The Deeper Love of Go. And unlike Concurrency in Go, it doesn’t attempt to do a deep dive into concurrent programming.

While not completely up to date, it’s still a great deal more modern than older books such as The Go Programming Language. The second edition does mention generics, but it doesn’t cover them in anything like as much detail as an advanced book like Know Go.

The Power of Go: Tools

Take your next step in Go, with my guide to building powerful, robust tools and packages that solve user problems.

The Power of Go: Tools by John Arundel is designed as a follow-up to his introductory book The Deeper Love of Go. It focuses, not on the Go language as such, but on how to apply it to building practical, useful programs that solve user problems, especially command-line tools. For example, it covers topics such as flags and arguments, dealing with files, JSON and YAML data, HTTP clients and servers, and connecting to databases.

Readers say

In general, readers described this book as useful, practical, and helpful. Some praised its emphasis on testing and real-world examples, while others enjoyed the simplicity-focused approach, with memorable phrases such as “obviousness-oriented programming”.

Reviews were mostly positive, describing the book as clear, concise, often amusing, and generally fun to read.

Compared to others

The Power of Go Tools is an intermediate-level book aimed at those with some Go experience, so it’s not as beginner-friendly as The Deeper Love of Go, which you should read first. It’s also not as comprehensive as Learning Go since it doesn’t aim to cover every aspect of the language, focusing mainly on command-line tools such as DevOps utilities. John Arundel’s books are regularly updated with every new release of Go, so it’s much more current than older books like The Go Programming Language or Concurrency in Go.

Concurrency in Go

Concurrency in Go is a book for advanced developers focusing on concurrent programming in Go. Clearly it’s not for beginners, but it is packed with useful information on patterns and practices for avoiding some of the most common mistakes in concurrent programming.

The author

Katherine Cox-Buday is a computer scientist.

Readers say

Some found the book difficult to read as it requires quite an advanced background in computer science and distributed systems. However, for people with that expertise, the book was recommended as a valuable guide to best practice. Readers praised the book’s examples and explanations of some difficult concepts, but some remarked that the book seemed poorly written in places and that there were many editing errors.

Compared to others

Unlike the other books on our list, Concurrency in Go is focused squarely on one niche topic. It covers that well, but it’s obviously not intended to be an introduction to Go like The Deeper Love of Go, nor an overview of the whole language like Learning Go. It’s a few years old now, but almost everything in the book is still current with modern Go, and it’s essential reading for anybody working on distributed systems.

Know Go

Interfaces, generics, and iterators are key to modern Go. Learn all about them with this easy-to-read but comprehensive guide.

Know Go by John Arundel is a book for intermediate or experienced Go developers, focused mainly on modern features of Go such as generics and iterators. It covers the new standard library APIs and shows many examples of how to use generics in creating flexible and reusable software packages.

It’s a very comprehensive guide to generics, including:

• Type parameters
• Constraints
• Type approximations
• Generic functions and methods
• Generic container types

It also tackles many frequently asked questions, such as:

• The performance implications of generics
• When you should and shouldn’t use generics in your API design

Readers say

Like John Arundel’s other books on our list, readers praised the book as being fun and easy to read with clear explanations even of some difficult material. Some readers said that they’d gone from being apprehensive about using generics to being excited about the possibilities. Readers liked the friendly tone and noted that the book contained many useful and practical tips.

Compared to others

Although Know Go covers mostly advanced topics, it’s quite accessible even to beginning programmers and makes a good follow-up to The Deeper Love of Go and The Power of Go Tools. Because it’s specifically about generics, it provides much better coverage of this subject than a more general reference, such as Learning Go or The Go Programming Language.

The Go Programming Language

The Go Programming Language was one of the first books written on Go, published in 2015, but it’s still regarded as authoritative. Everything it covers is still more or less correct, though it’s missing important recent additions to Go, such as generics and iterators.

It’s aimed at experienced programmers but doesn’t assume any previous knowledge of Go specifically. It contains many working examples of Go code and shows how to use various features of the language and standard library.

The authors

Alan Donovan is a Google developer and a member of the Go team. Brian W. Kernighan is a computer science professor at Princeton, well known for his work on UNIX and for his many books, including The C Programming Language.

Readers say

The Go Programming Language attracts much praise from readers who described it as well-written, thorough, and easy to understand, though some found it dated.

Compared to others

This probably isn’t the first book you should read on Go; The Deeper Love of Go is a better introduction, especially for beginning programmers. It’s also not as up-to-date a reference as a book like Learning Go. And it doesn’t have as in-depth coverage of concurrent programming as a specialist book like Concurrency in Go.

The biggest problem with this book is that it’s ten years old, and much has changed in Go and software engineering in the meantime. However, it’s still worth having on the shelf as a reference, and it’s an acknowledged classic in the field.

Let’s Go / Let’s Go Further

Let’s Go and Let’s Go Further are specifically about writing production web applications in Go. It’s not a tutorial in the language but it’s aimed at those writing real-world Go applications in production and dealing with things such as performance, authentication, and security.

Web applications and frameworks are complex with lots of moving parts. Let’s Go essentially presents readers with a complete and working codebase for them to read and use in their own programs.

It’s not aimed at beginners, and you will need a fair amount of experience with Go programs and related topics such as JSON, HTTP, and web technologies. Read our detailed review of Let’s Go Further.

The author

Alex Edwards is an author and software engineer living in Austria. He also works as a freelance consultant.

Readers say

Readers liked the practical approach and found the book helpful for designing production-ready web apps. Many praised its detailed and thorough approach, though noted it was perhaps a book more for dipping into the areas that interest you rather than reading from cover to cover.

Compared to others

Let’s Go makes a good companion to some of the more general books on our list, such as The Deeper Love of Go and Learning Go. It’s more aimed at showing you what to do rather than explaining why you need to do that. If you’re looking for a more hand-holding approach, try The Power of Go Tools. If you’re looking for more coverage of modern Go, try Know Go.

The best Go books compared

Here’s a quick overview of the Go books on this list showing their various features, strengths, and weaknesses:

Top picks

Here is a quick overview of the books on our list, including our recommendations for which book you should read first at your level.

Best for beginners: The Deeper Love of Go

John Arundel has a well-earned reputation as a clear and accessible explainer. The Deeper Love of Go assumes no previous experience of programming and is an ideal introduction to Go for those without a computer science background.

Best one-volume reference: Learning Go

If you are buying a Go book chiefly to serve as a reference to dip into when you need to know more about some specific topic, Learning Go is ideal. It covers the whole language—in breadth if not depth—and makes a good standard textbook for Go learners.

Best for DevOps and tooling: The Power of Go: Tools

The Power of Go: Tools is aimed specifically at those who want to write tools for systems administration tasks, platform engineering, API clients, file management, data wrangling, and so on. It’s ideal for those who’ve learned the fundamentals of Go and now want to know how to put that knowledge into practice by solving real problems.

Best for web services: Let’s Go / Let’s Go Further

If you’re writing or working on a web application in Go, Alex Edwards’s Let’s Go books are a great source of practical, working code. For more in-depth explanations, though, you’ll need to look elsewhere.

It’s better to be a fake somebody than a real nobody.
—“The Talented Mr. Ripley”

Faking it till you make it isn’t always the best strategy in life, but it can be a useful one in software engineering. In A hard rain’s a-gonna fall, we wrote some Rust code that sends requests to the Weatherstack API, and decodes its responses. So, how happy are we about the correctness of that code?

We’ve put it all into a magic function named get_weather, and we call it like this:

fn main() -> Result<()> {
    let args = Args::parse();
    let location = args.location.join(" ");
    let weather = get_weather(&location, &args.api_key)?;
    println!("{weather}");
    Ok(())
}

(Listing weather_2)

It looks believable, sure, and it does work. But I sense you might still be feeling a little bit unconvinced about the way we so glibly finessed testing get_weather.

We said, if you remember, that we couldn’t test that function because we don’t know what weather conditions the real API will return: that depends on what the weather is when we run the test!

So, in Elephants for breakfast we decided to break up get_weather into two smaller functions, each with its own test. Then we rewrote the get_weather function to use these two subcomponents.

On reflection, though, can we really be confident that get_weather works properly? I mean, it looks correct:

pub fn get_weather(
    location: &str,
    api_key: &str,
) -> Result<Weather> {
    let resp = request(location, api_key).send()?;
    let weather = deserialize(&resp.text()?)?;
    Ok(weather)
}

(Listing weather_2)

Also, when we run the real program, it does successfully call Weatherstack and produce believable results. So there can’t be too much wrong with get_weather. Can there?

If you’ve read my book The Secrets of Rust: Tools, you’ll have noticed that there’s a strong emphasis on testing all the way through. We use tests as a thinking tool, a design tool, and of course to check that our code works properly (and continues to work, even when we fiddle with it over time).

It’s great to have tests, so doesn’t it feel like a bit of a step backwards now to be writing a function without one? It does to me. Let’s dig into that a little.

What could be wrong?

Sometimes manual testing and inspection is good enough, but it’s worth taking a minute or two to think about what might still be missing here.

When I’m wondering how to test something, I usually ask a question like “What could be wrong?” What bugs could there be in get_weather that we wouldn’t detect with the individual tests for request and deserialize?

Well, one obvious one is that we might have forgotten to call request or deserialize! Even the world’s greatest and best-tested functions are no use if you don’t call them.

Also, we could imagine ways in which we might call them wrongly. For example, we could accidentally write:

let resp = request(api_key, location).send()?;

Rust will absolutely not stop us doing this, because both parameters are &str, so it’s fine—syntactically—to swap them around. We’ll end up with the wrong request, and the call to Weatherstack certainly won’t work, but our existing unit tests won’t detect this problem at all. After all, they call the functions correctly, but they can’t validate that other functions do so.

Similarly, we could call deserialize with the wrong argument, or forget to return its result, or make any number of other silly mistakes which nevertheless are bound to happen in a big enough codebase, in a big enough team, or over a long enough span of time.

Why don’t we want to make API calls in tests?

So I completely sympathise with your unhappiness about the current approach to testing get_weather. Can’t we do better? Ideally, we’d like to test this function the same way we’d test any other function: by calling it. So what’s the problem with that?

One problem we identified earlier is that we don’t know exactly what weather conditions the API will return, but that’s not the only thing stopping us. We also don’t really want to make HTTP calls to external servers as part of our tests, because that’s awfully slow (relatively speaking). Okay, one call won’t break the bank, but it’s not an approach that would work well in large test suites with lots of dependent services and endpoints.

We want our tests to be fast, but even more importantly, we want them to be reliable, and the internet just isn’t reliable in general. We might not be in range of a WiFi or cellular connection, or that connection might be slow and flaky, or insecure (we’re passing our secret API key over the wire, after all). Even if everything at our end is working perfectly, Weatherstack itself might be down, or slow, or faulty.

And it doesn’t make sense that our tests would fail in that case, because as long as our code is correct, the tests should always pass.

A fake server using httpmock

So we want to call get_weather, but we don’t want to make a request to the real Weatherstack API. Thinking about it, though, what are we really testing here? Is it critical that get_weather makes its request to Weatherstack specifically, or would any HTTP server do?

As long as it returns the kind of JSON data we expect, in fact, any server would do. What’s to stop us setting up our own local HTTP server, pointing get_weather at it, and having it return some canned JSON we prepared earlier—for example, the data we’re already using in the deserialize test?

There’s nothing at all to stop us doing this, and it sounds like a good idea. In fact, there’s a crate for exactly this:

cargo add --dev httpmock

The httpmock crate makes it easy to set up a simple HTTP server for tests, with a local URL on a random port, and configure it to respond to various types of requests.

This pattern is sometimes called a test “double” or “fake”, though it’s not, strictly speaking, a fake. It’s a real server: it’s just not the Weatherstack server. It emulates, or “mocks” a subset of the API server’s behaviour so that we can test our code against it.

We’ll also want the http crate, which gives us some handy status code constants to test against, like StatusCode::OK:

cargo add --dev http

Kicking the tyres

Let’s first see if we can use this to test a simple GET request, with no parameters, and respond with a “hello” message:

use httpmock::{Method, MockServer};
use http::StatusCode;

#[test]
fn mock_server_responds_with_hello() {
    let server = MockServer::start();
    server.mock(|when, then| {
        when.method(Method::GET);
        then.status(StatusCode::OK).body("hello");
    });
    let resp = reqwest::blocking::Client::new()
        .get(server.base_url())
        .send()
        .unwrap();
    assert_eq!(resp.status(), StatusCode::OK, "wrong status");
    assert_eq!(resp.text().unwrap(), "hello", "wrong message");
}

To create the test server, we call MockServer::start(). We now have a working HTTP server listening on a random local port, but it’s not very useful yet, because it responds to every request with “404 Not Found”.

Configuring routes on the mock server

To fix that, we’ll tell the mock server how to respond to certain requests:

server.mock(|when, then| {
    when.method(Method::GET);
    then.status(StatusCode::OK).body("hello");
});

We’re saying when you receive any GET request, no matter what the path, then send a response whose status is OK, and whose body is the string “hello”. Ignore all other requests (or rather, continue to respond to them with “404 Not Found”).

So, let’s make that request and see what happens:

let resp = reqwest::blocking::Client::new()
    .get(server.base_url())
    .send()
    .unwrap();

How do we know what URL to make the request to? The server itself tells us, when we call server.base_url().

Now we have a response that we can check to see if it’s what it should be:

assert_eq!(resp.status(), StatusCode::OK, "wrong status");
assert_eq!(resp.text().unwrap(), "hello", "wrong message");

Mocking the real API

That was a good confidence builder. We think we should now be able to extend this to a more complicated mock server that can check its query parameters, and respond with data from a file instead of a string literal. Here goes:

#[test]
fn get_weather_fn_makes_correct_api_call() {
    let server = MockServer::start();
    server.mock(|when, then| {
        when.method(Method::GET)
            .path("/current")
            .query_param("query", "London,UK")
            .query_param("access_key", "dummy api key");
        then.status(StatusCode::OK)
            .header("content-type", "application/json")
            .body_from_file("tests/data/ws.json");
    });

Again, we start the mock server and use its mock method to tell it what kind of request to expect (“GET /current”) and what parameters the query should have.

When the server gets this request, it should respond with OK status, a “Content-Type” header of application/json (meaning that the body will contain JSON data), and the data itself from our test file.

Injecting the base URL

This looks good, and now we’re ready to call get_weather. But there’s a problem:

let weather = get_weather("London,UK", "dummy api key").unwrap();

Where do we pass in the mock server’s URL? I mean, get_weather doesn’t take any other parameters, and if we don’t do something it will just go ahead and call the real Weatherstack API, which we don’t want. We need some way for the test to inject this extra information into the client code.

What’s the right way to solve this problem? You might like to have a think about it. Let’s look at one wrong answer first:

let weather = get_weather(&server.base_url(), "London,UK",
    "dummy api key").unwrap();

I mean, no? This get_weather function is out of control with all the paperwork we’re forced to pass to it. An abstraction is supposed to conceal things, but the machinery is dangerously exposed here.

We’ve made this function slightly easier to test by adding the URL parameter, but as a result we’ve actually made it worse for real users, who now have to write:

let weather = get_weather("https://api.weatherstack.com/current",
    "London,UK", "real api key").unwrap(); // wat?

Why should users have to tell us the Weatherstack API URL? Shouldn’t we already know? This just seems weird and unnecessary, and indeed it is.

I don’t mean to sound ungrateful to httpmock, which is terrific. It’s very handy for us to be able to create a mock server to test against. But if we had to make our API worse in order to use it, that would be a shame.

What we’d like is for our get_weather function to use the real API by default, so that users don’t have to pass the Weatherstack URL when calling the function for real. But we’d also like to be able to tell it the URL of our mock server when we call it in tests. So how would that work?

As usual, this slightly contrived cliffhanger will be resolved in the next post. Don’t miss it!

Pizza or ice cream? It’s a question that we as a species have grappled with since the dawn of time, and I think we’d all agree that the only sensible answer is “Both!” If you have the means, of course.

In Bobcoin, blockchains, and cryptocurrency we ran into a related problem with our otherwise delightful digital currency scheme: what’s to stop Sam the scammer from spending the same Bobcoin twice, once on pizza, and again on ice cream?

Digests and hashing

One way is to use a mathematical function to compute a kind of unique fingerprint of each transaction. The function takes a block of Bobcoin data, such as the one containing sketchy Sam’s pizza purchase, and boils its bits down to a single value. Anyone in the Bobcoin network can use the same function independently to compute the fingerprint value, called a digest of the block.

As the name suggests, a digest is usually shorter than the data it summarises. In fact, a digest usually has some fixed length—for example, 256 bits—regardless of the size of the message it represents.

The problem of taking some arbitrarily long string of bits (a message), and turning it into some representative fixed-length string is called hashing, as in “corned-beef hash”, or “making a hash of things”.

My book Explore Go: Cryptography is all about such mathematical magic, and in it you’ll learn how to keep your messages secret, how to protect your digital currency from scammers, and how to be truly random. We’ll implement lots of cryptographic code in Go together, including various kinds of hash functions, and see just how effective they are (or aren’t).

The specific hash function that we use doesn’t matter here, as long as it satisfies some key properties. The most important of these is that, while it’s straightforward to compute the hash of a message, it should be very difficult to reconstruct the original message if you only have the hash. Hash functions are sometimes called “one-way” functions for this reason.

The blockchain

And that’s what we use to stymie Sam. When each node—each computer in the Bobcoin network—has received a block’s worth of transactions, it starts trying to compute the next block. It hashes the contents of the previous block together with the transaction data, and the result is the next block, which it then sends out to the network.

Now we have a way for nodes to verify the order of blocks: by verifying their hashes. Since the hash of any block depends uniquely (or very nearly uniquely) on the previous block, a given set of blocks can only be verified in their true order.

This means we can order all transactions. When Sam spends their Bobcoin on Pat’s pizza, and then tries to spend it again on Irene’s ice cream, Irene can detect that double spending and reject the transaction. Success!

Well, nearly. There’s still a problem: what if two nodes independently compute and send out different versions of the “next” block? Won’t the chain then split into two independently-evolving versions?

To avoid that, a kind of consensus voting takes place. If a node receives two different candidates for the “current” block, it keeps both of them until it can decide which is the authoritative block.

Doomed stubs

How can the node decide which block wins this race? Well, sooner or later it’ll receive another block from the rest of the network. This will be “chained” to one or other of the two blocks.

In other words, the new block will be verifiable only against the hash of either Block A or Block B. Whichever it is, that’s what the network considers to be the authoritative block, and the node can safely discard the other one.

This doomed “stub” version of the chain might still grow for another block or two, depending on the speed of propagation, but eventually it will be discarded. When faced with two competing versions of the chain, nodes will always opt for the longer one, because it represents the majority opinion that this is the “correct” chain. More work has gone into it, so the network votes with its feet—or rather, with its hashes.

Verifying a given transaction, then, is simply a matter of looking at the block that contains it. The more valid blocks that follow this one, the more confident we can be that the transaction has been accepted by other nodes.

Since hashing takes some non-zero amount of computing power, it’s in a node’s interest not to waste time, heat, and money by computing hashes for a doomed stub. Thus, the network rapidly converges on a single authoritative ordering of Bobcoin transactions.

To repay nodes for their hard work in keeping the blockchain going, the node that successfully creates the next block earns a small fee, made up of the leftover fractional bits of Bobcoin from each transaction. Each node’s version of the next block includes a transaction in which this fee is paid to their account. Thus, their reward for winning the race is to have their fee-earning transaction validated by the rest of the network.

Integrity

In a decentralised scheme like this where each node marks its own homework, isn’t there a danger of fraud?

Surely all a malicious node has to do is compute the next block, based on a bogus transaction (“Alice pays Mallory one million Bobcoin”, let’s say), and then keep this stub chain going by computing all future blocks based on it (plus the subsequent genuine transactions). What’s to prevent this?

First, transactions are signed by the person who makes them: only Alice can create a transaction where Alice pays money to someone else, and anyone can independently verify it.

Second, hashing is hard work. It costs some amount of money to compute the hash of each block, so the longer this “fake” chain is kept up, the more it costs Mallory. Eventually it’ll cost them more than they could steal with the fake transaction.

In effect, Mallory is fighting the combined hashing power of all the other nodes in the network, which are collaborating to keep the genuine chain going.

The genuine chain will rapidly become longer than the fake one, because it simply has more hashing power behind it. Very soon, all nodes will discard the shorter chain as invalid, and Mallory’s efforts will be for nothing.

Sure, Mallory could recruit other nodes into their Bobcoin-stealing gang, but they’ll still be defeated by the honest nodes. Only if Mallory can commandeer more than half of the total hashrate in the network can they hope to subvert the blockchain.

Such a 51% attack is expensive and infeasible for large networks, which means that the more nodes join a given blockchain network, the more secure it is: safety in numbers.

Proof of work

A third way in which malicious blocks are discouraged is to make the hashing process as slow and expensive as possible. Usually, we want hashing to be fast and efficient, but in this case the point is that it should take a lot of time and effort.

In the case of the real-life Bitcoin network, for example, the hashing problem is made harder by requiring miners—nodes that want to produce blocks—to produce a hash value within a specific range.

For example, suppose you’re given some transaction data and asked to hash it so that the 256-bit hash value is less than, say, a million. Of course you can only do this by mixing in some extra data of your own—some nonce—to produce the required value. And since you can’t reverse the SHA-256 algorithm to find the input number you need (it’s preimage-resistant), the only way you can do it is by guessing.

It’s like a kind of benign brute-force attack, or a version of the “guess the number I’m thinking of” game for extremely patient players. You just keep guessing a different value for the nonce and hashing the block until, by pure chance, your hash value comes out as some number less than 1,000,000.

That might take a while: given the size of the 256-bit hashspace, your odds of producing an acceptable hash are about one in 10⁷¹ for each attempt. It therefore costs quite a bit of money just to produce each block, making attempted fraud less attractive. Mining fees are structured so as to make it more rewarding to mine genuine blocks than fake ones.

In Bitcoin, the required difficulty target is a little easier than our example: at the time of writing, the chances of producing an acceptable hash are currently about one in 10⁶⁴. The difficulty target gradually increases over time, to keep pace with the expected increase in computing power.

Up in smoke

If I tell you that, despite this astronomical difficulty, a new Bitcoin block is mined, on average, about every ten minutes, that gives you some idea of the enormous computing resources devoted to the network.

Tens of thousands of Bitcoin nodes are all hashing as fast as they can, five hundred billion billion hashes a second, trying to guess a nonce that will produce a valid hash, and only one node can win that race. All the work done by the losing nodes is wasted, and simply goes up in smoke. Literally, since most of the electricity they use comes from burning fossil fuels.

This proof of work scheme, then, depends on wasting a massive amount of precious energy. Bitcoin alone consumes over 120 terawatt-hours of electricity every year, which is more than many countries.

In effect, Bitcoin adds another energy-hungry country to the world, and we don’t really need another one of those.

It seems a shame that the most powerful distributed supercomputer in human history is being wasted on polluting our atmosphere and heating up the planet by solving math problems that are totally useless to humanity.

A stake in the future

Proof of work, then, is perhaps the worst idea ever, but fortunately there are some alternatives in prospect.

One is so-called proof of stake. In this scheme, rather than the most trusted nodes being the ones that have burnt up the most energy, they are simply the ones who have the most money. These nodes, the reasoning goes, have the biggest stake in the network, so they’re least inclined to undermine it.

That’s true to some extent, but those who find the democratising nature of cryptocurrency appealing may think that this is just another way of rigging the system in favour of its richest participants. And we have that already: it’s called “capitalism”.

Clearly we need a fair, efficient way of preventing fraud and double-spending that doesn’t literally set fire to the planet, and we don’t have that at the moment. Maybe someone will have a bright idea about this soon. Maybe it’ll be you.

It's a data explosion. Servers, clusters, pods, microservices, response times, resource usage, web traffic, monitoring statuses: modern infrastructure generates a heck of a lot of data, and it all has to go somewhere.

Metrics, metrics everywhere

These days, that somewhere is usually Prometheus, or something very much like it. Prometheus is an open source database for collecting, storing, and querying metrics data, which looks like this:

# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.175552e+08

The Prometheus server can collect (or 'scrape') data from a range of sources, and you can then use a standard query language (PromQL) to ask questions about it, or use it to generate graphs and dashboards using the popular Grafana tool.

If you don't want to run your own Prometheus instance, there are lots of hosted services which can import and export Prometheus data. It's the de facto standard interchange format for metrics data.

Checkly, meet Prometheus

While the Checkly dashboard is great for focusing on your monitoring information and the status of your various checks, it can also be useful to send this data to Prometheus:

• Perhaps you already have Grafana dashboards for your various services, and want to incorporate your Checkly data into these.

• Maybe you have an alerting system set up for other Prometheus metrics, and you want to use it with Checkly too.

• Maybe you want to do more sophisticated queries and alerts on Checkly data: for example, you might alert only if more than 10% of your checks are in a down state, or if response times rise above your service level objectives in more than two regions simultaneously.

It's easy to connect Checkly to your existing Prometheus server, and you don't need any new infrastructure. In this tutorial, we'll learn how to use Checkly's Prometheus integration in just a few simple steps.

Checkly metrics

So what kind of Checkly data can we use in Prometheus? There are currently three metrics available:

• checkly_check_status shows the up/down status for a given check

• checkly_check_degraded_status indicates if a check is degraded (that is, responding slowly, but not so slowly that it's considered down)

• checkly_check_result gives you the response time figures for each check, per region

When you connect your Checkly account to Prometheus, these metrics will automatically be available for all your configured checks.

Enabling the Prometheus integration

Let's get started! Assuming you have a Prometheus server or are using a hosted service which can scrape Prometheus metrics, you'll also need a Checkly account. If you haven't got one yet, sign up for a free trial.

You'll also need some monitoring checks configured. Follow the instructions to create a simple browser check, if you don't already have some set up. Alternatively, use the Checkly Terraform provider to create some checks using Terraform code.

Go to the integrations page in your account settings, and find the button that says 'Create Prometheus endpoint'.

When you click this, Prometheus data will be enabled for your account, and you'll see two important pieces of information: the API endpoint URL, and the bearer token. The URL tells your Prometheus server where to look for data, and the bearer token is the password that gives access to it.

Configuring Prometheus

You're now ready to configure your Prometheus server to scrape the new endpoint. To do this, add a stanza like the following to your prometheus.yml file:

# prometheus.yml
- job_name: 'checkly'
  scrape_interval: 30s
  metrics_path: '/accounts/993adb-8ac6-3432-9e80-cb43437bf263/prometheus/metrics'
  bearer_token: 'lSAYpOoLtdAa7ajasoNNS234'
  scheme: https
  static_configs:
  - targets: ['api.checklyhq.com']

Set the metrics_path to the API endpoint URL from your Checkly integration settings (everything from /accounts/... onwards). Set the bearer_token to your own bearer token value.

Apply the new configuration to your Prometheus server and within a minute or so, it should start collecting metrics. To check this, go to the Prometheus web interface and enter this query:

checkly_check_status

Once some data has been collected, you should see it show up in your Prometheus query browser:

Graphing your data

Now that we have the data available in Grafana, what can we do with it?

The first and most obvious thing would be to graph the check status across all your sites. For example, you could add a dashboard panel with a Prometheus query like this:

checkly_check_status

For this metric, a value of 1 indicates 'Pass' (site is up), and 0 indicates 'Fail' (site is down, or is returning an error, or an assertion is failing). So when all sites are up, as they should be, you'll see something like this:

All five of our test sites are at status 1, so there's just one line. If you want to see each site as a separate line, enable the 'Stack' option in the Grafana visualization settings:

This is a good way to see the current status of all your sites over time. But suppose we want to answer the question "What percentage of my sites are up at any given point in time?" Another way to ask this question is "How reliable is my site serving infrastructure?"

Reliability

We can answer the 'reliability' question by visualizing the average check status as a percentage. For example, if you have five sites, and two of them are down at a given moment, your 'reliability' graph would show 60% at that moment in the time series. Here's the query:

avg(checkly_check_status) * 100

And here's what it looks like:

You may like to review this 'reliability' graph from time to time with your engineers. Minor dips and wiggles are to be expected, of course, but over the long term you should expect to see this graph steady or improving. A gradual (or worse, rapid) decline in reliability wouldn't be a good sign.

We can also cross-reference the check status data against other metrics. For example, you could visualize reliability against your cloud hosting bill! As you spend more on cloud, you can see what return you're getting in terms of reliability improvements... or not.

Graphing latency

The check status is very useful, but we have more data to play with. The checkly_check_result metric gives us the response time in milliseconds (referred to as latency) for each site, for each region it's checked from:

It might be interesting to graph, for example, how the performance of a single site varies by region. We could use a query like this:

checkly_check_result{check_name=~"Site 1"}

This will show only results for Site 1, but with a separate line for response data from each region:

In this data, for example, we can see a spike in latency after 17:00, which affected the eu-west-1 and us-west-1 regions considerably, while ap-northeast-1 fared slightly better. Reviewing data like this over a long time period can help you make the right decisions about where to host sites for best performance, or which geographic areas may have been more or less affected by an outage, for example.

It's also useful to see the average latency across all regions, with a query like:

avg(checkly_check_result{check_name=~"Site 1"})

This evens out fluctuations which may be due to local network issues, and gives you a sense of what your global performance looks like:

Graphing region performance

It might be interesting to ask the question "Which region has the best response time for all my sites, and which the worst?" For example, we could write a query like the following:

sum by (region) (checkly_check_result)

This shows us the total latency for all sites from each region. The result looks like this:

We can see that, across all sites, the eu-west-1 region is clearly the fastest; customers in that region have the best experience of your sites. The us-west-1 region is a little slower, and the ap-northeast-1 region is slower still. It might not be a bad guess that all the example sites are hosted in the EU, which is in fact the case. The differing response times of the other regions reflect global internet topology: the EU has fast links to the USA, but traffic to Asia takes longer.

Where most of the customers for a particular site are clustered in one region, as sometimes happens, you can use data like this to help you relocate sites to the most appropriate region.

If a particular region suddenly shows a big blip on the graph, that could indicate a network problem affecting that region, and you might be able to take action to fix the issue for customers in that area by deploying a local replica, for example.

Combining Checkly with other metrics

The monitoring data supplied by Checkly is useful in itself, as we've seen, but it can be even more useful when you combine it with other metrics. For example, suppose you suspect that one of your servers is getting overloaded, and you see occasional alerts for excessive CPU usage. How can you tell if this is causing site outages?

Since Grafana can graph more than one PromQL query at once, we might give it the following queries:

(1 - avg(irate(node_cpu{mode="idle", node=~"web164"}[15m])) by (instance)) * 100
checkly_check_status{check_name=~"Site 1"} * 100

The first query calculates the percentage of CPU utilisation for a specific server over a 15-minute period. For example, if the CPU was pegged at 100% usage over that time, the graph would show 100%.

The second query shows the check status for a specific site, as a percentage (so that it's on the same scale as the first query). When the site is up, the graph will show 100%, and when it's down, 0%.

The resulting graph might look something like this:

It's now much easier to see if a particular CPU spike coincides with monitoring failures.

Performance against SLA

It can be hard, when looking at a bunch of graphs, to know what you're really looking for. For example, if you're looking at a latency graph for a particular site, is there a problem or isn't there?

Maybe, maybe not. We can simplify things considerably by setting ourselves a Service Level Agreement (SLA), which for our purposes here simply defines what 'problem' means. If we say the SLA for a given site is 2000ms, we are saying that if the response time is below that, everything's fine, but if it's over that, there's a problem.

This means that we can produce a graph that's much easier to read: whether or not a given site is over SLA. For example:

checkly_check_result{check_name=~"Site 1"} > bool 2000

This will show a value of 1 on the graph when the site is over SLA, and 0 when it isn't. The results might look like this:

We can see clearly that there have been several out-of-SLA incidents in the last few hours, mostly from the ap-northeast-1 region, but after 17:00, all three regions tripped the SLA alarm. Breaking the data down by region like this helps us eliminate 'local' latency problems, and spot incidents where (since all regions are affected) we probably have a server issue.

Tripwires

One other useful thing monitoring can do is supply us 'quick glance' information. Sometimes we just want to get a very high-level overview of what's going on: basically, answering the question "Is everything okay?"

So one final Grafana dashboard you may like to set up is one we call 'tripwires'. A tripwire is a danger signal, like a check failure, or a high-CPU alert. One or two individual signals probably aren't a concern. But if multiple things start tripping your tripwires at once, you may have a full-scale emergency on your hands.

Your tripwires will likely be repeaters of existing graphs or metrics you already have, but clustered together in one place for easy visibility. It's a good idea to use Grafana's 'Singlestat' visualization for this, which shows a metric or query as a single number, with an optional green, yellow, or red colour depending on the value.

For example, we could take the 'check status' graph we looked at earlier, and turn it into a Singlestat tripwire that answers the question "Are any sites down right now?"

We can start with a query like this:

avg(checkly_check_status)

We choose the 'Singlestat' visualization type, define the thresholds for yellow and red status as 0.1 and 0.9, and set the colour mode to 'Background':

Just showing the number '1' would work, but isn't very informative, so let's map the different values to some appropriate wording:

Now the range '1 to 1' will show the text 'ALL UP', and the range '0-0.99' will show 'MAN DOWN'.

Here's a small example set of tripwires that could be useful. 'Any sites down' is the most important one, followed by tripwires for each individual site, and then CPU, disk, memory, and network usage, with appropriate colour ranges defined for each one.

When you have multiple servers to monitor, you can have the tripwire show the highest value in each category. For example, if your busiest server is running at 89% CPU, your tripwire would show '89%'.

The exact set of tripwires that you find useful will be specific to your infrastructure, and you will probably be able to refine it over time, based on your experience. But if you start with a few sensible ones, such as check status, and 'site over SLA', for example, you won't go far wrong.

Conclusion

If you're not already using Prometheus and Grafana to collect, query, visualize, and alert on your infrastructure metrics, this is a great time to start. You'll find it gives you some important advantages over more traditional 'up or down' monitoring, and some powerful tools for diagnosis and troubleshooting.

Connecting your existing Checkly monitoring to Prometheus and Grafana is very easy to do, and really worthwhile. Not only can you bring all your infrastructure data into a single portal and graph it, you can set up more complex queries and alerts based on combinations and aggregates of metrics.

If you're not already using Checkly to monitor your sites for status, performance, and user experience, that's something well worth looking into. You can try the service for free for 14 days, and that includes the Prometheus/Grafana integration. Enjoy!

This article, written with the invaluable assistance of Andy Pearson, originally appeared on the Checkly blog. Header image by Aegir Hallmundur.

An interview with John Arundel for the JAX DevOps blog

Who is leading the DevOps show? Develops or operators?

The point is that they’re the same people, whether they realize it or not. Developers are intimately responsible for how their code performs in production; good developers relish that responsibility because direct feedback (such as being on call) makes better software and better developers. Meanwhile, operators are the people who write the code which provisions the infrastructure, deploys the software, monitors the services, and so on; they’re just as many developers as the developers, but they work on a different codebase. Developers are operating, and operators are developing; that’s DevOps.

The focus is slowly shifting from “What is DevOps?” Where do we start?”. How do we answer the second question?

Put your developers on call for production.

How important is it to incorporate security into DevOps (DevSecOps)? What are the benefits? Should it be a priority or an afterthought?

Security should be a mindset. Whenever you’re writing code that someone other than you will use, you have to think about ways they might try to attack and subvert it. Even better, you should try to do this yourself. This also applies to other people’s code that you rely on (for example, web servers). You will never have security because security is a process, not a feature. The minute you stop doing security, security is gone.

Put your developers on call for production.

How important is automation in a DevOps context and what are the areas where automation is really needed?

Without automation, there is no DevOps. No programmer would waste her time manually translating code into machine language—it’s tedious, error-prone, repetitive, boring, and a waste of time. Instead, she gets the computer to do it, using a program called a compiler. The same applies to provisioning infrastructure, configuring firewalls, installing dependencies, building containers, checking web services, and so on. It’s just that not everyone realizes that yet.

Should testing become an essential component of the CI/CD pipeline? Are we underestimating its importance?

Without testing, there is no CI/CD. If every commit results in a deploy, automated testing is the only way to know that you’re not going to break production. But testing has its limits. In modern, cloud-native, distributed systems everything can and will break at any time, and tests can’t catch that, so your code needs to be defensive about everything it does. Everything can fail. If you can handle the failure gracefully, then handle it; if not, crash, and let your infrastructure handle it. If your system is failure-tolerant, then testing is merely a matter of inducing deliberate failures. If the system is not failure-tolerant, no amount of testing will make it so.

Some companies are still struggling with DevOps metrics. What are the key metrics that matter and how can they enhance DevOps success?

The most important metric that people don’t track is the number of out-of-hours and weekend pages generated by their monitoring system. In other words, how many times were your people woken up by faults in production? If you optimize for this metric, you’ll have a happy team and a highly reliable service.

How important is it to not skip steps in the DevOps transformation cycle? What are the steps that companies and/or teams usually ignore or underestimate?

Not steps, but people. Some people just fundamentally don’t get it, or they do get it, but they don’t like it and won’t adopt it. There is no way around these people. They got used to doing things the traditional way, and resent the ground shifting under them. The most effective way to stop these people holding up the DevOps transformation is to replace them. Most companies are not willing to do this, and that is a major reason why most companies fail to transition to DevOps successfully.

Do you think the abundance of DevOps tools has helped or slowed down DevOps adoption?

There are no DevOps tools. Is a gun a crime tool or a law enforcement tool? It depends on who’s holding it. The difference between doing DevOps and not DevOps is in the people, not the tools.

There’s a huge demand for DevOps professionals. What skills do you need to have in order to tap into the perks that accompany the job description?

DevOps is not about skills either (it’s a lot easier to say what DevOps is not than to say what it is). Because it’s about people working together, one person cannot be a DevOps. *Teams* of people can *do* DevOps, which requires attitudes of mutual respect and collaboration, a willingness to learn and expand your conception of what your job is about, and a pragmatic approach to engineering. When one team of people writes some software, and another team of people runs that software, that’s not DevOps. When one team takes responsibility for the whole lifecycle of their software, from design to production and back again, and their management rewards and incentivizes them accordingly, that’s DevOps.

Thank you very much!

 John Arundel is the author of the recently published Puppet Cookbook, from Packt Publishing, which helps you build reliable, scalable, secure, high-performance systems to fully utilize the power of cloud computing.

Packt: Your book is published now. How does it feel to be a published author?

John: Pretty great! At one time I wrote technical manuals for Psion, the palmtop computer manufacturer. Thanks to a conservative house style, the kind of books I wrote said things like: “To access file operations, select the File menu”. Not exactly a page-turner. I’m very happy now to be able to publish a book which is written more or less exactly the way I want it, on a subject I find very interesting, and with a lot of jokes.

Packt: What benefits did writing a book bring to your specialist area?

John: The funny thing is that despite being a Puppet user almost since the very beginning, I really don’t use many of its features. In fact, most of them have been added since I started using Puppet, and I don’t have a lot of time to experiment with new stuff, so writing the book was a great opportunity to delve into all the Puppet features I didn’t know about. I hope readers will also find out stuff they didn’t know and that it will be useful. If just one person is helped and inspired by this book... then I’m not giving refunds to the others.

It’s done a lot to raise the profile of my consulting business; I was introduced to one potential client as “This is John. He literally wrote the book on Puppet”. I had to modestly point out that in fact, other, probably better books are available.

Packt: Our authors usually have full-time jobs whilst writing for us. Was this the case for you and how did you approach managing your time?

John: As any freelancer knows, the job is more than full-time. I practically had to invent new physics to figure out a way of using my negative free time to write a book.

I blocked out one day a week devoted to writing, and set myself a goal of a number of hours to achieve each month, which I mostly met. Because the book is so code-focused, I not only had to write about each technique I was describing, but also develop complete, working, reusable software in Puppet to implement it, and then test this on a virtual machine.

Quite frequently I’d discover later that I’d been doing something wrong, or a behaviour in Puppet had changed, and I’d have to go back and fix all the code. I’m sure there are still quite a few bugs, which I am going to pretend I’ve deliberately inserted to help the reader learn to debug and fix Puppet code; something they will, after all, spend a great deal of time doing.

In all, what with researching, writing, coding, testing, fixing, editing, and complaining on Twitter, I spent about 200 hours on the book over 8 months.

Packt: Whilst writing your book, did you find that it overshadowed personal life in any way? How did you deal with this?

John: Not really. It could have, if I’d got into serious deadline trouble. Fortunately, I managed to keep up a continuous, manageable level of mild deadline trouble. I don’t think my friends or family noticed, except occasionally I’d say things at dinner like, “Could you pass the Puppet? I mean pepper.”

Packt: Do you have any advice for other authors who may be interested in writing for Packt, but are still unsure?

John: Go for it! But make sure you have two hundred unallocated hours in your schedule. You’d be amazed how much time you can save by not watching TV, going out, putting on clothes, etc.

Really, my advice would be to plan the book carefully - agreeing the outline in advance with your editor helps a lot. Henry Ford said that there are no big problems, just lots of little problems. Breaking down a book into chapters and sections and subsections and tackling them one by one makes it seem less daunting. And managing your time well helps avoid last-minute-essay syndrome.

Packt: Do you have any tips for other authors, or tricks that you learnt whilst writing, that you'd like to share?

John: One good tip is, once you’ve written a chapter, let it lie fallow for a few weeks and then come back to it with a fresh eye. What you thought were immaculately-crafted sentences turn out to be pompous waffle. And what seemed clear and explicit now seems larded with techno-babble.
I read somewhere that P.G. Wodehouse would stick each page of manuscript to the wall as he wrote it, somewhere around the skirting board level, and as he obsessively reworked and rewrote and polished the text he would gradually move it higher and higher up the wall until he judged it good enough - somewhere near the ceiling. Well, I’m not saying I’m P.G. Wodehouse — I’ll leave that for others to say — but it’s a useful way to think about the writing process. Rewrite, rewrite, rewrite! “What is written without effort,” Dr Johnson pointed out, “is in general read without pleasure.”

Packt: How did you find the overall experience of writing your book for Packt?

John: It ranged from great fun to grinding tedium and crushing despair. It’s a bit like sculpting a block of marble - at first it’s all hard work and it seems like you’re getting nowhere, and standing in a pile of rubble. Then as you go on, things start to take shape and you’re motivated to go on because you see your imagined shape emerging from the stone. In the final stages it’s very exciting because you’re just removing the last few crumbs which are clinging to the finished sculpture.

What really made it a terrific experience was the help and support I got from people in the Puppet community, on Twitter and IRC, at meet-ups and user groups, and by e-mail. People kept asking about the book and sounding really excited about it, which was a great motivator. Many Puppet users who are much more expert than me gave up their free time to help me fix my code or my text. Many of the ideas and techniques in the book came from blog posts or code that other people have generously made public, and I’ve made a point of citing the sources for things wherever I can, and giving a link for the reader to go and explore the topic in more detail.

Since the book’s been published, I’ve been really touched that so many people have said kind things about it. I never imagined that it would be so widely read or attract so much favourable comment, and it makes me glad now that I put the effort into making it the very best thing I could do.

Packt: During the writing process, did you come across any issues/ difficulties that affected your writing and how did you overcome these?

John: One of the main difficulties is that Puppet is a moving target. We decided early on in the process to pick 2.7 as our reference version, since that was shortly to be released and would still be current when the book was published. Including lots of caveats such as, “but if you’re using Puppet 2.6 it’s this syntax, and with 0.25.4 you can’t use this feature...” would have cluttered up the book and made it unmanageable. Similarly, I had to choose one operating system — Ubuntu. Elaborating all the tiny changes you need for Debian, CentOS, Red Hat Linux, and so on would have made the book a massive doorstop. It’s my conviction that people who buy this book are smart enough to figure out those differences for themselves.

However, I’m sure that there are still several things in the book which don’t work as they should with the current version of Puppet, and as Puppet develops, it will break more and more of the examples - but if I’m really good all year, perhaps I’ll be allowed a second edition.

Packt: Was there anything interesting that happened during the writing of the book?

John: Only in the sense that the Chinese use, when they curse you to live in interesting times. Quite often I wrote myself to a standstill and just stared blankly at the laptop, hoping it would explain something complicated for me, or think up a useful and instructive example when I couldn’t. On one occasion I decided that the best thing to do with a certain long, difficult, and laboriously-constructed section was to delete it altogether, improving the book immeasurably as a result. The deleted scenes will be available on a forthcoming DVD, together with a ‘making of’ documentary which consists of me frowning at a screen for 200 hours and intermittently making tea.

Packt: How did Packt’s Acquisition Editors help you - what kind of things did they help you with and how did they support you throughout the writing process?

John: The biggest help at the start was giving me structure by insisting on an outline, and then setting individual chapter deadlines to plan the writing time - then gently but persistently enforcing them.
What was also very useful was to see sample chapters of other books, to get an idea of where I was supposed to be going, and getting very detailed feedback on the early chapters about exactly how to lay things out and how to make everything consistent.

Beyond that, I was pleased and surprised by how little the editors interfered with what I was doing. By and large I was allowed to write my own book the way I wanted. No one suggested I write sentences like “To access file operations, select the File menu.” When I asked for help, I got it, and when I didn’t, I was left in peace and trusted to do the right thing. That’s a great way to write.

Packt: What projects, if any, are you working on at the moment?

John: Several people have asked what the next book’s going to be. I have said, only half-jokingly, that I might do one on Chef. I have a kind of a semi-formed idea about a book of system administration patterns and practices, based on my several decades worth of experience (read: mistakes). But just now I’m enjoying a break from writing, and I’m spending my negative free time reading other people’s books, playing Beethoven on my toy piano like Schroeder out of Peanuts, and learning to bake the perfect Cornish pasty. Ah! Excuse me, that was the oven timer.

Editor’s note: John went on to write quite a few other books. None of them on Chef.

We live in a world of walls, unfortunately, and some people would like to build even more of them. Whatever you think about that, the walls between software developers and IT operations staff don’t do anybody any favours.

Looking over the wall

If you’re a developer, have you ever wondered why ops seem so antagonistic? Here’s why: they’re fed up with your buggy software that doesn’t work in production, and your apparent lack of interest in fixing it.

If you’re an ops person, how do you think you’re seen by developers? The answer is, they think you’re grumpy, unhelpful, resistant to trying new things, and unresponsive to requests for changes. Oh, and you’re a real buzzkill about security.

I’m going to tell you something now which will shock you rigid.

The fact is, those folk in the other team aren’t idiots, and they don’t hate you. They’re smart, motivated, and professional, and they’re focused on doing their jobs. But you’re not making it any easier for them. Here are some ideas on how to change that.

People who make music together cannot be enemies… at least while the music lasts.
—Paul Hindemith

Getting together

First, empathise. Understand a bit more about what your colleagues in the other team do, what they care about, and why it matters to them.

Second, collaborate. When you work closely with someone, you get a great insight into what it’s like to do their job.

Software developers, get more involved in how your stuff is deployed and run in production. “Throwing code over the wall” won’t fly any more. Your ops friends will help you get a development environment that mirrors production. You can use Vagrant boxes or cloud instances built by the same automation that builds production. No more “it works on my laptop”; when something breaks in production, you will have an identical environment to troubleshoot it.

Ops professionals, you already write and maintain software that runs your infrastructure, so make sure you’re using the same workflows and tools as your friends in development. Get them to do code reviews for you. Their entire working lives are focused on good software engineering practice; there’s a lot you can learn from them.

When there’s a new application or service to be deployed, involve the developers from day one. The software that configures the servers, installs the dependencies, and manages deployments, is as much part of the application as the source code itself.

Do pairing. The best way to collaborate with someone is to pair program with them (or pair sysadmin, depending on the task). That means you’re both sitting at the same screen and keyboard, talking about what you’re doing and working by consensus.

You might be coding, troubleshooting a problem, or anything else that’s part of your normal work: you’re just doing it together. If a disagreement comes up about what to do, talk it out or take it to a whiteboard. Involve other people if you think they can help. If you don’t have the information you need to solve something, find the person who does, and pair with them to solve it.

I am a man of fixed and unbending principles, the first of which is to be flexible at all times.
—Everett Dirksen

Re-thinking your work

New or experimental projects often need a lot of flexibility. If IT can’t offer this to developers, they’ll have to go around IT to get the job done, and that doesn’t spell collaboration.

If developers need your help to get virtual machines running, make it so; if they need the ability to spin up cloud instances to test things, make sure they have it. Re-think your priorities as an operations engineer. It might seem like answering questions, helping people, and working with developers is taking time away from your real work. Guess again. That is your real work!

Developers, you might think your job ends with a git push. But software that doesn’t work in the real world is a waste of bits. You need to understand where your code runs in production, how it gets there, how the servers are built, how the cloud provisioning works, what happens when your stuff breaks, and how to fix it.

You might think learning about Linux command lines, TCP/IP, and network latency is a waste of your time. Actually, it’s making you a better developer. If you think it’s not your job to know this stuff, you misunderstand what your job is.

The truth is there was never a neat line between dev and ops. The overlap is precisely where things get interesting. Lots of important work simply can’t be done without having a foot in both worlds, and the way to do that is for dev and ops to share their particular fu.

If deploys are fragile and often result in unplanned downtime, work on that together. Building a safe, reliable, easy-to-use deployment system is right in the centre of the Venn diagram between dev and ops. If you get that right, much else will follow.

If releases pass tests, but fail intermittently in production due to weird edge cases, you’ll need to work together to debug that. If performance is a problem, it takes dev and ops collaboration to fix it.

Show me a completely smooth operation and I’ll show you someone who’s covering mistakes. Real boats rock.
—Frank Herbert, ‘Chapterhouse: Dune’

Closing the loop

Finally, monitoring connects it all together. Monitoring tells ops that the services are up, and it tells devs how the software is performing.

Good automated monitoring checks don’t just test that a webserver is responding: they match text strings that prove it’s working; they fetch multiple URLs that exercise different parts of the system; they do queries which verify the whole stack.

If the system uses login sessions, the monitoring checks log in and behave like users: searching, filling forms, uploading content. The developers know what needs to be tested, and the ops team know how to write checks that test it. Good monitoring demands empathy and collaboration from dev and ops, and it closes the loop between those who write the software and those who run the software.

“But we already write unit tests!” Great, but monitoring is different. Unit tests demonstrate that your code works in theory. Monitoring tells you whether it’s working in practice. Unit tests only show what happens in the failure modes you can think of; the real world will throw you failure modes you won’t believe. Tests are essential, monitoring is essential, but they’re not the same.

When monitoring detects that a service is down, that alert needs to go to the person who wrote the service. This is one area where developers can be surprisingly resistant to change. Some people have got used to the idea that their responsibility ends once the code ships. But that’s not the case.

If software is breaking in production, fixing it needs to take priority over new features, and that means developers need to get that information directly: ops don’t want to spend their time nagging devs about bugs, and it creates a tension between the teams which is unnecessary and unhelpful.

A good way to get developers interested and involved in operations is to set up a highly-visible dashboard screen, showing current system status and uptime. If all is well, the board is green. When there’s a problem, that should be visible to all developers, and when there’s an outage, developers should be getting paged. It’s amazing how being on-call for your own stuff concentrates the mind… on fixing it.

Outside ideas of right doing and wrong doing there is a field. I’ll meet you there.
—Rumi

Tearing down the wall

When devs and ops collaborate, good stuff happens, so start breaking down the walls and coming out of your boxes.

• The devs learn about how to deploy, run, and monitor services at scale

• Ops learn good coding practices, the power of pair programming, and how to build software as a team

• We all learn how to be better at our jobs, how to be less defensive, and how to be more empathetic.

Don’t wait for the management memo, then. Just start today. Instead of throwing stuff over the walls, try tearing them down instead.

After many years as a professional sysadmin, my best tip for those just starting out is: take notes. Our job is basically about solving hard problems. (If they were easy, users would have solved them already.) This being so, keeping a detailed notebook of everything you do is like having an external brain pack:

• You don’t have to remember or rediscover how you solved that bizarre DNS problem back at your last gig; it’s in the book

• You don’t have to reread the man page for half an hour to remind yourself how to self-sign an SSL certificate; it’s in the book

• And you don’t have to waste time wondering where you got to on the current project at the end of last week; it’s in the book

How to keep a notebook

The sysadmin notebook is one place technology won’t help you. A simple text file is best; you can check it into version control, and access it everywhere. It’s searchable, and if you organise your entries by date, you can find stuff that way as well.

I don’t recommend using a paper notebook, as one of the most useful functions of my notebook is to copy and paste information into it directly from the console session, or from web pages, and then to copy that information out again to save retyping. If you need your notebook with you when you’re on the move, try keeping the text file in your Dropbox and accessing it from your smartphone that way. Alternatively, if you keep it as a Simplenote document, you can sync it with Simplenote on your mobile device.

What to put in it

Working scientists keep a lab notebook in which they write down everything they did, and what the results were. Sysadmin work is also about careful, scientific, detailed experimentation, especially when troubleshooting. When there are many combinations of things to try, you need to document which ones you’ve already tried, and what happened. You can copy and paste console output directly into your notebook for later reference.

One handy tip is to record your console sessions with the script command, for later reference. For example:

$ script session.txt
Script started, output file is session.txt
bash-3.2$ echo Hello, world
Hello, world
bash-3.2$ exit
^F
Script done, output file is session.txt

$ cat session.txt
Script started on Mon Sep 27 17:30:40 2010
bash-3.2$ echo Hello, world
Hello, world
bash-3.2$ exit
^F
Script done on Mon Sep 27 17:30:52 2010

Once the job is completed, you can use the session file as a reference for writing up your notes, including copying and pasting commands and output where necessary.

A record of what you did

If you’re a freelancer, it’s important to be able to justify the time you’ve spent on a client project, and to be able to account for what you did and why. If you’re on salary, it’s even more important to be able to communicate regularly and accurately with the rest of the organisation, explaining what you do, and letting management know how valuable you are. The notebook is ideal for ‘weekly achievements’ meetings and emails, and a goldmine when it comes to annual review time.

A valuable reference

The nature of our work tends to involve complicated, lengthy processes requiring accurate command-line input and configuration settings, which take time to get right. This is where the notebook really scores: you need never rely on memory again. Having gone through the process of trial and error to get a system working, you can go back through your notes and distil them into a ‘how to do it’ document which will help to share knowledge with your co-workers, and form part of your essential procedure documentation.

A project management tool

You can also use it to store ‘things to do’ tasks and other reminders (keeping those at the bottom ensures that you see them every time you write a new entry). When you meet with clients or management to discuss project status, your notebook is an essential reference for what you’ve done, problems and roadblocks you’ve encountered, questions that need answering, and tasks that still remain to be done.

A communication tool

Wherever two or more sysadmins are gathered, confusion and problems can result from them all working on the same system - either at the same time, from remote locations, or consecutively, in a shift system. A common document where everybody details what they did and when, can be very useful. When something goes wrong it’s essential to be able to trace through all the changes that were made recently, and find out who or what broke the system.

Anyone who works in a team (and we all do, whether we realise it or not) needs a way to share information with their colleagues about what’s happening, and a team notebook is an excellent way to do this.

Examples

Here’s a couple of illustrative entries from my own notebook:

# 2009-10-09 Friday

Got Cribbage Corner site migrated from HostMonster and ran upgrade
script OK, but now having some trouble delegating the DNS from HM to
DynDNS.

Need to accept mail on peterman for help@cribbagecorner.com so trying
out sendmail virtusertable as detailed here:

    https://www.rushworth.us/lisa/?p=1526

This works fine so email to CC now works - everything is ready to
delegate the domain, but HostMonster doesn't accept the DynDNS
nameservers. Possibly it just hasn't been loaded yet on DynDNS so will
try again later.

Now fixed - the right sequence was:

    Set up the domain on DynDNS and 'pre-activate' it ('Force
    activation') 
    
    Wait for DynDNS to load the records:

	$ host cribbagecorner.com 204.13.249.76 
        Using domain server:
	Name: 204.13.249.76 
        Address: 204.13.249.76#53 
        Aliases:

	cribbagecorner.com has address 94.76.216.167
	cribbagecorner.com mail is handled by 10
	peterman.bitfieldconsulting.com.

    Then go to HostMonster and enter the nameservers:

	ns1.mydyndns.org ns2.mydyndns.org

    It didn't accept them as IP addresses which is why it kept saying
    'Invalid nameserver'. Dumb or what?

I found when the DNS was delegated over, accessing the main page as a
normal user gave 'Access denied' - though subpages worked OK. Found
this page:

    http://drupal.org/node/492734

which suggests going to Site Configuration / Performance and clicking
'Clear cached data'. This worked!

# 2010-09-08 Wednesday

Working on PHP build. Downloaded PHP-FPM:

  [link]  
Generated patch:

  php-fpm-0.6-5.3.0/generate-fpm-patch
  
Downloaded PHP:

  [link]
  
Patched:

  patch -p1 < ../fpm.patch
  
Prepared for Debian build:

  dh_make -e sysadmin@example.com -f ../php-5.3.0.tar.gz
  
Edited control file and added custom configure line to debian/rules:

  override_dh_auto_configure:
          ./buildconf --force
          ./configure --with-fpm \
            --with-libevent=shared,/usr/local/lib/libevent.so \
            --with-mcrypt \
            --with-zlib \
            --enable-mbstring \
            --with-openssl \
            --with-mysql=/usr/bin/mysql_config \
            --with-mysqli=mysqlnd \
            --with-pdo-mysql=mysqlnd \
            --with-mysql-sock \
            --without-sqlite \
            --with-curl

per

  http://www.debian.org/doc/maint-guide/ch-dreq.en.html
  
Then:

  dpkg-buildpackage -b -uc -rfakeroot

In general, whenever I have to expend brainpower to solve a problem, I’ll consider it worth recording the solution (and the problem) in the notebook. Similarly, if there’s a command line that took some trial and error to construct, it’s worth a couple of seconds to copy and paste it into the notebook.

Start today

It doesn’t matter if you haven’t been keeping notes up to now and think it’s too late to start. It’s not. Just open a new text file, put the date, and type in what you did today.