geek!daily

Setting Up A Minecraft Server For Kids, Part 1: Preparation

2013-04-12T09:00:00-07:00

I've got a little more time on my hands than usual at the moment, so I decided to set up a Minecraft server for my son and his friends. I figured it would force me to play around with cloud APIs, some linux configuration that I haven't done for a long while, and other various bits of geekery that would just make me happy.

As I started to do some research on how to go about it, I found about a gajillion posts on how to set up a server (including this slightly outdated gem), including a few from smart friends of mine. After a bit of reading, it became clear that this could be a much larger undertaking so I decided to chunk out the project into four steps:

Preparation: research and learning
Installation: set up the instance and server
Configuration: tweak the server config and add plugins to get what you're looking for
Productionization: make the installation reproducible and automatable; set up backups; etc. And yeah, that's not a real word. I'm okay with that.

This post will focus on the preparation required to get as many bits sorted out as you can before installing. I'll write about the rest in the next few posts.

So, Minecraft ...

If, like me, you came to this with little practical Minecraft experience, the first thing you should do is try playing a bit. There are a few free ways to do this — such as Minecraft Classic in your browser or Minecraft Pocket Edition Lite on an iOS device — or you can just spend the ~$30 and jump in. I figured that to be an op I'd want my own account, so I went for the full experience.

I died a lot. I crafted a bit. I created a bunch in single player. I joined a few multiplayer servers. I died some more. Through all of it, I got a sense of the game and what my son and his friends would be in for. This helped a lot as I started to figure out what it would take to start up and run a server, as it identified some key questions, like, "How can we have a creative world as well as a survival world?" and "What's the difference between 'easy' and 'peaceful'?"

These led me to the Minecraft Wiki. Frankly, had I read much of it before playing I would have died less and crafted more. However, it helped me get clear on a few things that matter when you're considering running a server for others.

Game Mode

The default is survival mode, which means that, in order to survive and craft things, you have to walk around and collect block types required by recipes while avoiding being killed by monsters ("mobs" in the old MUD parlance). There's also creative mode, where you gain the power of flight, you have access to every block type instantly, and you can make anything your heart desires. My son is currently constructing a flush toilet in the hotel he's making. His heart desires strange things.

Difficulty

This ranges from Peaceful, where any aggressive mobs are removed or don't spawn, through Easy and Normal, right into Hard. The wiki does an excellent job explaining.

World Types

There are a few settings in there that weren't obvious on the face of it, such as what Large Biomes is or why Superflat would be interesting. It turns out that Superflat is a great canvas for Creative worlds, and Large Biomes is pretty much just like the default biome-based terrain generator but each biome takes up more real estate.

It was clear to me that we'd want a Superflat Creative world and an Easy Default Survival world.

Servers, Servers Everywhere

There are a lot of available servers out there. I could tell you that I did an exhaustive search and evaluation, but the truth is that I trusted my smart friend, Sarah, and chose Craftbukkit (CB) for many of the same reasons she did: we both know and trust Tom Enebo's opinion, it's easy to extend using JRuby, and it seems to be the leading server beyond vanilla Minecraft.

Another bit of validation comes from cheracc, who runs Sandlot Minecraft server for kids and families, and who also uses CB with many plugins. More on that later, though you can read his excellent Reddit post on setting up a Minecraft server for kids now.

One other minor note: since CB is a third-party server, it takes them a bit of time to catch up when Mojang releases a new client version. This means you'll want to wait on client updates until there's a beta version of CB that handles the new client version. As of this writing, the current client requires a 1.5.x server which means craftbukkit-beta.jar

Does Size Matter?

Finally, I wanted to figure out how big an instance I'd need to host a decent server. I flailed my way through several posts trying to figure this out until I discovered Can I Host A Minecraft Server, which makes it easy to take your bandwidth (both up and down) and the available RAM and figure out about how many people you can host. Remember to use the bandwidth figures for the cloud provider you're planning to use. For me, it turns out I can handle ~12 players in a 1GB instance with the usual cloud host network speeds of >5Mbps, which is what I was hoping to cover.

So that's it for prep. Now it's time to instantiate and install.

Bashing tcsh

2012-03-26T22:45:04-07:00

After 15 years of non-stop tcsh action, I've finally made the jump to bash. It was painful and I miss all my lovely variable expansion (^X $), wildcard expansion (^X*) and term completion (ESC-/), but there are so many tools like rbenv that expect that you're using bash with no tcsh-friendly alternatives.

On the upside, I've got functions now. I guess that's something.

Oh, and hi again.

Advanced YAML: Tricking Out Your database.yml

2010-08-13T16:03:33-07:00

I've been cultishly carrying this snippet around for years:

login: &login
 adapter: mysql
 host: localhost
 username: myuser
 password: mypass
 encoding: utf8

For a couple of reasons around running DB migrations in differently configured environments, I wanted to factor this into bits common to all environments and bits particular to the config on the local machine. I came up with something like:

# Provide default local block
local: &local
 socket: /var/run/mysqld/mysqld.sock
 
common: &common
 adapter: mysql
 encoding: utf8
 reconnect: false
 pool: 5
 username: myuser
 password: mypass
 <<: *local

development:
 database: myproj_test
 <<: *common

This works swimmingly, but I was tired of ritually mimicking and extending the original snippet without actually understanding it. I decided to dig deeper into what was actually going on and found, unsurprisingly, that I stand on the shoulders of giants like Doug Alcorn, James Duncan Davidson, and Ben Bleything. To render proper credit, Ben's post from June 2006 builds on a missing post from James's blog, both of which were preceded by Doug's observations in March 2006, and all of which draw on tricks originally found in the Typo database config. None of them seems to have detailed what's going on with the &login/*login construct, so I thought I'd dive into that a bit for posterity.

YAML is nothing more than a human-readable way to serialize basic data structures like lists and associative arrays (aka hashes), which makes it ideal to represent the configuration values needed by Rails. It turns out that YAML provides anchors (&), references (*), and associative array merges (<<), all of which allow you to include by reference either by assignment or by merging a referenced array into another array.

Here's an example of each:

# Generate a reference
mammal: &mammal_ref
 warm_blooded: true
 lays_eggs: false

# Define via reference assignment
beaver: *mammal_ref

# Define including a hash merge
otter:
 cute: true
 <<: *mammal_ref

# Define including a hash merge, overriding a value in the reference
platypus:
 <<: *mammal_ref
 lays_eggs: true

I found that the Wikipedia entry on YAML was a good, quick overview; the official YAML spec has all that and more, but it's not a quick read. Meanwhile, both Doug and Ben observed that bringing ERB into the mix let's you seriously customize things. I took a bit from both of them; here's my database.yml now:

# Provide default local block
local: &local
 username: myuser
 password: mypass
<% if File.exist? "/opt/local/var/run/mysql5/mysqld.sock" %>
 socket: /opt/local/var/run/mysql5/mysqld.sock
<% elsif File.exist? "/var/run/mysqld/mysqld.sock" %>
 socket: /var/run/mysqld/mysqld.sock
<% elsif File.exist? "/tmp/mysql.sock" %>
 socket: /tmp/mysql.sock
<% end %>
 
# Allow for local DB configuration
<%= File.read(File.join(File.dirname(__FILE__), 'dblocal.yml')) if File.exist?(File.join(File.dirname(__FILE__), 'dblocal.yml')) %>

common: &common
 adapter: mysql
 encoding: utf8
 reconnect: false
 pool: 5
 <<: *local

development:
 database: myproj_development
 <<: *common

test:
 database: myproj_test
 <<: *common

production:
 database: myproj_production
 <<: *common

Works a treat. Thanks, Doug, James, and Ben.

Live Blog: Comet & Other Browser Stuff

2009-10-15T13:27:25-07:00

It's a big component of what we call "real-time"

This is the kind of stuff that puts up a banner on a Twitter search ("# New Tweets Found") and makes things feel real time.

List of what's being done to move notifications from server to browser:
* Ajax polling: Anybody using the JS set-timeout function, then refetch. Used by GMail, Campfire, et al
* Long polling (typically Comet): Used by FriendFeed, keeps an http connection open as long as possible (keep alive), reopen when timed-out. Keep a local thread pool.
* Flash sockets: Same as long polling, but from Flash. Don't have the same-origin policy. No limitaions on number of connections. Does gzipping.
* Reverse HTTP: hosting a little webserver inside a JS connection. Long pushing?
* Silverlight ("MS's version of Flash")
* HTML5 websockets: part of spec, doesn't exist and not implemented. Expected to be like flash sockets.

(things discarded as too old/obscure/painful/absurd):
* Java applets (David Weekly is about to get himself kicked ;)
* Big ugly JS kludges (iframe tricks, etc)
* What's that Opera thing ... unite? ubiquity? Unite. Every browser is a webserver. Sounds like reverse HTTP.
* Using XMPP format, arbitrary JSON structure, Atom.

Libraries:
* Orbited (?) tunnels TCP thru HTTP, treats each end like a socket. JS front-end, Python backend. Often paired with Twisted. Specifically meant to be Comet.
* stropheJS: javascript, can use flash sockets
* Tornado: python
* Cometd: Java, made by Dojo to work with dojo
* Dojo: javascript, can use flash sockets
* APE project (?)

What formats are people using to send data:
* JSON
* XMPP

Apps and their libs/methodologies:
* Meebo: ??
* Google Wave/GTalk: GWT RPC, long-polling, their own JS
* FB Chat: ??
* FriendFeed: Tornado server with long-polling, their own JS
* Superfeedr: uses BOSH, stropheJS. BOSH is kinda long polling, bidirectional (two cs open all time). Very similar to comet, more friendly than strict proxies
* Collecta: uses BOSH
* Twingly: orbited
* PBWorks: long polling on network dashboard to see updates come in live; wrote their own libraries, also use stropheJS
* StatusNet - identi.ca and ??, orbited and cometd (you can do either)

Flex and Air apps? The most interesting stuff is what used to be called Flash Media Server (been renamed). They've got their own P2P protocol. All sorts of funky stuff you can do. OpenSource version of FMS is Red Five.

In what situations are these libraries breaking down?
* Transparent/Opaque proxies that give repeats/dupes/hangs. Keep buffer on server and check for these. Sometimes have to close connection to flush thru proxies.
* "Everyone focuses on the newest hottest event-based framework, but the hardest part is that HTTP wasn't designed for long-polling."

Have to hold open request/response pairs. Connection setup and teardown is expensive

Guy at UK telecom has only ~3K IP connections available in London area. Comet is going to force an upgrade of their hardware, which will be expensive. Real world constraints will always present.

Real-time at UI can be distracting at best, horribly annoying at worst. Charts work great, but text moving too quickly becomes hard. UX is challenging. Would be nice to add items while autoscrolling relative to the focussed item

When you want to add a pause button to your site, that's FAIL. But the hover-over-conversation to pause semantic is pretty good.

Go to jschat.com to see a bad resize.

FB pioneered notifications really well in UX. You get a "toaster popup" that doesn't disturb your screen and it fades after a few moments plus a bar at the bottom to persist the aggregated notifications/count. Really nice.

Growl is also mentioned as a good model. Is there interest in a notification aggregator with contexts in the browser? Meebo wants to go that way as a notification aggregator. Adobe wants to do this on your desktop. BrowserPlus also hits Growl. Prowl == Pushed Growl.

If everyone started using XMPP, it would increase the message load ("three bazillion individual messages"). No one is bundling XMPP updates; no reason you couldn't. ActivityStreams could also serve.

Why are people using Atom/JSON for this? They're so extensible, but when you extend them so far they're just as verbose as XML. "But we just don't like XML."

Would be nice if you could select things to pull out of the stream and hold onto.

Seems to be consensus that chat belongs at the bottom of the browser, append at end. Everything with a permalink (blogs, tweets, etc) gets added to the top (?). Seems to be related to height issues.

Infinite scroll vs. "More" button

Titlebar flash (Gmail and GTalk)

Some sites make a tiny flash widget to play a sound

Haptic: twitter dmesg to cellphone buzz

Notifications can become another form of info overload/noise

(We flailed at making a 1D/2D chart to represent frequency and value ... FAIL)

Phonetop apps are like desktop apps. Leah feels strongly that they're headed down-and-to-the-right and browser-ish apps are taking supremacy due to interoperability.

Apple's surprise: phonetop apps took off. (?) vs. John Gruber: "The most used app on my iPhone is Safari."

You can find the whiteboard pics in my Flickr stream.

Live Blog: Web Aggregation, What Works and What Doesn't

2009-10-15T12:17:59-07:00

[note: I originally scribbled this on paper thinking I could hand it off immediately, preventing the obligation of typing, posting, etc. Turns out I don't get off that lightly, so here's the spew in electrons.]

Scraping isn't a scalable model.

There are biz issues around aggregating data: many businesses don't want you to get their data, though many are becoming more open.

Doing aggregation right:
* minimize latency
* maximize engagement

When latency is high, it causes confusion and takes you out of real-time

Doing conditional gets can be somewhat useful.

Plaxo had to shard their crawlers, which lands you in the shared state/sync problem of any stateful system you want to scale horizontally.

Gnip integration has been good:
* Offload the long-running processes
* Gnip offers alerting or "fat ping" (ping includes update data)

Plaxo likes using the alert to escalate the priority of the crawler which fetches the rich data related to the update. This approach allows you to use a consistent model for content ingestion vs. get info from fat ping, then augment later.

Smarr: "Brad Fitzpatrick said, 'Make polling a special case of push.'" He attributed this to someone but I missed the attribution.

(Don't try to keep up with Joseph Smarr on paper. He's thinks too many cogent thoughts too quickly to preserve legibility)

Plaxo uses TripIt's RSS feed as alerting, grabs item ID, then uses their APIs to fetch rich data.

There's a move to homogenize the info from sites, which may not be a good idea. It suppresses the distinctive look and feel/experience of the publishing site. Allowing for these differences means more labor spent on making one-off shims, which increases maintenance. Still, right choice in order to provide value to the user.

Activity streams seek to provide more rich data in a somewhat normalized, extensible format.

Many/most sites aren't yet perfectly architected for real-time's push, ping, etc.

PubSubHubBub and Activity Streams are externally represented data shards

Plaxo's Pulse started with known architecture issues (in order to ship) and hit the wall sooner than expected. Threw hardware/software optimizations at the problem to move the wall far enough to give time for rearchitecture, sharding, and working out how to propagate changes throughout the system properly.

None of the NoSQL alternatives are quite ready for prime-time. Smarr: "It should be something that's just a primitive."

Conversation platforms are slightly different sorts of aggregation platforms. There are UI diffs (e.g. pause the stream when indicating interest). Handling the transition from slightly-latent/passive real-time to synchronous real-time/active not yet well-developed (think: when a comment inspires a conversation)

90-99% of the value of the real-time web is realized in not-real-time [unreal-time? ;] This is a big deal for discovery. Twitter and FB make this harder by obscuring history.

Ideal scalability/performance would be an index per user. This would be grossly inefficient due to the number of duplicate entries.

No one has nailed reader-controlled aggregation (Show me Joe's tweets and blogs but not his photos) quite yet.

Smarr: "If we're all kinda [sharing], we're all making each other smarter"

The firehose of info is a hard model to scale to. Ben Metcalfe proposes the garden hose -- a firehose filtered at the source according to your interests, which helps aggregators by allowing them to request the superset of all filters from a given publisher.

We really want to push contexts to the publishers and let them determine which content fits that context. Context shifts over time: Joe doesn't normally read my tweets (and why would he?) but when we're at a conference together, he's much more interested (thus the popularity of hashtags). This is a geographic and purpose-driven context (the conference) as well as Joe's context on me (Jim knows where the good bars are).

Folks like Twitter are so overloaded with info that they might not recognize non-immediate contexts that are interesting to me.

There's also the risk of exposing users to the amount of correlatable public data they have. Many don't want you to apply a transitive closure to identify them in all spaces even though doing so allows you to present a much more convenient UX around what they want you to aggregate.

Someone likened the real-time aggregation problem to a bar conversation: you get snippets here and there and follow your own thread of interestingness.

Three fundamental themes:
* How to specify contexts to data provider/publisher
* How to control access to private data (and carry ACLs with that data)
* How to do all this efficiently

Plaxo implemented polling-back-off (poll infrequently updated sources less frequently). Turns out this is a bad idea, as it introduces latency which makes it feel broken.

There's also the issue of aggregating conversation about web objecs (like blog posts) and how not to divert the conversation from the publisher's site. However, sometimes you want a private discussion of a public object (cf. LinkedIn company groups discussing an article)

Q: What's the state of open standards around this?
A: PubSubHubBub and Activity Streams are very exciting. OAuth as access delegation. There's still a lot of ground to cover.

Stepping Into BDD with Cucumber and WebRAT: Structuring the Work

2009-09-13T14:45:32-07:00

This is the first of a series on my adventures in going BDD with Cucumber and WebRAT with a greenfield project. Watch for the next article coming soon.

I'm working on a tiny project for the dads club at my son's school and using the opportunity to plunge fully into BDD with Cucumber. I've been tinkering with it a bit here and there, but this time I'm committing to fully specifying my acceptance criteria before I start writing specs for my implementation. I have to admit it feels a bit onerous right now, but I can see the value coming so I'm sticking to it.

My usual working methodology is:

Write a quick set of use cases
Infer features from the use cases
Divide the features into logical delivery phases
Start spec'ing the implementation
Code to the specs
Profit!

... so really I'm just adding a "Specify features using Cucumber" step between 3 and 4. Sounds small (but it's not).

The use cases for this project are pretty easy:


Use Cases:

NON-MEMBERS:
* Join

MEMBERS:
* See news postings
* See calendar events
* Subscribe to a calendar feed
* Read and send email to the group
* Maintain a member profile (email, phone, location, kid info, etc.)
* Browse and search a member directory (by grade, class, proximity/map, etc.)

PREZ FOR LIFE:
* Write news postings
* Post new events
* Send news/reminders to various publishers (group mailing list, school newsletter editor, etc.)

(Yeah, we have a President For Life. He's awesome.)

Time to break these into features. I try to scope each delivery to be a useful increase in features and something I can delivery quickly (I don't get much geek time these days ;). Some of this is old hat (it's mostly just a CMS), but some of it calls for me to learn about things I've never played with (Google Maps APIs and Geolocation, for example). Additionally, it was important to get something up fast last week so the main URL could be included in some printed materials for back to school week.

Here's the feature rough divided into phases:


Phase 0: Site
Non-Members can see front page

Phase 1: Members
Non-Members can join
Members can sign in with email and password or OpenID
PFL can manage news items
PFL can manage events
PFL can manage pages
PFL can suspend accounts of non-PFLs
Members can view news and events
News snippets and event titles show on top page

Phase 2: Admins
Members can be made admin by PFL
Admins can manage news items
Admins can manage events
Admins can manage pages

Phase 3: Feeds and Mailings
Members can subscribe to news and events feeds
PFL can set reminders to be sent by email
Members can recover passwords by email

Phase 4: Profiles
Members can add email, phone, address, availability, notes, profile pic
Members can add kids: name, age, picture, teacher
Members can mark info private, admin-only, or open

Phase 5: Directory
Members can browse directory info

Phase 6: Maps
Members can see people, events on map

Phase 0 was "get something up fast;" we made it a simple redirect to the existing Google group. Phase 1 is the biggest chunk of features, but they're very straightforward. Each phase after that introduces enough new functionality to be interesting for the members, but in small enough chunks that I should be able to knock 'em out in a weekend, plus-or-minus.

Okay, so now I'm ready to generate the app and cucumber-ify it. I'll cover that in the next article.

Generate the regex for a TLD hostname from Perl

2009-08-18T19:05:19-07:00

This was a quick, fun exercise to remind me that I can still write Perl. It fetches the list of TLDs from IANA, does a quick bit of munging, then renders a regex which should match any valid FQDN:


#!/usr/bin/env perl
use strict;
use warnings;

use LWP::Simple;

my $fqdn_regex;

if (my $content = get('http://data.iana.org/TLD/tlds-alpha-by-domain.txt')) {
  $fqdn_regex = '(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+(?:';
  $fqdn_regex .= join('|', grep (!/^(#|xn)/i, (split /\n/, lc($content))));
  $fqdn_regex .= ')';
}

my $regex = $fqdn_regex . '(?:\s|\/|$)';
print "$regex\n";

Several caveats:

It doesn't match IPv4 dotted quad nor IPv6 ::-notation
It intentionally ignores Internationalized Domain Name in Applications (IDNA) domains
It borrows from my favorite reference for this, regular-expressions.info's page on email address regexes.

Maybe I'll extend it for completeness and/or rewrite it in Ruby someday. Until then, it'll always be ~/bin/tld_regex for me.

Gmail no longer beta? True dat.

2009-07-07T10:12:31-07:00

I guess it's no longer under construction, or so sayeth the GMail blog which also shows those who can't live without that ubiquitous beta tag how to bring back their old friend:

Michael Jackson's Obit via Wordle

2009-06-25T18:40:55-07:00

Story by Yahoo!, picture by Wordle.net:

LiveBlog: 10+ Deploys A Day: Dev and Ops at Flickr

2009-06-23T13:47:46-07:00

Update 1: Slides are now on SlideShare.

Update 2: Video now available on blip.tv

John Allspaw (Ops) & Paul Hammond (Eng), Twitter

“Actually work together and aren’t huge assholes to each other.”

(omitted: photo stats … that’s a lot of kittens)

Dev vs. Ops

It’s not my {machines,code} it’s your {code,machines}

Spock v. Scottie analogy

Ops as grumpy old man, says no all the time, cycle of “no all the time because no one tells them anything because they say no all the time”

CW: dev job to add features, ops job to keep site stable and fast

Flickr: Ops job is to enable the business (Dev’s, too)

Business requires change, otherwise you’ll be overtaken by the new guy … but change is the root cause of most outages.

Discourage change vs. Allow it to happen as often as it needs to (via tools and culture)

Lowering the risk of change via tools and culture.

Increase confidence in change goodness
Increase ability to react to those changes

You need {ops,devs} who think like {dev,ops}

1. Role and Config Mgmt

2. Shared Version Control: everyone looks in the same place for everything

Code and config in same place
Everyone has access—transparency
Everyone knows how to use it

3a. One-step build

Everything you need to do to convert svn co’d code into what goes to the site—one command
They have “Perform Staging” button at bottom of a page with stats on latest commit

3b. One-step deploy

Top of page is deploy log with notes: who, when, what (link to changes)
Bottom has “I’m feeling lucky!” button to deploy
Continuous deployment

You can’t pretend to deploy 10 times a day if you go down 10 times a day. That’s not being agile, that’s being retarded.

They use Hudson to generate packages which can be deployed by ops

Small frequent changes make it easier to see what went wrong and recover when needed

4. [missed that tag]

Branching is all about managing bugfixes
Always ship trunk
Branch in code instead: use conditionals to block out pre-release features and configure off/invis—provides an operational lever for adjustment
Makes these open for private beta in production on production hdwe, etc.
Allows dark launches, which allows you to size appropriately, fix major oversights, take the fear out of major new launches

They have a couple hundred “free contingency switches” to turn things off/throttle things down. Gives broad operational control to minimize effects on the site.

Tend to fail forward using these and fix the problem.

5. Shared metrics

You can see mine, I can see yours
Use ganglia as console
Devs know where dashboard is, and watch as obsessively as ops
Includes app-level metrics (which exposes them to Ops)
(helps drive accountability in both directions in the org—both can see and feel ownership)
This begins to create opportunity to gracefully collaborate to back off an oversub’d resource/degrade/throttle as needed
Show last site deploy info on every page/graph; you can corellate a change in the graph

6. IRC and IM bots

Heavy IRC users for ongoing dialog between dev/ops, remote/local
Last.fm wrote a tool to inject events into IRC (monitoring, events, deployments, builds)
Log it all and put it in a search engine

Culture

All the tools in the world won’t help if you have a contentious culture

1. Respect

No stereotypes: not all devs are lazy/cowboys, not all ops are obstructive
respect their expertise and opinions and responsibilities (they influence priorities)
Don’t just say “no”—it’s like saying “I don’t care about your problem/perspective”
Best solutions are collaborative. Memcache is an excellent example; written to solve the problem of DB overheat, which impacted both
Don’t hide things: share your solution even if you think they’ll say no; you deny their expertise and input

Talk about the impact of your code push

What metrics change
what are the risks
what are the symptoms of somethign going wrong
what are the contingencies (rollback)

2. Trust

When you walk up with all the above on hand, you demonstrate that you care enough about them and the business
“I don’t want to tell X …” == you’re a cowboy, and “cowboys are losers”
Have shared playbooks and contingency plans so all understand.
Provide as many knobs and levers as you can so Ops can tweak to match the env
Ops: be transparent, give devs insight and access to the systems. Playing telephone around shell commands is dumb.
It’s hard to help if you can’t directly see

3. Have a healthy attitude around failure—it’s going to happen.

Think about how you’ll respond more than you think about how you’ll prevent it
Would you rather be treated by a GP who deals with heart attacks infrequently or an EMT who handles them weekly?
Fire drills: when ops and sr engineers are fixing a problem, have others diagnose live in parallel (but make no changes!)

4. Avoiding blame

they have a rule of no-finger-pointing; it doesn’t need enforcement, folks step up
fixing blame wastes a ton of time, why not skip it? Feel guilty afterwards if you must.
They’ve got a bit of a potlatch culture as people try to assert responsibility in order to fix things.
Remember that when your code breaks, someone’s going to have to wake up to fix it. Own it and apologize, at least. Otherwise, you’re back to not respecting each other (“Screw you … aren’t you getting paid to do that?”)

Ops should provide constructive, continual feedback on how it’s going. Point out interesting things before they’re critical