We spent a couple of days looking in detail at the application architecture and found two fundamental problems:
1. The designers had not factored in that the broker cluster might be unreachable for an extended period and would have to deal with inbound traffic somehow in those situations.
2. They were using the librdkafka client library (used in C, C++, .Net, and many other language environments) in a way that it wasn’t designed to be used. There were gaps in handling errors because corner cases weren’t fully considered, and the application was inadvertently causing feedback loops that worsened the data backlog.

Yesterday, a post of mine was published on the Confluent blog that covers the intricacies of how this library works and how it should be used – “A Deep Dive Into Sending With librdkafka” .

This is the second in a two-part series along with “How to Survive an Apache Kafka Outage” informed by this incident. Taken together, the two posts are an essential read for anyone working with high-performance systems with limited control of their inbound data rate that must deal with Kafka service interruptions.

Many thanks to Magnus Edenhill for helping me understand and articulate the behaviour of this library.

If you’re interested in the Byzantine failure modes of distributed systems, I highly recommend reading “The network is reliable” – it will make you question your professional life choices.

Finally, we’re moving away from the body of the shuttle and onto the real meat and potatoes of the build – the engines. Lots of mechanical components here.

The engine installation is interesting to build – the panel that they are installed on clips onto a bar on the base of the body and moves slightly backward and forwards. The top of the three main engines is able to be rotated – it is mounted onto a long axle that connects to a rotatable unit behind the engine plate with two ball and socket beams that connect to horizontal bars that move the ailerons of the shuttle.

The effect is that pushing the left aileron mounts down results in the right aileron mounts lifting and the top of the three engines rotating. This sort of left-right coordination is how an aircraft turns on its longitudinal axis.

The idea in this set is that the turning effect is instigated by rotating the top engine. Very cool. Looking at the model down the length of the body shows the exposed mechanism in action.

We now have the main engines installed!

This post is part of a series. You can read about the backstory behind this pet project of mine and navigate all the posts via an index on Lego Space Shuttle Mega-Build.

This bag is pretty straightforward – lots of top panels for the wings, no structural components to speak of.

Just about every piece in the bag is a flat white plate. We lay out the panels across the top of the wings, and the base of what is going to be the cargo bay. The finished product looks good!

The only structural parts of the body that are left exposed from the top are the rear landing gears.

Pretty happy – it’s all starting to look closer to the finished product.

This post is part of a series. You can read about the backstory behind this pet project of mine and navigate all the posts via an index on Lego Space Shuttle Mega-Build.

Welcome to bag 8! This one’s all about internal details and the flap that is going to sit under the engines.

First stage of construction is assembling a body-length unit that’s going to sit over the smooth plate section in the shuttle body, where the black-and-white checked section is visible.

The engine protector flap is attached with a couple of grippy pieces that allow it to pivot vertically. Once installed in place, the whole unit slides back and forward by about a centimeter, constrained by a gap in the bricks within the middle of the shuttle.

Couple of non-standard pieces are used to build up the front landing gear.

Once mounted, the landing gear is compressed into place. This is the first real stuff-up that I’ve made. Something’s one stud off, and of course this entire unit involves a rebuild of the nose section.

A few minutes and a bit of cursing later, the landing gear is mounted and spring-loaded! Now it all makes sense – the landing gear going up pushes the whole mid-section piece backwards, which extends the flap beyond where the engine will be!

That whole midsection isn’t attached to anything – it’s free-floating.

The final result, for this bag. 9 bags to go!

This post is part of a series. You can read about the backstory behind this pet project of mine and navigate all the posts via an index on Lego Space Shuttle Mega-Build.

Welcome back to the Lego Shuttle mega-thread! We’re onto bag 7/17! Three sub-bags in this one – going to be working on the body and wings today.

Straight out of the gates, we’ve got an unusual hinge piece, and immediately we fix it into place so it won’t pivot. #headScratch

In the context of the front of what looks like the shuttle’s leading edge, it provides a flat protrusion. A couple of pivoting pieces at the back. They’re sandwiched into place.

Clipping the edge onto the shuttle, you can see why these pieces are necessary – there’s no other way to fix stuff at that angle. The protrusion sits on top of a flat panel, preventing downward motion. Clever!

Attached view.

Build a mirror section, and starting to look good!

Those protrusions get fixed into place, as do the pivots. Now we’re solidly fixed. Again I’m left marveling at the ingenuity of the @Lego designers.

A couple of small blocks end up going upside down to fill a small right-angled gap in the wing.

A highly specific angled piece gets sandwiched into the center of a wing’s leading edge. Attaches perfectly. Genius.

Put together its mirror section, attach it, and my 5yo copilot and I are 100 build steps in!

Click in some flat panels and some small curve units, and this bag is done!

The next stage is going to be the front landing gear, bulking up the body and the rear section that protects the engines.

If you’re on a Zoom call with me, you’ll see the work in progress on my bookshelf. Also, some cow pics.

This post is part of a series. You can read about the backstory behind this pet project of mine and navigate all the posts via an index on Lego Space Shuttle Mega-Build.

After 3 weeks, welcome back to bag 6/17. My co-pilot and I are working on the front part of the undercarriage.

More non-standard pieces in this part of the build. Particularly interesting is the vent-like block that goes in the nose section. Looking forward to seeing the logic behind it.

Making the connection between the front and back sections. Combination of pegs and long overlays is used here.

The result is impressively strong under strain and torsion. The black non-standard L-blocks make a lot more sense now in context. Engineering in the small!

The final result looks good! Had to do my first bit of debugging around the nose section, but ready to go for the next stage!

The underside looks implausibly flat for a Lego build that’s so strong.

How strong? This strong.

Arm for length comparison. Seriously impressive.

This post is part of a series. You can read about the backstory behind this pet project of mine and navigate all the posts via an index on Lego Space Shuttle Mega-Build.

It’s the weekend, so it’s Lego time again! Bag 5/17, and we’re starting on the Space Shuttle proper! Wings and undercarriage.

My co-pilot is appropriately themed.

Loads of interesting pieces to construct the landing gear. Solid as a rock once assembled. Hats off to Lego’s design engineers.

As usual, lots of non-typical blocks here. All with a structural function.

Looking forward to seeing how the shock absorbers integrate later on in the build.

An hour and a bit later, we’re done! You can see the beginnings of a pivoting mechanism.

The undercarriage drops down when raised, but not connected to anything yet. Pegs prepared to connect the front half of the shuttle. I’m impressed with how solid the construction is, given the number of small pieces.

This post is part of a series. You can read about the backstory behind this pet project of mine and navigate all the posts via an index on Lego Space Shuttle Mega-Build.

Welcome back for bag 4/17. This weekend we’re building the stand for the Space Shuttle. Bag in a bag. Looking at the final result, it’s hard to believe it’s made up of all these tiny pieces.

Some of the more unusual pieces. The toothy one is a support beam. Again blown away by not just the precision at scale but by the application of engineering principles in a model of this size.

And after 2 hours, we’re done! This little stand not only pokes out on an angle but is rock solid and bears a considerable amount of weight. So it should, given the Shuttle will be over 50cm in length.

This post is part of a series. You can read about the backstory behind this pet project of mine and navigate all the posts via an index on Lego Space Shuttle Mega-Build.

And we’re back! On to bag 3/17

Lots of small fiddly pieces in this one

2 hours later, the Hubble Space Telescope is done!

3 bags of Lego. 6 hours. The next part is the Space Shuttle – 14 bags! My little helper and I will finish around Christmas

This post is part of a series. You can read about the backstory behind this pet project of mine and navigate all the posts via an index on Lego Space Shuttle Mega-Build.

Bag 2 is the body of the Hubble. It’s made up of 3 sub-bags. So many pieces.

The cool thing here is how Lego has colour-coded these interior pieces. They won’t be visible once the build is complete, but you’d be lost trying to find the right piece otherwise.

There are some really unusual pieces in this set. This one’s an L-bend on a socket.

Four of these are used in a 45-degree pivoted combination to attach a circular plate. Clever!

More unusual pieces – flat plates with male connectors underneath and some lightsaber-y posts and cylinders.

And we have the body of a space telescope!

That’s enough for today. 2/17 bags completed!

This post is part of a series. You can read about the backstory behind this pet project of mine and navigate all the posts via an index on Lego Space Shuttle Mega-Build.