Continue reading ‘Need’ »]]> It’s not about profit – making money does not help customers.

It’s not about cost-saving – saving money does not help customers.

It’s not about shareholders – pleasing shareholders does not help customers.

It’s not about features – delivering features does not help customers.

It’s not about testability – that something is tested does not help customers.

Satisfying a need does help customers. Amongst other things it might make their lives easier, make something possible, educate them or entertain them. It engages them, enthrals them, creates emotion within them. From all of this, many good things will come.

If you are developing a system in absence of a focus on satisfaction of needs, you’ve lost already. First question then:

Who are your customers?

And if you think the only customers are those paying for what you build, you’ve lost once again. In fact, you’ve signed your own happiness away.

Continue reading ‘On The Practice of Design’ »]]> Technology is not architecture or indeed design, it is a means for implementing a design. Various technologies (e.g. languages or frameworks) will be more or less compatible with implementing a specific design.

Design is an abstract exercise. It becomes constrained by our own choices which can include using existing technology or creating anew. By default, it should not be constrained, this is closer to the ideal. The more constraint exerted by technology the less ideal things are likely to be. Less ideal can be acceptable, there are always cost limits and such but it should never exist without consideration of the consequences.

Some argue that design cannot be an abstract exercise at all because real-world considerations demand otherwise. Performance is often cited as being too significant to ignore. Are they right?

The nature of performance in a system can be generalised into a set of guiding principles. In the case of computing, there’s a well known hierarchy driven by locality (starting with the fastest component):

1. Register-based CPU instruction
2. On CPU cache access
3. Off CPU cache access
4. Main memory access
5. I/O (network, conventional disk, SSD)

Jeff Dean and others have expressed this in a table of “Numbers every programmer should know”. The performance relationship amongst these components is sufficient guidance for design work. Clearly, incrementing a number across a network connection is something to be avoided (though there are ways to make this work). It would generate significant chatter as would naive distribution of an OO design.

So to answer the question: Performance is too important to ignore in a design but the amount of consideration required is no more than other aspects such as coupling and cohesiveness.

Apple decide what they want to build first then create and/or select the technologies they need leading to great products. They ask themselves how do we make this idea, this concept we have in mind, real? This is the point at which technology becomes relevant. NASA, when set the moon-shot challenge created the technologies they needed to deliver the end result over a period of years. They iterated on engine, flight control and many other aspects to get the ultimate embodiment. 37Signals ended up creating Rails, embodying a new way of building product to deliver their vision.

The best designs start out as concepts or ideas and are largely un-constrained by technology (there are limits of course, e.g. a phone must have certain components possessing certain properties). They retain their elegance, a sense of style and taste. Designs that are forced to fit with early, uninformed technological choices are likely to be brittle and die.

Developers have a bad habit of selecting tools and technology well ahead of consideration of a problem and potential design approaches. History is littered with examples of the consequences, balls of mud and expensive “surprise” project failures that should have been “easily dispatched” because of this or that silver-bullet technology.

There is nothing harmful in the general discussion of technology tradeoffs, it’s what leads to useful guidance such as that of Jeff Dean above. It also makes sense in one’s early career to work with a variety of technologies to help gain an understanding of tradeoffs and patterns that work or don’t. However, excessive technology fixation is destructive for quality design work.

One can certainly design from a technology driven perspective (choose your language, frameworks etc and constrain your design to fit them) but that won’t be good enough for a moonshot, a class-leading product or a high-quality solution.

Continue reading ‘200’ »]]> In aviation circles there is a thing known as the 200th hour rule. It goes something like this:

After 200 hours of flight time you are expert enough to feel confident in what you do but amateur enough to still screw up. Most worryingly it’s said that come the 200th hour you will screw up and quite possibly in grand style.

I’m figuring that applies in many other situations and there’s probably more than one 200th hour event in many cases. One would hope that pilots, should they survive the experience, learn from the mistake and improve. Can we all say we do the same?

Continue reading ‘Blueprints’ »]]> For a long time, I’ve wanted to write something about the state of our software practices. It’s always proven quite challenging as I find myself unerringly drawn towards philosophy, creativity, engineering and a myriad of other voluminous subjects. Producing something succinct has proven consistently elusive. They say you can’t force these things and so it would appear as it’s taken some writing from Leslie Lamport to help me distil out some specific points that I want to make.

The article that started this chain of events is Blueprints in which Dr Lamport discusses the practice of coding. I found it thought provoking yet judging by the comments many felt it was irrelevant, out of date or simply wrong. Reading through those comments and a tweet discussion with Nic Ferrier led me to a bunch of observations which appear below.

Foundations

It appears that the focus on “practical” aspects of systems building (e.g. knowing how to code in popular industry languages rather than the fundamentals that underpin them all) has significantly impacted the corpus of common knowledge. Specification as a practice is not well understood:

• It can be formal or informal – ultimately the end-goal determines what is appropriate. Formal specifications provide the opportunity for proof and verification which in critical systems is highly desirable. The relevance goes beyond critical systems though to any situation where high confidence in a piece of behaviour is required.
• Specification in its various forms isn’t a theoretical exercise – there are a number of examples of its application in real systems. Google mention it in a variety of circumstances including Chubby and Spanner.
• Proving correctness can be done via formal specification, it cannot be achieved by testing. Imagine standing in a dark room with a pencil beam torch trying to establish what’s in the room, dimensions etc. This will take a long time and things are easily missed unless you cover the entire room with that pencil beam (which will take forever). Formal specs allow you to simply turn the light on in the room.
• BDD, TDD and the like are testing processes and thus cannot be directly compared with specification (certainly not the formal variety) which is a tool.

Abstraction

The ability to deal in abstraction is important for the disciplines of architecture and design. However there is a more fundamental need to satisfy when coding, the limits of the individual mind to retain and reason about detail. Our only tool for coping with systems of detail larger than can be held in an individual mind is abstraction. Abstraction creates coarser constructs that hide some of the detail and allow us to scale our reasoning to broader levels. It also makes it possible to communicate and test our reasoning with others.

Reading the responses to Lamport’s article shows that some of us are too literal in our interpretation not pausing to consider the more abstract possibilities. We are unbalanced in our view, focused on detail and specifics in a world where grey and uncertainty (a natural consequence of dropping detail for sake of abstraction) play a critical role. Some examples:

• Software systems have nothing in common with buildings. Consider for a moment the challenge of changing an old or large system to cope with a radical new requirement say going from single machine to massively distributed. Is that so different from taking an old Victorian-age school and putting in the trunking and cabling required for modern systems development? In both cases, there will be a desire for an understanding of the current structure (dare I say blueprints?), then some consideration of options, perhaps some testing out of tools and practices before actually doing the work (which undoubtedly will be iterative, component by component or room by room).
• Skyscrapers are big systems, toolsheds are small ones. Lamport himself states otherwise in the article: “While the specs I write are almost all informal, occasionally a piece of code is sufficiently subtle, or sufficiently critical, that it should be specified formally — either for precision or for using tools to check it. I’ve only had to do that about a half dozen times during the past dozen years.” He’s clearly talking about pieces of code, could be one method or a couple of classes or indeed entire systems.

Being too literal, ignoring the grey and reducing abstractions to strict constructs (ironic considering the vehement resistance to formal specification because it’s too constraining) has ramifications beyond design quality for aspects such as human communication, essential in any good team, agile or otherwise. We stop ourselves from considering the greater context, the bigger possibilities which might explain why some techies cannot easily relate to customer needs.

Research

A couple of Google searches reveals that Lamport has a body of work (including tools) related to reasoning about concurrent systems using specifications. This is notable because it focuses on the non-functional, something not often seen in discussions pertaining to TDD or BDD. For example, have you ever run across a test specification like this?

Will sort n items, distributed in any of the following orders (already sorted, exponential etc), in (n log n) time subject to the availability of memory being sufficient to hold 4 * n.

Returning to Lamport’s work, how often would you see any explicit treatment of concurrency or parallelism in applications of BDD or TDD? Isn’t consideration of these non-functionals relevant?

Some have argued that Lamport as the author has the responsibility for including all of this in his article. Really? Are we saying that an audience has a right to a complete, finished work that they can just apply verbatim, without thought or further development? Do all the best films have a definitive ending? Of course not, because there’s value in allowing a viewer to invent and go further.

Lamport opened the door to an opportunity for personal development and an improvement in the quality of one’s work, maybe some innovation too. Those who sought no further reading (a mere google away) and pronounced what he was writing irrelevant or covered by BDD or TDD have missed out.

These failures to dig deeper and put into context lead to stagnation of our practice (ironic given the focus on “practical” aspects such as coding). Research is essential to learning and growth but seemingly is becoming a lost art to many.

Wrapping Up

Lamport wrote in his article: “Few programmers write even a rough sketch of what their programs will do before they start coding. Most programmers regard anything that doesn’t generate code to be a waste of time.”

Meanwhile, Wired headed the piece with this statement: “With widespread access to free, online coding courses and tools, “coding” has become the new writing – the everyman’s skill.”

Given many of the responses to the article it seems that the readership proved Lamport right and Wired wrong. Not everyone possesses the skill to code competently and those that think it’s just about code are missing the key factors to make themselves so.

Continue reading ‘Motivations’ »]]> Almost as soon as the discussion of building services starts, there are questions about latency of remote calls and what is or is not a service

I have a basic rule of thumb for remote call performance tradeoffs:

if the total compute time (that includes background/offline work to keep things up to date) required to service a remote request is greater than the round-trip time + a little fudge, it’s acceptable to make it a service.

However, there are other things to consider outside of local performance concerns. An optimised protocol running over even 100Mbps ethernet can manage round-trips of 1ms or less. To put that into perspective consider that Google reports average worldwide round-trips to their site are ~100ms and within US ~50-60ms. Google has presence across the globe and does all the right things in terms of content serving etc. Most other sites don’t do nearly that well. These numbers don’t cover mobile internet either which can have substantially worse performance. The point is that losing a couple of ms on a remote call in the context of round-trips outside the firewall is no big deal.

That said, many point to the fact that once we make a reasonable number of service calls, the latency of customer response can increase significantly. Which is true, if we choose a synchronous model of remote invocation where we wait for each call to complete before we dispatch the next. The thing is, that’s not necessary, asynchronous requests make more sense as they allow for better support of timeouts (important for failure and load handling) to protect thread pools and such. Further, asynchronous requests give us the ability to dispatch work in parallel which is exactly what Amazon does in order to ensure that the 100+ service calls they make don’t severely impact page rendering.

Beyond round-trip performance there are a bunch of other motivations that contribute to a decision to make something a service or not. In my experience, these sorts of things rarely come up in architectural conversations but are absolutely essential to getting good results:

• Performance – a single application that contains disparate functionality can become difficult to tune meaningfully as load patterns interfere with each other – e.g. caching policies, storage performance characteristics, I/O versus compute intensive
• Scalability – an architecture that supports substantial scale for one type of function may not work well for another – e.g. a read-mostly load can be served from disk with a filesystem cache. This is entirely inappropriate for volatile, rapidly-changing, transactional information. These conflicts are further exacerbated by geographical dispersion.
• Availability – a single application containing disparate functionality can be brought to its knees by a single fault affecting a shared resource. The result is a situation where one functional problem ensures nothing functions at all – e.g. memory leaks, all data for all functions is retained in a single database or a single code fault generating exceptions that eventually exhausts thread-pools or causes a JVM to exit.

All the above have implications for our ability to support high-quality SLAs

• Manageability – dependency chains become deep or wide or both such that it becomes impossible to accurately predict the consequences of a change. Refactoring is difficult, automated testing options are limited. Builds take longer and longer. A single update of a library requires all teams to co-ordinate the change to ensure all code is brought up to date. All technical staff become expert in all aspects and all technologies. This creates long training cycles, makes recruitment difficult or alternatively requires tight control/standardisation of tools to limit technological proliferation which in turn inhibits correct solution construction and innovation.
• Data Management – some data is externally regulated and requires specific policy and infrastructure (e.g. PCI). When this data is mixed with other data (as is typical with monolithic codebases), the entire codebase and all data become subject to the same stringent requirements slowing development, increasing infrastructure costs etc.
• Operational – as for development, ops staff are compelled to understand all aspects of everything for the purposes of diagnosis. The sheer quantity of information produced from the single codebase can make separating signal from noise challenging – e.g. log files containing all messages for all functionality. Releases must necessarily be slow and careful because there can never be much certainty of code quality and rollback is difficult. Further, staging out of updates in small chunks is made challenging by virtue of the number of things that can cause compatibility problems during upgrade. 

The basic force underlying all of this commentary is:

A one-size, fits all approach where everything sits in one big build and is replicated across many app-servers all backed by a single database fronted by caches (and other variant architectures) will only get you so far.

Thus there is a point in growth (load, features, infrastructure, availability demands etc) at which this one size fits all approach starts to hinder progress and increase cost. Smart techies will watch out for relevant trends and make plans to transition to a more service oriented style as appropriate.

[ Sidenote: I'd normally advocate not doing an SOA from the get-go. However, infrastructures with properties like EC2 make typical one-size fits all architectures less appealing. They don't cope well with failure or relocation like a decent SOA can. ]

Continue reading ‘Postmortem’ »]]> I’ve been noodling a little on placement of infrastructure to serve a worldwide customer base. One can house that infrastructure in ones own purpose built datacentres, various colos or a cloud provider. Cloud is hot right now, Amazon is thus a heavily favoured option for many.

In terms of locations, they have pretty good coverage: US, South America, APAC, EU.

But there are skewing factors that mean these locations are not all equal:

• Some of them don’t have all the services on the AWS menu
• Some of them have limited capacity (2 EC2 Zones in South America vs 5 in US-EAST for example)
• Some of them are lower down the pecking order in terms of new feature rollout
• Cost is not uniform

When you look at it, the actual root causes are such they could occur anywhere in any of the regions. They all run the same software (accepting there’s some skew in releases/fixes as pointed out by Adrian Cockroft, see above) which would no doubt suffer the same ailments given the same triggers. Some claim, the age of US-EAST is a factor but the root causes suggest otherwise (power supply infrastructure has performed about the same for years). There is of course, another key difference which is that most customers are in US-EAST alone. It’s the biggest zone, if you wanted to cover Europe and US using AWS for the least amount of cash latency would drive you here. If you were savvy about Availability Zones and Regions, you’d probably design in another region, perhaps EU-WEST but we know that’s rarefied air.

That load going into US-EAST likely amplifies the effects of recovery storms in clusters in ways not seen in other regions. So is US-EAST such a bad-boy? I believe the answer is no but environmental factors make it appear so. If you wanted to avoid US-EAST, you’d probably select one or both US-WEST regions paired with EU but you’ll pay Amazon more and your engineering complexity will be higher.

Continue reading ‘Manifesto’ »]]> For various reasons that will become apparent over the coming months I’m drafting up my third set of SOA kick-off bits and pieces. There’s loads of stuff that ultimately needs to be thrashed out and detailed but I’m not one for doing all that up front, it’s decidedly anti-agile. All tech teams are different as are the organisations they work in so the basic questions are always the same, the rest is always “as you find it”. Writing a big chunk of stuff up front just isn’t worth it then.

Okay, so my chosen minimum is a manifesto and thanks to Messrs Bezos and Yegge, a chunk of the workhas been done for me. I’ve added a couple of other points that reflect my experience. In some ways, they can be inferred from the other points if you have the right experience but people in that category wouldn’t need this manifesto to take aim at.

1. All teams will henceforth expose their data and functionality through remote service interfaces.
2. Services must do all communication with each other through these interfaces.
3. There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever.
4. Any state transitions relevant to a service user that happen asynchronously to their requests will be made available via an appropriate push mechanism.
5. It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter.
6. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world.
7. All services will be designed with operations in mind. That is they must log appropriately, provide relevant monitoring and be easily started, stopped, installed and removed.
8. All services must be designed to handle failure gracefully. They must define and enforce SLAs, throttle appropriately and account for failure (slowness, loss of endpoint etc) in downstream resources.

[ Update: After some feedback (thanks Asher) I've added a point regarding state transitions. It is often tempting to make timing assumptions about state transitions that will not hold in the presence of challenges such as downstream failure or excessive load. ]

Continue reading ‘Pathology’ »]]> When the underlying demand for IT is cost reduction, the result will ultimately be increased cost as the business short-changes itself in a variety of arenas.

The core of any technology group is its people. It is their mindset, experience and capability that ultimately dictates the outcome of any work undertaken. It is essential the right people are acquired and developed, tight budgets for training and wages will make this almost impossible. The best one can do with such an approach is to hire inexperienced, intelligent individuals such as undergraduates who will hopefully learn faster on the job than most. One might be tempted by outsourcing but similar issues exist. In absence of talent, how does one assure the level of talent and experience or indeed the quality of work produced by a company over which there is limited influence?

Many companies maintain approved supplier and technology lists with onerous processes for making changes. The justifications are many and varied including the desire for a single point of contact for issues, bulk purchase discounts and the belief that a mature product is more stable. The effect however is to constrain the options for tackling any particular challenge typically leading to inappropriate use of a technology, bending it to solve a problem it was never intended to address. The result is a sting-in-the-tail architectural compromise that saves money momentarily after which a potentially growing long-term cost is suffered. Other undesirable side effects are to discourage staff from experimenting with new technology thus limiting the potential for innovation.

An attitude of cost reduction often drives mis-guided attempts at automation where the intention is to eliminate staff. What follows is a death-march attempt to build an end-to-end system that implements all the rules of thumb, guidelines and exceptions that computers are so poorly equipped to handle. If one takes cost-reduction out of the equation, it becomes clear that a focus on eliminating waste and drudge from repetitive tasks freeing staff to be more creative and innovative in their contributions is the better option. Taiichi Ohno describes it thus:

“First, work and equipment improvement should be considered. Work improvement alone should contribute half or one-third of total cost reduction. Next autonomation or equipment improvement should be considered. I repeat that we should be careful not to reverse work improvement and equipment improvement. If equipment improvement is done first, cost will go up – not down.”

A climate of cost reduction will also lead to a variety of software development compromises under the umbrella of “do something quick and cheap now, pay the price later”. One obvious example would be when a bug fix is done crudely in the name of saving time rather than doing the necessary small amount of re-design. System maintainability is increasingly compromised such that development costs for any given change increase over time. Quality also suffers as testing and reviewing are cut back.

The list of undesirable consequences of a focus on cost reduction is very long, let’s cut to the chase with a quote from John Seddon: ”If you manage costs, costs go up.”

What then do we do about this? We work with the board to develop a focus within the business on delivery of value to customers (e.g. innovative products, a reliable website and friendly support) which ultimately underpins revenue. Anything that does not improve the overall customer experience should be viewed as costly and ineffective.

Even the humble phone system can be evaluated in the context of customer value. Consider how it allows for more direct interaction with customers and provides a better environment for discussion of product ideas than, say, email. Importantly, the value-based approach makes it easier for staff to relate their work to the bigger picture with the potential for increased quality, engagement and motivation.

The value-based approach is exemplified by the likes of Jeff Bezos and the late Steve Jobs, can anyone say it doesn’t work?

[Originally published here]

Continue reading ‘Product’ »]]> All developers love to code, me included, it’s a fun part of the job. I wouldn’t say it’s the most fun, I find that in the design work maybe or perhaps in shipping product.

Shipping code is easy, shipping a product, that’s hard. You can write some code, shove it in an .msi or .jar and install it on a box no problem but to me at least, that’s less than half the job. Shipping a product means dealing with a bunch of additional concerns driven by one simple requirement:

We want our product in front of our customer, working well and rapidly getting better with each release.

This simple requirement has a bunch of consequences including…

We need to build little and often. Code degrades over time, that includes unreleased code. The larger the body of code, the harder it is to get it back into shape if it’s been allowed to degrade. For example, if one leaves testing until late in the project there is a high chance of many errors being found, much re-writing and re-testing. This is incredibly wasteful and breaches our simple requirement to get product upgrades out of the door fast. We want small pieces of code tested in tight loops.

Functional testing in tight loops with code is not enough. We need to focus on the non-functional aspects as well, with a collection of automated performance tests and regular use of a profiler to check on memory usage.

Problem resolution must be efficient. The first step in problem resolution? Answering the question “what changed?”. We need to know what we put out in a release. We might also engineer feature flags that allow us to slowly enable features and assess health as we go. If we start seeing problems, there’s a reasonable chance the last thing we enabled is the culprit. Obviously, releasing large bodies of code with all features enabled makes answering this question difficult leading to extended periods of problem resolution.

Having worked so hard to get our product in front of the customer, the last thing we want is to have failures persisting and disrupting their usage. That means we want substantial insight into what our product is doing, how it is being used and what’s wrong. For web applications that means we need quality monitoring data (covering function and performance) from all levels of the system, clean logging (we don’t want routine, harmless exceptions being logged as errors) and user activity logging (conveniently that helps us understand more about our customers and what they like or dislike – A/B testing anyone?). For a desktop application, “phone-home” type infrastructure is immensely powerful. One needn’t go as far as Microsoft have but some means to capture errors, reporting them back with relevant state and no customer involvement (fill in this bug form) is a useful minimum.

There are some other things we can do in this area, particularly in the case of online applications it’s possible to employ dark testing. More generally, limited user-testing via early releases can help here though one must be careful with customer data (corrupting files is a no no for example).

Performance – customers use things in all sorts of unintended ways, data-gathering on their exact habits can be limited. One must be artful in finding means to gain insight and react. One must also have in place some key indicators of system stress, think queueing theory as a way to identify flows and choke-points for measurement/tracking.

Keep the release process simple. Having got to the point where you want to make the product available, the last thing you need is an onerous, error-prone mechanism by which this is achieved. Large quantities of configuration should be avoided, favour convention over configuration and where possible have the product figure things out for itself. Manual work should be substituted with automation as even the simplest of repetitive work is error prone with human involvement (humans aren’t robots). For desktop applications, we don’t want emails about upgrades and user downloads from websites, much better to do automated updates a la Chrome or Tower and others.

Understand the environment. The interaction between software and hardware (in which I include network and storage) affects the overall consumer experience. One cannot design, build and test on machines with 16 cores, 64 Gb of memory and multiple cinema displays if the desktop of the average customer is 4 cores and 8 Gb of memory. To have such a difference is asking for trouble in algorithm selection, memory usage, storage utilisation and so on. Developers must spend time with the hardware their software will run on.

A Word On The Nature Of The Beast

A product is something we deliver into the hands of customers and once it’s out in the wild, our levels of control and insight are greatly reduced. We can’t make changes as and when suits us nor can we be sure that our product will be used in ways that match our expectations. Metaphorically speaking we screw all the bits and pieces together, do some testing, package them up and ship them off hoping that once the assembled product is in the hands of customers it’ll serve them well and they’ll be back for more.

Real-World Examples

Think about Apple, when they build a product, they intend for it to be low maintenance, almost a sealed unit with no user-serviceable parts. This isn’t dis-similar from having software in production, once it’s out there, it’s largely frozen, sealed from changes. When things break, just as with Apple, one must understand what went wrong through various forms of forensics, come up with a fix and rev out another version.

When one is not good at forensics, when one’s rollout of new versions is painful for customers or operational staff or both the result is poor reputation leading to reduced revenue and ultimately job losses.

Consider Formula 1 where the cars are covered in sensors to drive telematics that allow the pit crew to see what’s going on and start reacting to it quickly during testing, practice, qualifying and the race. There are considerably fewer outright failures these days as a result. The level of effort they go to in monitoring and testing is huge as they even sample fuel and oil over the course of a weekend to identify the onset of engine or gearbox issues, check on wear rates etc.

Continue reading ‘Messaging’ »]]> Assume we wish to consume an ordered stream of messages.

Assume one or more messages may be lost from the stream.

Assume that we wish to develop and retain some state built against those messages.

Assume each message contains a monotonically increasing sequence number or some other mechanism by which ordering and gaps in sequence can be identified.

If we encounter a gap in the sequence of messages, we need to recover all the messages between the last message we saw and the one we’ve just seen that allowed us to deduce messages were missing.

The simplest way to recover these messages is to query the originator of the messages. If the number of messages required is substantial this can be a costly operation. We can reduce the cost of this operation by having the source maintain a checkpoint that summarises the message stream prior to a particular point in history. Each checkpoint includes the sequence number of the last message that is digested into it.

When we detect a gap in the sequence we query the source for the checkpoint. We compare the low end of our gap with the sequence number in the checkpoint. If the gap ends above this checkpoint, we request a replay of the message stream for the gap otherwise we recover the checkpoint and request replay of all messages from that point on.

This mechanism works well if we are for example, trying to generate a somewhat partial price history from a market. One can afford gaps in this circumstance and replay from a point in time (a checkpoint) is sufficient to get back in sync and recover a certain amount of lost history.

A complete history would require the ability to recover any range of log records since the earliest point in time the history is required. This presents the source with a problem where it may need to retain all records for the duration of a market’s life. A related problem is one where the consumer of the messages is constructing some form of aggregated state that is not available in the checkpoint such as a full order history for a market.

In such cases, it is desirable to have a number of replica consumers each of which independently produces an aggregate. These replicas shouldn’t all be located in the same area of the network as, if they are, it is highly possible they will all lose the same messages.

Each replica maintains a checkpoint of its aggregated state which, as per the case for the source, should contain the sequence number of the most recent message digested. A replica that finds itself out of date now contacts another replica for a recent checkpoint and requests replay of messages since the sequence number in the checkpoint from the originating source. The replica can then re-synthesise its state by applying the recovered messages to the checkpoint.

As a consumer detects a loss of messages from the arrival of later messages, natural pauses in the message stream (e.g. because market activity is absent) can result in the consumer being out of date for significant time. A source can assist in this situation by emitting a checkpoint (or heartbeat) message periodically during inactivity. If the consumer knows the frequency of checkpoints it can compute the maximum period of silence it should endure before seeing a message that is either a genuine update or a checkpoint. Should it see such an extended period of silence it can immediately assume messages are lost and set about recovery.