I think at the end of the day, programmers will usually always have a bias to doing things in their programming language oppose to outside. We see this shift happening in some databases and how all these frameworks have tried to eliminate as much configuration as possible.

Nobody likes a gigantic XML document of configuration that describes how a WCF service wires up or some messaging middleware routing is configured.

There’s a huge problem though when all of this is eliminated. Operational flexibility is missing.

A few days ago I wrote about how a tightly coupled situation increased operation complexity. That situation was multiplied in difficulty due to the lack of ability to dynamically re-configure these systems.

One option and a really good idea for services is to provide a command set that you can use to tell the service to do things you need like re-configure routing. This becomes really useful for operation teams against production systems.

In some cases the changes you make will need to be persisted so that if the service restarts it doesn’t revert back to the old behavior.

But haven’t we just made a really complicated glorified configuration file?

It seems our industry bounces from one extreme to another all the time but we never quite settle in a middle-ground which means we lose some value that can be extremely important.

If you are still dead against configuration files, at least provide the operational flexibility in some fashion.

I hate configuration files too but I value my system availability and operational flexibility much higher than my personal preferences and inconveniences.

The paper points out that the literature on Paxos has been known to be hard to understand and difficult to apply in actual code. There are a lot of Paxos implementations in the wild but it’s hard to gauge what the quality and reliability of these are. It seems the goal with RAFT was to design an alternative that was easier to describe and in turn easier to understand and implement.

I’ve read the Paxos papers several times and after reading the RAFT paper once I feel I understand more after my first reading than I did in my first reading of Paxos.

I plan to give this paper a few more reads.

In one case I saw a v2 of a message contract get introduced that was designed well and did not introduce any breaking changes. From an outside point of view, v1 services should be capable of processing the v2 message and down stream systems that may be in the chain who if upgraded to v2 should function as expected.

Unfortunately the private implementation wasn’t so tolerant. The v1 service was being lossy (as explained in the blog post above) and down stream systems were broken.

Accidental operation complexity

Thought out strategy can work around this tight coupling but it becomes increasingly complex and sometimes extremely expensive to do so. This tight coupling created a lot of operation complexity.

We had to put in place an elaborate strategy that upgraded specific parts in sequence and do message routing to ensure no v2 messages went to v1 servers. In the process we had to grow our cluster size temporarily to accommodate the production load while deprecating these v1 services.

Since the message should not have broken anything if the v1 service wasn’t being lossy during the serialization process, upgrading the cluster would have been extremely simple and much less expensive. It would have been a rolling upgrade where any v1 service could still handle any published v2 message.

The life cycle of a piece of software isn’t how long it takes to ship v1. Although using serialization saved some development time, it’s foolish to track the development cost of v1 bound to its deployment. The cost of v1 trickles into v2 and the operation cost of moving from v1 to v2.

Yes the complicated strategy did work, it was expensive and it required a lot of planning, and frankly none of that should have been necessary.

Version tolerance

Creating loosely coupled systems that only couple to required data and don’t discard data they should be ignorant about make them more version tolerant.

Version tolerant systems introduce less accidental operation complexity. These systems are easier to upgrade and can evolve quicker and more frequently.

A cluster I was using was configured in a way that all the virtual machines were using a pool of resources and when that pool experienced a resource failure affecting a fraction of the pool, all the nodes on that physical machine crashed.

They all crashed because they were all sharing the same pool and when something failed they all experienced the failure.

In distributed systems it is important that we accept that failures will happen but its also good to try and reduce coupling at the software and hardware level to reduce the failure foot print.

1 failed node is better than 32 failed nodes.

I can’t think of a quicker path to failure than thinking about the solution before the problem. I’m not great at problem analysis yet but it’s something I’m constantly reminding myself to do.

I consider problem analysis a fairly creative process. I try to brainstorm with my team (and invite other team members in for diverse opinions). Sometimes we spike some research to gain more understanding (time boxed). We write down the problems challenges and prioritize the areas with the most risk. We write down all possible options and try to identify their strengths and weaknesses and eliminate potential options that don’t align well with the problem. The obvious ones to eliminate are the ones where the risk collides with the weaknesses.

I think it’s crazy town to try and twist a problem into a potentially wrong solution. It’s my belief that if you keep choosing the same option every time, you’re guaranteed to be wrong some of the time.

Why would I do that?

He wrote about HAT, not CAP: Introducing Highly Available Transactions where he describes the concept of highly available transactions and how traditional database designs don’t perform well in the event of a network partition.

Highly available transactions provide strong guarantees (not quite as strong as ACID) while providing higher availability than ACID can under partitioned conditions. Peter suggests HAT’s provide the strongest set of guarantees with high availability.

HAT’s are a masterless design.

HAT’s cannot provide Serializable Isolation (the strictest of the isolation levels in ACID). Peter previously posted an article titled “When is ‘ACID’ ACID? Rarely” that reported the state of ACID in many existing databases which points out how most do not come with strong ACID guarantees out of the box because many don’t ship with serializable isolation enabled. Some of the databases don’t even have serializable isolation as an option.

This is really interesting because it points out HAT’s are aligned quite well with how people generally use databases today but using new algorithms to provide higher availability.

I highly suggest keeping an eye on any material coming out of Peter. Amazing stuff.

Charles Bachman in 1973 won a Turing Award for building a “NoSQL” database in the 1960′s called the Integrated Datastore (IDS) and this was what is called a Navigational Database. One of the pains is it was very hard to evolve.

In the 1970′s relational databases came along and provided a high level model which allowed the databases to encapsulate the internals.

Also in the 1970′s Unix came along. “The most important job of UNIX is to provide a file system” – original 1974 Unix paper. This is interesting because both databases and operating systems had similar goals but using different approaches.

I’ve always found in database debates there’s a tension between different views and I couldn’t really put my finger on it but Eric put it into great context by describing Relational View vs System View. Once I saw it described in that way I thought “that’s it!”. Some key points I got out of this are below.

Relational View

• Data outlasts any particular implementation (the model doesn’t change)
• A top-down world view

System View

• Bottom-up world view
• Layers and composition

Also great point is that BASE doesn’t necessarily mean “always inconsistent”. It can mean only inconsistent during partitions.

Eric also describes how during a partition there needs to be a mechanism for compensating for bad actions. This is where I think Saga’s can come into play and it is mostly an ignored area for research. I think there’s a lot of growth still to be made on this topic.

Eric explains the myth about the most common supporting argument of ACID being the canonical banking “but money!” examples. He describes things like ATM’s not using ACID transactions but using auditing instead and how ATM’s have “partition mode” and how they use commutative operations so that balances end up correct and compensations occur if something goes wrong.

In the beginning I loved the convenience of serializing data from various formats like XML or JSON into classes (that more resemble structs actually).

I got into many debates where I held this opinion strongly.

But later on I built a system that you could not shut off to deploy updates. Availability was important and the system was large enough that an upgrade had to be a rolling operation where there would be multiple versions of the system online in production until the system reached consensus on a single version.

During these operations I witnessed something that completely changed my opinion on serialization (I can hear Darrel Miller saying “I told you so!”).

The components that used serialization were more brittle than the ones that did not.

When a v1 service gets a v2 message and then deserializes it into a struct it has lost the v2 data. If that service then decides to do something that then ends up on another v2 service, the v2 context isn’t available.

Some messaging frameworks or document databases make serialization very convenient and make lossless more difficult and you may not find out until it hits production because it may not be super obvious up front you’re making this trade-off.

Serialization is a lossy copy operation. It doesn’t matter if its for messaging or a database. If you try to deserialize v2 into v1 struct and then serialize you end up with v1. It doesn’t matter if you use formats like protobuf etc. If you serialize from the struct which is a lossy copy of the original data it doesn’t matter how flexible the format is if you’ve left it behind.

During that experience the systems that didn’t rely on serialization experienced no problems during the upgrades because instead of doing a lossy copy of data, they were interpreting the data and only coupling to the data they knew how to interpret but without losing the data they were ignorant existed.

Espresso provides consistency, lookups on secondary fields, full text search, limited transaction support.

Espresso uses MySQL/InnoDB as storage and secondary indexing.

Espresso uses Lucene for full text search.

Espresso uses Databus for replication.

Espresso uses Helix for cluster management.

While some people may be surprised to see a document database using MySQL as a storage engine, it’s not terribly uncommon and mature databases often have very capable database internals.

In large systems and with the diversity in the database industry these days one of the challenges people face is moving data from one database to another database so that they can leverage the data in multiple ways.