I’m sure you’ve heard a lot about it, and seen a lot being done about it.

But are we making progress? Let’s take a quick look:

CO2 Emissions Are Still Increasing Most Years

To get climate change under control, we need to get greenhouse gas emissions from human activities first to stop rising, then down to zero (or negative!). Here’s a chart about CO2 emissions, which is the largest category of GHGs (75%).

As of 2023 (where this dataset ends), CO2 emissions were still going up most years. This is obviously not good. Climate models that show how to limit global warming to +1.5ºC rely on emissions peaking between 2020 and 2025, and roughly halving by just 2030.

If you spend enough time online, you might have seen the above summarised by an anti-renewables campaigner as “We’ve spent all this money, and it hasn’t even stopped emissions going up!”

But ! … Peak CO2 Emissions Is Very Close

… and already in the past for many countries.

While looking at continually growing emissions is not a great feeling, looking at the rate of growth (the first derivative of the above chart) shows us some very encouraging signs:

1. In high-income countries, emissions are already heading down, not up. They’ve been doing this for 15 years now. And the world hasn’t ended.
2. While the rest of the world had a period of rapid growth in emissions, that growth has rapidly reduced. They’re still emitting a lot each year, but at least the amount isn’t growing quickly any more.
3. The total CO2 emissions of the world is now very close to going negative, averaging growth of less than 1% per year for the last decade.

In short, the richest nations are already weaning themselves off fossil fuels (and seemingly faster as time progresses) while the rest of the world looks like they’ll be following that lead soon.

Some Reasons To Be Hopeful

It’s disappointing that we’re only now looking like we’re about to turn the corner of preventing emissions from increasing, and we still have to reverse 200 years of growth in a very short time frame. But there’s some positive feedback loops that might help swing things around rather rapidly:

• A significant portion of fossil fuels are used just to extract, process, and transport fossil fuels! So as we start consuming less fossil fuels, that in turn requires less fossil fuels to get them out of the ground and into a burning machine somewhere. If we’re lucky, perhaps the sources that require the most in-process emissions will be the first to be left in the ground.
• Many renewable energy assets (solar panels, wind turbines, batteries, EVs) currently have significant CO2 emissions in their supply chains from mining, industrial processes, and transport, simply because that’s where a lot of our energy comes from today. As we deploy these assets, building the next generation of renewable assets will result in less emissions, because they’ll be partially built using clean energy. Each time we go around that cycle, replacing a given amount of fossil fuels with a renewable asset “costs” us less emissions than replacing the same FF amount did previously.
• The rate at which we’re deploying solar and wind power is the fastest in history, and it looks like it’s still getting faster. Battery energy storage system (BESS) prices are still plummeting most years, and EVs are only just getting a foothold in many countries but rapidly accelerating on their take-over of the transport world. If these exponential tailwinds continue, there’s a good chance that GHG emissions could drop off not gradually but precipitously – though this relies on renewables deployment outpacing growth in energy demand.

So, Are We Making Progress On Climate Action?

Absolutely, yes!

By zooming in on growth rates and separating out the leading nations vs the followers, we can see the trend that is happening amongst the world’s richest nations: the massive win which is the now negative growth (i.e. contraction) in CO2 emissions. And as that trend is moving further away from zero, it’s an accelerating shift. We also see that the rest of the world is not too far behind in terms of moving closer to a phase where they’re shrinking their fossil fuel dependence.

Of course, we should be going faster. As fast as practical! But the efforts we’ve been putting in over recent decades are paying off, and there’s reasons to hope that a rapid downwards shift in emissions across the globe is just around the corner.

Charts Data Source

Global Carbon Budget (2024) – with major processing by Our World in Data

The post Are We Making Progress On Climate Action? appeared first on Evolvable Me.

Why can’t nuclear power help Australia solve the climate crisis?

Read on to find out.

I Had To Explain The Energy Transition To My Local Federal MP…

My local Australian Federal MP, a member of the Liberal party (that’s roughly equivalent to Republican / Tory / conservative, for overseas readers) – posted a video on Facebook implying that recent blackouts in the area, as well as increasing energy costs, are Labor’s fault. It accused Labor of making energy security “partisan”, and called for “a mature debate”.

I replied, trying to stay as factual as I could, pointing out that the blackouts in NSW are the result of one third (4 out of 12) of our coal power plants being out of action, nothing to do with (unmentioned but dog-whistled in “Labor’s energy plan”) renewables, and that the finger point for a lack of energy supply likely shouldn’t go to the government that only came into power 2.5 years ago, but the (Liberal + Nationals) one that was in power for the decade before that and did little to prepare for the inevitable exit and breakdown of many ageing coal plants.

He replied – politely, I will say – asking:

Should we be closing coal early and taking it offline?
That is what is in the AEMO plan.

… along with platitudes that he is focused on the future, not what previous governments could have done differently. He also dropped in a recent quote from a very highly-paid, conservative-leaning, and polarising former bigwig of the public service, which included, “We are rushing into a third-world country.”

Here’s the thing about that question, though:
IT’S A TRAP!

Wrapped up in the notion of “early” is some sense that there’s an inherent reason not to close a coal plant before… well, when, exactly?

Anti-climate action folks like to play on this notion imbued in that word “early” to try to make climate action advocates look like we want to waste things, or perhaps even ruin society! (Ye gods, could anything be further from the truth?!)

So, I thought that trap needed addressing.

Now You Can Read About It Too…

Since he’d taken the time to write a polite, sensibly-lengthed reply, and asked a question of me, I took the time to respond in kind, though, admittedly, with a bit of passion thrown in. But I answered his question honestly – and thoroughly – I checked my facts, and I walked through the logic of why nuclear doesn’t get to have a ‘Climate Action’ badge slapped on it in Australia.

Proof-reading it before sending it to Zuck’s cloud, I thought, “Hey this isn’t half bad. Other people might benefit from reading this.” So, here it is. Hope it’s useful to you in understanding and/or explaining to others where we find ourselves.

I wrote…

Thank You For Your Reply

I think it’s important we are clear with people about what “early” means. Early in regards to closing coal plants does not mean “earlier than when the grid is ready”. It means “earlier than when the plant owners currently plan to switch them off”. 

Should we close coal plants “early” if we can? Absolutely! We should be ending all use of fossil fuels AS SOON AS PRACTICAL. Solving the climate crisis depends upon this transition happening right now, and rapidly. We should all be heavily invested in making “as soon as practical” earlier, wherever we can. So yeah, if we can accelerate when we can switch off coal generators, 100% we should do that.

This does not imply putting the grid in a bad state!
It is about accelerating the grid’s transition into a better state, where coal isn’t needed.

How do we bring forward the date when we can safely turn off coal generators? Simply: By accelerating the deployment of firmed renewables and their integration into the grid ASAP.

On nuclear power: It has no role to play in accelerating the transition [in Australia]

Unequivocally. Why?

Currently half of remaining coal fleet owners WANT to retire their plants by 2035, with those closures being voluntarily brought earlier all the time. AEMO’s ISP (that’s the long-term national plan for Australia’s electricity) advises us that, for Australia to meet our climate goals, that needs to be closer to 90% by 2035.

There is no pathway by which starting a nuclear power industry here is going to deliver any power before 2035, and yet ALMOST ALL of our coal should be replaced BEFORE then to meet our goals. That means it can only be replaced by firmed renewables. (Or those who speak on behalf of the gas lobby might say lots more gas, too. But that would be giving up on the climate goals.)

What’s great news is this: Renewables don’t need a debate, don’t need a new industry, don’t need 15+ years of lead time, and don’t need to be 100% bankrolled by taxpayers. Firmed renewables are available now, are cheap, and constantly getting cheaper, and there’s a huge industry and workforce ready and willing to roll them out.

There is a narrative from fossil fuel lobbies and many conservative politicians and pundits, trying to fool people into thinking that the people most vocal about climate action want to do it faster than it’s safe to do. That is a straw man argument.

What people concerned about the future want is to accelerate a safe and just transition, not a transition faster than is practical. That is why all this talk about nuclear is so infuriating and concerning, because a plan for replacing coal with nuclear power in Australia can only achieve delay, not acceleration.

P.S. I think talk of Australia becoming a third-world nation is farcical hyperbole, and I don’t think it belongs in a mature debate, as your video calls for.

The post Climate Action: Accelerate. Safely. But, Accelerate. appeared first on Evolvable Me.

If you work in tech, you’ve almost definitely heard about microservices: the trendy style of software architecture where a system is split into multiple, independently-releasable services, that are modelled around business domains, and communicate via a network. Some people rave about all the amazing things they’ve achieved by using microservices’ benefits. Others rant about how much of their time it wastes and how much they hate it.

Like pretty much everything in tech, microservice architecture is a trade-off. Will it do great things for your organisation or not? That depends largely on how well set up you are to take advantage of the benefits and to absorb the costs.

Microservice Architecture is a trade-off. Whether it's going to do great things for your organisation depends on how well set up you are to take advantage of the benefits and absorb the costs. Read more: Microservices: The Benefits and Costs
Share on X

If you’re looking to make a decision about whether to use microservices in your team (or reverse one!), here’s a list of many of the pros and cons which you’ll want to consider.

Benefits of Using Microservices

Microservice architectures are about having independently releasable services that are modelled around business domains. There’s a number of benefits to pursuing this style of software architecture. Here’s a really quick run-through of the main benefits people typically cite:

Team Autonomy

By assigning ownership of services to teams and enabling them to deploy services individually, they can release changes on their own chosen timeline. This minimises the need for coordination and negotiation with a wider group around deployments. That, in turn, can greatly reduce the time between making changes to the software and getting feedback about the value that has (or hasn’t) been delivered.

Fault Isolation

When designed well, a microservices architecture can insulate failures in one service from affecting the functionality of others, reducing the “blast radius” of any fault. When designed poorly, though, failures in microservice systems can be just as bad as with monoliths, and much harder to track down!

Technology Choice

If you properly isolate services from each other using technology-agnostic interfaces, you can make use of different technologies in different services. That means, for example, you could employ a different programming language or framework where that will provide a benefit to the software. It also provides the opportunity for teams to experiment with new technologies with minimal impact on the overall system. These things are very hard to do when the whole system is built as a single monolith.

Independent Scaling

In contrast to a monolithic architecture, independent services allow you to increase the resources for only one part of your system, rather than needing to provision all instances of your monoliths with the resources to handle the most demanding peak loads.

Smaller, Simpler Codebases

Microservices is all about building smaller pieces of software focused on individual, cohesive business domains. For developers, this results in it being simpler to navigate and comprehend codebases.

Simpler Deployments

Deploying individual pieces of software separately can mean many parts of the system can be deployed with little or no effect on customers, partners, or other internal systems. You may still have parts of the system that need extra caution to be applied when they’re deployed, but that caution no longer needs to get in the way of deploying everything else.

Some Costs of Microservices

So there’s no doubt there’s significant microservices benefits. If there weren’t it wouldn’t have become so popular.

But, of course, your team won’t get any of those benefits without huge changes to the way you do things. Some of those changes are going to require costly adaptations from your team.

Here’s some of the most frequently highlighted costs of moving to microservices:

Complexity Is Distributed

Ideally, any operation in your system will involve as few services as possible (because you designed your services around business domain boundaries, right?). But ultimately it’s a system, and not just a bunch of independent software, because the services interact. And understanding those interactions will be crucial to understanding your running system.

Infrastructure Needs More People-Power

Microservices results in more infrastructure, and often more types of infrastructure. That means more infrastructure work, so you’ll need more people working on infrastructure. As a result, you’ll have less people working on initiatives that add value to your organisation’s primary mission. Infrastructure work may happen mostly in a dedicated team, or it may be spread across all your teams and somewhat hidden. Either way, you’re going to be investing a higher proportion of your team’s time in infrastructure with microservices than without.

Increased Network Traffic & Latency

Splitting software into multiple services that communicate across a network obviously increases the network traffic and the bandwidth required to support that. It also adds significant latency to any synchronous process that needs to cross a service boundary. Your team will need to become proficient at recognising and minimising network latency. They’ll also need to learn techniques for avoiding latency when it’s introducing unacceptable delays.

New Network Failure Modes

Dropped packets, connection limits, read timeouts, DNS… There’s a myriad of ways any network interaction can fail. Moving to microservices will probably bring a lot of them into the scope of your developers’ daily work, but they weren’t something many of them would have had to think about while working in a monolith.

Monitoring Scope Explodes

Monolithic systems often have very few things needing monitoring – be that hosts, networks, databases etc. Even when there’s a lot of horizontal scaling, you may only be monitoring a very small number of types of thing, which means each new node added requires a fairly trivial change. With microservices, this is not the case. Instead, we’re creating many different types of things, each having some shared and some unique monitoring requirements, as well as having far more activity occurring across networks. All of that adds up to a need to invest far more into monitoring than a monolithic approach would require.

More Codebases

This is the flip side of the coin for the benefit of “Smaller, More Easily Understood Codebases“. There’s now not just one codebase, but many. What’s more, they integrate with each other across a network, not through function calls, so navigating between them will likely involve some manual searching rather than a simple click-to-browse. Again, a well-designed microservice architecture will minimise the number of services you need to look at in order to understand a single operation or workflow. But you’re not going to get away from sometimes needing to open up several projects in your IDE and trying to piece together how the code interacts.

More Build Pipelines

Independent releasability typically means independent build and deploy pipelines for each service. You might have had one or two people that looked after your build pipeline in the past. Now you’re going to need at least some expertise in each team to manage the pipelines for their services. You’re going to need a build / Continuous Deployment tool that allows your whole tech team to manage large numbers of pipelines and builds. You’ll want to figure how to get fast builds with minimal delays, but without wasting a lot of resources on idling servers. And you’re probably going to need to think about how standardised or bespoke you want your many build pipelines to be, and how you’ll manage any shared or mandated build patterns.

Testing Has To Adapt

Think about testing software, both automatically and manually. And now think about how microservices means changing your software architecture to be multiple different pieces of software, owned by different teams, that can be deployed independently, and which communicate with each other over a network using APIs and/or asynchronous messages. That means a lot of things need to change when it comes to how you’ll test your system and build confidence about its quality. Unit + integration tests alone aren’t going to cut it any more. Testing the full cycle of some features may become so complex that it’s impractical to automate it. And your quality advocates are going to need to learn about a whole new set of risks, failure modes, and tooling in order to deliver expert advice to the team.

Technology Governance

Microservices introduces the possibility of thousands of different technologies across all of your services, managed in many different places. Most organisations won’t want a technology “free-for-all”, for a variety of reasons. That means you’ll probably need a way to record which technologies are currently recommended, deprecated, and being experimented with, and a process for moving things through that lifecycle. (A common approach is to maintain an internal tech radar.) At the very least, you’ll want to be monitoring all of your dependencies for security vulnerabilities. You might also want to consider nipping this in the bud by consciously choosing boring technology.

Increased Security Attack Surface

While there are many aspects to application and system security, the interface between the network and software is commonly a choke point where many security controls are implemented. Microservices architecture vastly increases the number of interfaces that the software exposes to the network. Because of that, microservices can enlarge the attack surface of the system by an order of magnitude. The end result is you’ll have a lot more security work to undertake in order to protect your system from attackers. The opportunity to use a variety of different technologies, and to be using various different versions of each one, also adds to the complexity of managing microservices security.

Comprehensive Data Analysis Is Harder

Monolithic applications usually have monolithic databases. Microservice architectures following best practices around information hiding almost definitely won’t. That means if you need to do some reporting that will combine data from multiple services, you probably won’t be able to just whip up a SQL query. Many organisations with microservices and non-trivial reporting requirements end up investing heavily in building a data warehouse or data lake.

Module Refactoring can get Harder

Few things make the costs of microservices apparent to software developers more than needing to refactor some functionality that is distributed across two or more services. In a monolith, shifting code around and just running the tests to check everything still works is child’s play. On the contrary, refactoring code between services is positively painful. It often requires unhealthy amounts of planning, cross-team coordination, multi-stage migrations, and the moving or re-writing of many existing tests. And all of that is just to make the system do what it already does but in a better way. Ouch.

Do Microservices Benefits Outweigh The Costs for YOUR Team?

So that’s the main microservices benefits and costs you’ll need to be aware of as you make a decision about whether or not to employ the approach in your team.

Okay – it was a lot. The list of microservices downsides may seem a lot bigger than the upsides. Depending on your team size and what you’re trying to achieve, though, the benefits of decoupling the work of individual teams may be huge, and the costs manageable as long as you’re aware of them.

So, knowing all this, is it the right decision for your organisation to take on microservices or not?

To answer that, we need a way to compare these benefits and costs to your organisational context. In the next post, I’ll be sharing 5 heuristics that you can use to get a feel for whether your team is in a better position to leverage microservices, or be inundated by them.

Image Credits:
‘Mycenaean Bronze Scales’ by Mark Cartwright
‘Hong Kong International Commerce Centre’ by Bernard Spragg
‘pipes’ by morisius cosmonaut

The post Microservices: The Benefits and Costs appeared first on Evolvable Me.

Switching to a new software architecture style is costly. I advocate choosing where to employ microservices first by using them to solve existing problems in the team. This helps the organisation extract value from the new architecture with each step towards the new world. This series is working through a set of problems that your org may have, and how microservices might help solve them.

Tricky Deployments

Some components of your system will be tricker to deploy than others. This is true regardless of whether you’ve already got microservices or you’re deploying a monolith. Some components might be in use by 1000s of customers 24×7, while others might only be used once a month. Upgrading a component might require other components to be placed into a certain state, while others can be upgraded independently of the rest of the system. Coordination may be required between multiple teams in order to effect a trouble-free deploy of some components, whereas others might be actioned by a single person.

Ultimately, we’d like all our components to fall into the second half of those categorisations. The reality of prioritisation and resource constraints means we’re unlikely to reach deployment nirvana for all artefacts.

Let’s think about a monolithic architecture which, in this context, refers specifically to a monolithic deployment. In this type of architecture, deploying a change to one component means deploying everything – you’re unable to deploy components separately. That means that whatever is the trickiest step of the trickiest component to deploy becomes a necessary step in every deploy.

Costly Deployments

I’m going to flip the language now from “tricky” to “expensive”. Usually, when something is tricky to deploy, it’s expensive to deploy. It may be expensive because there’s manual steps to make it lower risk; or there’s coordination required that takes people’s time away from other, more valuable work; or maybe you literally lose revenue every time a component is taken down for a few minutes. Imagine if each component were able to be deployed independently. It’s easy to conceive that the deployment of any component has an average cost to the organisation. And now we can see clearly a huge downside of monolithic deployments: the cost of any deploy is at least the cost of the most expensive component to deploy.

In an architecture with a monolithic deployment, the cost of deploying a change to any component is at least the cost of the deploying the component that is most expensive to deploy.
Share on X

Actually, the situation might be even worse. If these “costs” of deployment can’t be significantly mitigated by the fact that the deploy happens all at once, the cost of deploying any component may be the cost of all components combined.

So, do you have a component in your monolith which costs you significantly more to deploy than other components? If you do, you can reduce the cost of deploying the monolith by extracting that component from the monolithic deploy.

Here’s an Example

At Tyro, we had a piece of software which routed financial transactions from payment terminals to banks and card schemes. It also had many corollary responsibilities. Most notable of those were reporting the results of transactions to backend systems and providing payment terminals with their configuration.

As well as doing the routing, this software maintained the actual links to the banks and schemes. That included Layer 7 protocols for connection establishment, keep-alive and security; multiplexing of message requests and responses; and marshalling/unmarshalling of messages. As much of this code was based off specifications used by hundreds of banks across the world, it was very static. The big international schemes typically issued changes which we might incorporate once every six months. The local banks were even less likely to change.

Now, there was an expectation from our partners about these links. The expectation was that the links remain up – that is, the TCP connections should remain active for a long time. Long time here = months. If a connection went down, they expected to see it re-established almost instantly. If the link wasn’t going to be re-established instantly, e.g. because of scheduled maintenance, they wanted us to tell them in advance. From memory, they wanted to be given a week’s notice.

The code for operating the links rarely changed, but other code in this mini-monolith changed regularly. We were constantly adding new features to the terminals, so the corollary functions were changing constantly. We were rolling out these changes on a two-week schedule.

So, we had a piece of software which contained components that were hardly ever changed, but which were deployed every two weeks because their deployment was tied to other frequently changing components. There was an expense of having to advise third parties of “scheduled maintenance” every other week. More often than not, though, the message about the maintenance wouldn’t get delivered to the right people in one of these partners. A panicked call would then come in during the deployment from one of the partners. They would see the link had been down for 1 minute, their incident management process would trigger and we would then need to respond to that. 

Reducing our Costly Deployments

To resolve our costly deployment issue, we decided to move the links to the third parties out into their own piece of software. In fact, we moved each link into its own individual service. This meant that, going forward, each link would only be taken down if the code for that specific partner changed. The links are now able to meet the partners’ expectations of staying online for months. Consequently, the expense for us to coordinate with the other parties was constrained to the few times a year we did an upgrade.

Some years later, the work to separate these components into their own services also paid extra dividends. As the company’s business scaled and transaction traffic increased, the transaction routing software was able to be scaled independently of the links.

Gotchas

The first caveat to this approach is to ensure that deployments of the component you intend to break out will be infrequent. The key payoff comes from just deploying the expensive-to-deploy component less frequently.  If the component is actually in active development, then you’re not going to see much saving by moving it out. There’s a relationship between how much non-valuable work is created by deploying the component and the amount of deploys you’ll want to avoid in order to get a positive ROI.

The second caveat is that it’s a good idea to first look for ways to reduce the cost of deploying the expensive component before extracting it. Think about how you can somehow remove the coordination with other teams, or make it less time-consuming, or automate any manual steps associated with costly deployments. Extracting a component from a monolith into its own service its often a costly exercise itself. Reducing the pain associated with deployment may be a cheaper investment in many cases. However, also keep in mind that having components moved into their own services can have many other advantages in the short and long term.

If you want to be notified when future articles are published, sign up using the form just under my photo.

Previous article in this series: Use Microservices to Solve Developers Stepping on Each Other’s Toes

Image credit:
‘US Navy 070720-N-8119R-040‘ by Gretchen M. Roth (U.S. Navy)

The post Using Microservices to Make Costly Deployments Less Frequent appeared first on Evolvable Me.

Switching to a new software architecture style is costly. I advocate choosing where to employ microservices first by using them to solve existing problems in the team. This helps the organisation extract value from the new architecture with each step towards the new world. This series is working through a set of problems that your org may have, and how microservices might help solve them.

Is your team wasting a lot of time working around each other?

If you have a large team of developers working in a single code base, it’s inevitable that the work of some team members will interfere with the work of others. Many of us are now using tools (like git) and processes (like continuous integration) that can lessen the impact of interference. (Although other modern practices like feature branching can make conflicts worse.)

Despite improved tooling, merge conflicts are still annoying and sometimes expensive to recover from. Everyone fears finding someone else has been making wholesale changes to code that they were also making wholesale changes to.

This can get exacerbated when people from multiple teams are working in the same area of a codebase. It’s especially true if the organisation isn’t set up to encourage regular communication between such teams. And the more your monolith’s internal architecture looks like spaghetti, the more people will bump into each other’s changes.

How Microservices Reduce Merge Conflicts and Enhance Collective Ownership

One of the key advantages of microservices is that it places natural limits on the amount of cross-team interference. Giving teams their own codebase/s to work in means they know that only the people they work closest with will be changing the same repository.

The first problem this (obviously) helps with is reducing merge conflicts. Conflicts cost time to resolve and increase the risk of defects, so minimising them is a good thing. The other issue it can help with is the pressure that regular interference puts on collective code ownership. I’ve seen developers avoid making improvements to code because they want to limit the chances of a merge conflict. With their own codebase, developers have the confidence to re-shape code so that it’s the best design it can be. There is a scale beyond which collective code ownership doesn’t work for a single codebase. But that doesn’t mean you should abandon it. What’s better is to create smaller scopes of code and teams within which it can continue to work well.

There is a scale beyond which collective code ownership doesn't work for a single codebase. But that doesn't mean you should abandon it. What's better is to create smaller scopes of code and teams within which it can continue to work well.
Share on X

If you’re hearing that developers stepping on each others toes in a monolithic codebase is causing a lot of pain, here’s what you can try. See if you can find upcoming work that will allow one or more of the teams to develop almost completely outside of the existing system. You might even decide that the team working on the new service won’t do any code in the monolith. Instead, they can rely on one of the other teams to implement any changes required in the existing code.

Here’s an Example

When I first led a team at Tyro, we had recently split into three Engineering teams. Two of the teams were working almost full time on our big backend monolith. The third team would also jump in there from time to time as needed to support their work. It was a big repository, so we didn’t butt into each other’s changes every day, but it did happen. It also made other things more effort than they should be. When the build broke, or an alert came in from production, deciding which team should respond wasn’t always straightforward.

Right at this time, a new project came up to build a private health claiming solution. This was a new product that would integrate with our existing payment products, but also be distinct in many ways. Though it would need to integrate with the existing monolith, we made the call to develop as much as we could in new services. This resulted in three new artefacts. One was for handling online requests and was used by the software our payment terminals connected to. Two backend services were created for two different types of configuration that needed to be managed, one of which integrated with the payments backend monolith. The decision meant the team spent most of their time developing new software which other teams weren’t touching. There was almost no chance of conflict, except with the other three pairs on the same team.

Gotchas

The first thing to watch out for here is the same as when using microservices to resolve long build times. Avoid separating out something which requires some work right now, but won’t require much attention past the short term. If you do, you’ll only get temporary relief from teams stepping on each other’s toes before people return to working on top of each other in the monolith. Different services will (and should) have varying degrees of complexity and ongoing work based on how you define your bounded contexts. Some domains are simply more complex and/or will change more as the product evolves. If you’re prioritising to get teams off each other’s toes, you’ll want to look at separating out something that has a long-term development roadmap associated with it. That way, the team that builds it can stay in there and stay separate from other teams for an extended period.

The second thing to watch out for is creating a distributed monolith. By that I mean code deployed as separate components but which is tightly coupled in the way it operates. You don’t want development of features to regularly require changes across a large number of services. If it does, you’re still going to end up with multiple teams working on the same pieces of code and getting in each other’s way.

If you want to be notified when future articles are published, sign up using the form just under my photo.

Previous article in this series: Using Microservices to Solve Slow Build Times
Next article in this series: Using Microservices to Make Costly Deployments Less Frequent

Image credits:
‘Car Crash 7-1-18 2252‘ by Charles Edward Miller

The post Using Microservices to Solve Developers Stepping on Each Other’s Toes appeared first on Evolvable Me.

Switching to a new software architecture style is costly. I advocate choosing where to employ microservices first by using them to solve existing problems in the team. This helps the organisation extract value from the new architecture with each step towards the new world. This series is working through a set of problems that your org may have, and how microservices might help solve them.

Slow Build Times?

Does your team have an application that takes a long time to build and run tests? Is a large portion of the development team working on it? That could be a huge amount of idle time you’re pouring down the drain. The classic XKCD ‘Compiling’ comic makes light of developers enjoying the time they spend waiting for their code to build. The truth for most devs is they would far prefer to be making fast progress on their work than sitting around waiting.

If you can identify a slice of upcoming work that (a) would typically be added in where the slow build is, and (b) could be easily cordoned off into a new artefact, it might make sense to create a new service with a fast build and do the work there.

Here’s an Example

At Tyro, we had this slow build situation with our single internal webapp. Up to a certain point, all the features of our system used by all our internal staff were clumped into a single web application. It had a lot of screens and a lot of tests. The build took ages – 30+ min. on a beefy dev desktop. And that’s with all the calls to the backend API service mocked.

We were embarking on a new project that would re-write a number of the functions used by the Finance team. While planning, we identified the webapp build time as something that would slow us down. We decided that a new, empty web application – where we would build just financial functions – would allow us to move a lot faster. We:

• copy+pasted the existing webapp;
• deleted everything except the login and home page functions and the test framework;
• find+replaced the webapp name;
• bumped all the dependencies to the latest version; and
• started iterating with our sweet, sub-1-minute build.

No sword-fighting was required. It was a successful strategy that we repeated many times. Tyro now has a suite of smaller internal webapps serving different functions and business domains.

Gotchas

There’s two things to watch out for when breaking out microservices to solve slow build times.

Firstly, you want to ensure that the domain you carve out is one that’s going to see active development in the future. In particular, you ideally want it to be a piece that will see much more active development in the medium term than the pieces that you’re choosing not to break out. If you give some functionality a really fast build, but are still doing most of your development in the domain with the slow build, you’ve lost out.

Secondly, you want to ensure you actually get rid of the cause of the build slowness when you make the new service. In the example above, it was the sheer volume of features that made the build slow, so dropping to zero features dropped the build time to near zero. For other apps, though, the build may be slow for some other reason, like dependencies being downloaded too often, long-running steps like unoptimised Docker builds, or an integration testing suite with a long setup time. You may miss out on the build time savings if you repeat the mistakes of the past.

If you want to be notified when future articles are published, sign up using the form just under my photo.

Next article in the series: Using Microservices to Solve Developers Stepping on Each Other’s Toes

Image credits:
Snail from MaxPixel.net

The post Using Microservices to Solve Slow Build Times appeared first on Evolvable Me.

A common question that comes up for teams at around this time is:

“What should we split out into a microservice first?

And why?”

A lot of organisations start with a monolith and move to a more distributed architecture once the business has some traction. At the point where microservices becomes “a thing we want to do”, there can be many options for which parts of the product could be used as the pioneering experiments.

Over the next few weeks, I’ll be publishing a series of articles that looks at heuristics you can use to choose valuable parts of your system to separate out. First, though, let’s talk about the overarching heuristic that drives all of the suggestions in the following articles:

The #1 Heuristic for Your First Microservices: Choose a Problem to Address

You’re doing microservices for a reason, right? Maybe you have some problems already that you think it will address. Or maybe your foresee problems that will rise up soon if you don’t start doing things differently. You’ve decided to either split up your monolith, or do new work outside the monolith as much as possible.

My over-arching guideline for choosing what to attack first is this: You’re moving to microservices to solve problems, so your first microservices should be things which will solve your current problems.

You’re moving to microservices to solve problems, so your first microservices should be things which will solve your current problems.
Share on X

In the upcoming posts, I’ll cover a slew of problems which your organisation might already have identified. I”ll show how each of them could be solved as part of your migration to a microservices architecture.

Get a Return On Investment From Your First Microservices

One thing you want to keep in mind with each problem area I’ll cover is to always judge the return on investment. Just because you identify with one of these problems doesn’t mean that it’s a problem that your team must solve now.

Remember that the cost of deploying microservices is usually high for orgs who are new to this. You’ll have to figure out new approaches to deployments, monitoring, and security, among many other things. If you choose to attack a problem that has a small impact, you could find that deploying the functionality into a new service won’t pay off in the near term. Ideally, you’ll want to look for the service opportunities that are quick wins in terms of the value returned to the organisation.

However, there is also a counter-argument to consider here. Each new service deployed will give the team feedback and start building their muscle around deploying and operating a distributed system. That muscle will result in compounding returns in the long run. So you won’t get all of the benefits immediately, but ideally you’d love to get a large chunk of payback quickly.

Focus on Making Measured and Continual Progress

Finally, I recommend being careful about not chewing off too much at once. Don’t try to change too many things or tackle multiple hard problems in one go.

For example, let’s think of an organisation that’s building their first microservice, and everything in their monolith currently uses a single database schema. Such an org should consider making their first service something which doesn’t need a database. That way, they can focus on the problem of getting a second piece of software deployed into production, without the added concerns of getting a new schema into production and all the ripple effects that has on usernames, passwords, backups, and the like.

However, you also don’t want to leave those extra hard things too long. As soon as you’re comfortable with deploying new things that don’t have databases, move on to deploying something with its own database. If you leave the harder bits for too long, that extra challenge may start to appear too hard and become a mythological danger zone where no team wants to tread.

Problems That Could Be Solved By Your First Microservices

I’ll be adding each blog in the series in a list below as it is published. If you want to be notified when future articles are published, sign up using the form on the right just under my photo.

• Using Microservices to Solve Slow Build Times
• Using Microservices to Solve Developers Stepping on Each Other’s Toes
• Using Microservices to Make Costly Deployments Less Frequent

Image credits:
‘Project 365 #71: 120317 Box Of Cogs‘ by Pete (from Liverpool)

The post How to Choose Your First Microservices appeared first on Evolvable Me.

Our project was to effect an architectural change to our system, changing the source of truth of some data, moving it out of the database owned by a legacy monolith into a new database controlled by a new microservice. However, much of the code requiring the data would remain in the monolith.*

Some examples of the types of things we feature-flagged are:

• • whether to go down a refactored code path or not;
  • whether to publish messages to a message queue when a certain event occurred;
  • how to publish those messages (we tried multiple variations of batching and transaction boundaries to achieve acceptable performance);
  • whether to just delete messages at the receiving end or actually handle them; and
  • whether to use a local source of data or remote one.

We were working on a pretty important piece of code; the kind of business function where, if we stuffed it up, someone would probably have to spend several days doing remedial fixups or making phone calls to chase up millions of dollars. Hence, most of the feature flags we put in were to ensure we could test new code paths in production, and rollback safely & quickly. Some of them we even deliberately switched on for 10 minutes, let the new code run, shut them off again and then looked at the results offline. From memory, all of these flags were simple on/off switches, although we do have the capability to do per client/customer/category flags when we need to. These tips should work for either type.

Because we used so many different feature flags in a relatively short period of time, we got to experiment with and see the affect of a number of different patterns, both good and bad, and developed the following set of guidelines.

For context, in this particular app, our oldest and biggest monolith, most classes have a 1:1 unit test. That’s not our current preferred practice any more, but it’s a legacy we live with when working in this app. It also has a large collection of integration tests.

Goals for Feature Flags with TDD

There’s a couple of goals that we were aiming for as we did this work:

• • We want to make sure all currently tested paths that the flag affects are tested with the flag on and off.
  • We don’t want our test suites to become too large and/or hard to understand because of feature flags.
  • We want a git history that’s easy to understand despite the temporary complexity of a feature flag.
  • Ideally, deleting a feature flag will be a quick task that requires no in-depth reading of the code or tests.

Ideally, deleting a feature flag will be a quick task that requires no in-depth reading of the code or tests.
Share on X

Feature Flags and TDD: Our Five Guidelines

1: Make a copy of every affected test

The first step to ensuring tests don’t get unwieldy is to not try and test all the feature flag ON and OFF cases in one test. So: don’t open up the existing test file and start adding new “Feature X ON” test cases. Instead, make TWO test files: one for flag on, one for flag off. Yes, we’ll probably have a bunch of duplicated test cases that are exactly the same in both files, maybe even most of them, but this is only temporary.

There are three benefits to this approach: First, we’ve avoided creating a confusing monstrosity of a test with flag on, flag off, and flag-agnostic tests all mingled in together. Second, once we’ve decided to keep the flag on permanently, we can just delete the whole “flag off” test file. “Simples!” Thirdly, we can do the setup for the flag once at the top of each test file, rather than having to set it at the start of each test case and hoping people notice that detail when reading the test.

2. Make the copied file the test for the “Old” functionality, and the original file the test for the “New”

When we make a copy of the existing test to a new file, we want to make the new file the  tests for the old functionality – with the feature flag off (e.g. cp FooTest  FooPreFeatureXTest) – and change the original test file to have the test cases for when the flag is on. This helps to keep a contiguous SCM history for the enduring test suite showing how the functionality progressed.

The alternative is to put the “feature flag on” test cases in the new file, then, when we get rid of the feature flag, deleting the original test and renaming the “new” one to the “normal” one (e.g. mv FooWithNewFeatureXTest  FooTest). We found problems with this latter approach when we looked at our git history: it would often show the whole test being deleted then re-created, which makes it much harder to inspect what changes were made to the tests when the new change was introduced.

3. At the end of any test that enables a feature flag, always set the flag back to what it was before the test started

This is important because, if our flags are stateful in a way that isn’t automatically reset  between tests, we can end up with tests that don’t specifically turn the flag on or off then randomly running with the flag on depending on the order in which our tests are executed. For most tests this won’t make a difference, but for some it will and we could get failures in our build that don’t make sense because we can’t see from the single test’s code that the flag was left on by another test before it ran.

Note also the careful wording of this rule: we don’t always disable the flag at the end of the test; we set it back to what it was before the test. Being disciplined in resetting the flag is crucial to making the next tip work.

4. Run the whole build with the feature flag ON

Once we think we’ve created pre-feature and post-feature versions of all the test suites that we believe are affected by the feature flag, we need to run the whole build with the flag ON. This will flush out tests that are affected by the flag where we haven’t yet created a divergent suite. If we’ve been sufficiently comprehensive in our test modifications, nothing should fail, because all tests of paths that care about the flag should already be explicitly setting the flag in the test.

In a simple microservice, we’re probably unlikely to find anything with this step, although it should be quick to run so it’s still worth doing. On the other hand, in a complex monolith with many components, it’s quite possible that there are parts of the system that we’ve forgotten rely on the behaviour that we’ve just changed, and their integration tests may well fail as a result of the changes. We really want to find these transitive breakages before switching on the feature flag in a production system. If we don’t, we actually end up running an untested version of that dependent component in prod.

5. Switch on and delete feature flags ASAP

Feature flags are a device for assisting in controlled, reversible migration in production systems, but they also complicate the code. We want our code to be simple, so when there are feature flags in place, we make it a high priority to get them switched on in production. If we don’t prioritise switching them on, we can’t prioritise deleting them. This becomes especially taxing if we find the need to put other feature flags in the same area and start getting feature flag interplay.

The instant that a migration is completed in prod and we’re happy that the functionality should remain, the feature flags become technical debt. We want to remove them as soon as possible afterwards so that we revert our code to being as simple as it can be.

The instant that a migration is completed in prod and we’re happy that the functionality should remain, the feature flags become technical debt.
Share on X

Yo, what about the Strategy pattern?

When I shared some of the above tips on Twitter, a few people responded that they like to use the Strategy pattern when they’re doing feature flagging.

While I’m a big fan of the Strategy pattern, I would generally avoid it when feature flagging. My reasoning is that feature flagging is ideally a temporary measure, whereas the Strategy pattern is a design construct for supporting multiple implementations. While feature flagging often results in multiple implementations for a short time, I don’t think it’s worth introducing a design construct for this short period, only to delete it again shortly after, once the flag is removed. So we generally just used an if/else in all places that rely on the flag, knowing that the ‘else’ branch will be removed shortly. We would probably make an exception to this if the feature flag pertained to a large or complex piece of code that was being almost entirely re-written, and it was clear that readability would be greatly improved by having the two implementations in separate classes.

How do YOU do Feature Flags and TDD?

These are the main guidelines that our team developed through almost a year of feature-flagging our way through a complex piece of migration work. If you’ve got other tips for how the work with feature flags, particularly in a TDD environment, I’d love to hear about them in the comments below.

* I just want to note that this is not a good design, nor one that we were keen on implementing. It was a compromise forced by a data migration effort with a deadline and some very old code that couldn’t be extracted from a monolith easily.

Image credits:
‘Fork in the road‘ by Bs0u10e0
Archer by Unknown
‘Strategy pattern in UML‘ by Jason S. McDonald

The post Feature Flags and Test-Driven Design: Some Practical Tips appeared first on Evolvable Me.

After 12.3 years, yesterday was my last day at @Tyro.

It’s been a huge journey, from a perilous payments startup with less than 20 employees and only customer, to now a VC-backed, fully-licensed bank with well over 400 staff that is Australia’s 5th largest EFTPOS provider (by transaction volume).

I’ve done time as a Software Engineer, Development Lead, Engineering Lead (part of the management team), Technical Project Manager, Team Lead, Product Lead, and this last year, back to Software Engineer. I must have hired 50+ (awesome) people and interviewed many times that.

We listened to new voices in tech and followed their common sense advice. Often we only discovered years later that we were on the bleeding edge of movements like agile, DB evolution, microservices, and continuous delivery.

We built a bank.

From scratch.

In Java.

While still figuring out how to do microservices.

WE BUILT A FRIGGIN BANK!

And what I’ve come to realise is that every achievement on my LinkedIn profile while at Tyro will look like this:
“We…
We…
We…
We…”
Because it was all about the team, all the time. *I* achieved very little. Together, we’ve achieved so much. And that’s how it should be.

What’s next? It’s very exciting. A short break, then I’m starting my own business.

It’s no secret that I’m all in on #microservices. The “post-DevOps” style of architecture solves some big problems, but creates a bunch of new ones. Our mission will be to help teams solve those, pulling the brakes off microservices so teams can reach their full potential.

If you want to hear more about that journey, DM me your email address on Twitter with the message “Add me” and I’ll put you in my personal mailing list. (You should also turn on notifications for @TechDownunder!)

Finally, I want to confirm @BillGates’ observation: “Most people overestimate what they can do in one year and underestimate what they can do in ten years.” If you only stay in a company for a year or two, don’t expect to see any significant progress come out of your efforts. But if you hang around for a decade, you might just get to be part of changing an industry.

The post Time to Evolve Again appeared first on Evolvable Me.

We were 12 months in and were about to finish the project bang on the week we’d estimated, when the inevitable happened: we received a new raft of requirements that would add at least another 3 months of work. The extra work wasn’t glamorous; it was mostly security enhancements, disaster recovery preparation, and a complicated behind-the-scenes tweak of the product that added little benefit for users but was a requirement from a regulatory point of view.

Understandably, morale took a hit, but we needed to keep the pace up or we’d risk missing the deadline we’d been set for acquiring our license. After an interesting discussion with my father-in-law about the project, I found a new perspective on where we were at, and it prompted me to write the internal blog below to help motivate the team towards our final goal. A few people told me they appreciated it, and grumbling seemed to subside as everyone hunkered down and pushed towards the finish line.

Just recently, though, I was surprised when I overheard a colleague, in a team that is in a similar last-20%-grind scenario, recommending this three-year-old blog to a newer comrade and looking it up for them to read. Curious, I went back and had a re-read of it myself, and I found that what I thought was a point-in-time pep talk for a particular team, was actually a fairly general motivational piece.

So, whether you’re currently in a team slogging its way through the boring final stages of a project, or in the exciting first phases of something that will eventually end up in the same place, I hope you find this useful in granting you some perspective and grit for the long haul.

“This work kinda sucks right now”

Hi folks,

I’ve had a number of people comment to me over the last month or two that [Project T] work “seems to be drying up” or “we seem to be just doing lots of little things”.

I can understand where these sentiments are coming from. Over the last year, we’ve built…

• a transaction account, integrated with our payments services;
• a phone app;
• integration with Xero;
• a new accounting system;
• an on-phone registration experience;
• an application processing and customer support web app;
• account provisioning, orchestrated across a distributed system;
• bank statements and other crucial regulatory reports and features;
• undoubtedly a whole bunch of other stuff I’ve forgotten.

But now, over the last couple of months, each team has been moving onto the huge mound of security remediation work we created for ourselves. Some of this work has been large and somewhat interesting, for example the principal propagation with [technology redacted] piques my interest. Other parts have been less glamorous, like preventing log files from having newlines or dodgy data injected into them. (You win some, you lose some, hey [team who did the log filtering, but also some of the most interesting early work]?)

At the same time, many of us have been tying up a lot of loose ends: pact tests, tools for Ops to configure Rabbit MQ or firewalls, build processes for iPhones, preparing SIT and Production environments with test data, and documentation. Documentation, for goodness sake! How un-Tyro of us.

Big Projects

I was chatting with my father-in-law about this “large project” of ours last weekend. He used to build roads. Big roads. He built the F3 from Sydney to Newcastle. (Not by himself of course.) He has some great stories about tricking magpies to come and sit on explosives just when they were going to blow them. But I digress…

He said they had exactly the same circumstance when building freeways. At the start there was all kinds of interesting stuff going on: blowing up mountains, building bridges, levelling huge swathes of bushland and surfacing them with huge machines that were the latest technology.

As the project neared the end, though, it was the little things that started to become important. On-ramps to local roads were built, safety fences had to be erected, road signs put in place, paint laid down for the lines, and yes… documentation. The project tailed off with the little things, but they’re still crucially important. Without onramps, you can’t get on the road in the first place; without safety fences, the road is too risky in the infrequent but inevitable occurrence of an accident; without road signs, the road can’t be navigated and drivers might drive at unsafe speeds; without that paint marking the lanes: bedlam! These aren’t jobs anyone would write home about, but without each one being completed, not a single public car would be allowed to drive the road.

And so it is with our project. These “little things” we have been focussed on for the last few months, and will continue to focus on for a month or two ahead, are no less important than those big things that have come before. They’re no less important because, without doing them, we have nothing – we will have no license and without a license we have no product.

A few of you may have been down this road before, but some may not. Some who’ve been at Tyro for a while may have forgotten what the end of a project feels like, because it’s rare for us to batch this much functionality into a single “release”. The fact is, the end of projects always has this runoff, and it’s often a hard slog and seems like it’s never going to end.

I’ve been through this experience personally a couple of times in recent years, having developed and released both an app on the App Store and a commercial Atlassian Stash [now Bitbucket Server] plugin. In both situations, developing the product was only 50-70% of the work, and the rest was “admin”: creating icons, setting up accounts, writing marketing copy, making web sites – nothing to do with the interesting coding problems that got me started in the first place. It was super hard to be motivated while doing this stuff, especially as I was spending my minuscule spare time on it. I literally considered giving up a number times with each project, but I kept going, spurred on by the idea of finishing something significant and seeing it released into the wild, in other people’s hands.

So what am I really writing about here?

Three things:

Firstly: What is a Software Engineer? It’s more than a programmer, or a software developer. It’s a T-shaped role with very long sleeves, and that’s why we use that “Engineer” title rather than other ones. We don’t have architects or designers or DBAs or performance tuning gurus_[1] – the software engineers do all that. At the same time, we don’t have a security infrastructure team, or a security logging team, or an iPhone build maintenance team, because, as Software Engineers, we can stretch our arms to those skills as well. The role is broader than a full-stack developer – it’s a technologist that does a lot of programming but can also do lots of other things. Sometimes those things are big and exciting, sometimes they’re neither, but the people we hire here have the skills and the attitude to get stuff done that needs to be done. And you’re doing that, so well done.

Secondly: I think quite a few people are struggling to feel motivated about the work we’re doing right now. I get that. I’ve been there. Eight and a half years I’ve worked at Tyro now and I’m not going to pretend that every single day has been a mind-blowing celebration of awesome coding fun. One of the great things about Tyro, though, is that, a lot of the time, the work is interesting enough that it really is self-motivating: you want to get in to work each day and start coding on that feature or solving that problem. Right now, a lot of you may not be feeling that.

So, how can you stay motivated? My suggestion is to focus on the end goal, which is not that far off. Think about the collective thrill we’ll feel when the first pilot merchant deposits their hard-earned cash into a Tyro Account and then uses it to pay a bill through our app with just a PIN number and one tap. This is history in the making. That’s worth repeating: This is history in the making. You’re a part of it. The work you’re on right now might not be stimulating your mind, but without it we’ll never be able to step across that finish line and launch this awesome product that we’ve spent the last year on. This is the last 500m of a marathon. Keep your chin up, look at the goal, and keep moving.

Thirdly: What comes next? We’re clocking through this security work at the rate of one person-month per day. (Yes, between 6 [Project T] teams you do a month of work every day! It stresses me out!) What are we all going to do as we wrap this stuff up? Well, that is not up to me, and not entirely decided yet, but [the Head of Product] has lots of ideas for where to go next and is working right now on picking exactly what the next priorities are going to be. He’s kindly offered to do another “state of the [Project T]” session and share some of the ideas about where release 2 might head. We’re going to do this at the end of this week’s Tech Talk after lunch this Thursday.

tl;dr

• You’re Software Engineers. You can do anything, big or small. Sometimes that means doing small stuff.
• If the “what” of your work sucks for a little while, try to focus on the “why” and push ahead towards the goal. Stick in there. It’ll get better. (If it doesn’t, tell us so we can fix that for you.)
• [Project T] release 2 is right around the corner. Stay tuned.

Finally, if you want to talk about any of this, or about how you’re feeling about work, feel free to come and grab me for a chat any time.

By the way, we got across the finish line, and we got a banking license. We made history.

But that’s another story.

Image credits: Tired man running, public domain
‘Colleen is bored‘ by Jason Scragz
‘Excavator Working to Remove the Road Base on Rim Rock Drive‘ by NPS
‘Victorinox Swiss Army Knife‘ by James Case

_{[1] Since 2015, we have hired architects and DBAs, though delivery teams are still largely responsible for designing their own software and databases.}

The post “This work kinda sucks right now”: Staying motivated in the uninspiring phases of long projects appeared first on Evolvable Me.