Confused Life - Reloaded

Terminators at the Terminator

2026-08-07T17:00:00.000-07:00

The nicest thing about the word terminator is that it already sounds like a machine sent back from the future to explain your capital expenditure problem, or to make you realize the capital expenditure will not save you from a distant AI dominated dystopian doomsday.

In orbital mechanics it means something calmer and stranger: the line between day and night. A dawn-dusk sun-synchronous orbit rides that line around Earth, keeping the spacecraft in near-continuous sunlight. For Earth observation satellites this is useful because the illumination geometry stays predictable. For solar-powered orbital infrastructure it is more tempting: maximum duty cycle, fewer battery cycles, and the delicious thought that your data center can spend most of its life with the Sun on tap.

So yes, the terminators at the terminator. AI training and inference workloads running in low Earth orbit, fed by photovoltaics, cooled by radiators, and connected through optical links to ground stations and other satellites. A nice science-fiction phrase once you have solved the heat and materials balance to get up the gravity well.

The dawn-dusk sales pitch

The pitch is not silly. It starts from real constraints.

AI data centers are going to become giant electrical-to-heat converters with a side effect of model weights. Reminds me of the time when I used to dry my laundry on my bitcoin miners. On Earth we feed them through grids, substations, diesel backup contracts, cooling towers, chillers, water rights, fibre backbones, and increasingly awkward community meetings. In orbit you can imagine skipping some of that. Put the compute near continuous sunlight. Use high-efficiency solar arrays. Use direct-to-space radiators instead of evaporating water or warming a river. Place inference near space-borne sensors. Put pre-processing close to Earth observation instruments so the useful bits come down instead of the whole firehose.

NVIDIA’s Vera Rubin platform is the more immediate reference. Vera is the CPU and Rubin is the GPU, assembled with networking, memory, and rack-scale infrastructure as the successor generation to Blackwell. The roadmap promises another large increase in AI training and inference throughput, but throughput arrives with denser power delivery, cooling, and interconnect requirements. An orbital operator would inherit the useful work per watt and every unresolved watt of heat. Rubin therefore belongs in the radiator calculation as much as in the model-training benchmark.

Sensor-side inference offers a different use for that compute. The instrument looks down, the processor sits nearby, and the network carries detections, embeddings, tiles, compressed products, or model updates instead of raw sensor exhaust. A satellite constellation does not want to be a fleet of USB drives with antennas. It wants to be a distributed science platform with a power budget.

That is where dawn-dusk orbit starts to look like a scheduling primitive. Solar availability shapes the compute cluster. Training jobs, inference bursts, checkpointing, downlink windows, thermal soak, battery reserves, and radiator orientation all become part of the same planner.

The cloud scheduler gets an ephemeris.

The thermal model is the business model

On Earth, a data center engineer can cheat emotionally because air exists. It may be hot air, badly ducted air, or air pushed around by fans with punishing maintenance schedules, but it is there. You can move heat by convection. You can move it into water. You can bury part of the problem in a cooling tower plume and another part in a utility bill.

In orbit, there is no convenient atmosphere to carry the embarrassment away. Heat leaves by radiation, which puts the radiator alongside the processors and power supply in the core computer architecture.

The governing shape is brutally simple:

P = ϵσAT⁴

Radiated power depends on emissivity, area, and the fourth power of absolute temperature. Want to reject more heat without growing a ridiculous radiator? Run hotter. Want to run hotter? Your semiconductors, packages, solder joints, optics, interconnects, dielectrics, memory, and power electronics all need to tolerate it. A space data center is a semiconductor materials problem wearing solar panels.

This is why the recent attention around high-temperature semiconductors is interesting. Wide-bandgap devices such as silicon carbide and gallium nitride are already changing terrestrial power electronics because they switch efficiently and tolerate higher temperatures than conventional silicon in many roles. Push the idea further into ultra-wide-bandgap materials, diamond substrates, high-temperature packaging, and radiation-tolerant device physics, and the space qualified semiconductor variant emerges: hotter electronics can make every square metre of radiator work harder.

But that sentence hides several doctoral theses and a lot of broken hardware.

A transistor surviving a hot test coupon proves very little about the complete system. The whole compute stack has to behave: memory retention, timing margins, optical alignment, thermal cycling, electromigration, single-event effects, packaging stress, and the boring connectors that always become interesting at the worst possible time. Better high-temperature semiconductors help, but orbital data centers need a complete materials and packaging stack. A spectacular junction-temperature result does not provide one. We do not have the advanced PDK’s or foundries to do this today, but definitely a warm blue ocean of high temperature hardware to swim into.

Insulation is not the opposite of cooling

The intuitive mistake is to think space is cold, so cooling must be easy.

Space is not cold in the way a cold beer is cold. Space is mostly empty. A surface in sunlight gets blasted by roughly 1,361 W/m^2 before geometry and efficiency. A surface looking into deep space can radiate beautifully, but only if it is allowed to see deep space and only if the heat can get there.

So the architecture becomes a discipline of separation. Keep solar absorption away from the radiator side. Keep hot compute planes thermally connected to heat pipes, pumped loops, or thermal straps. Keep sensitive optics and timing hardware isolated from thermal gradients. Use multi-layer insulation where you want less radiative coupling. Use high-emissivity radiator coatings where you want more. Stop random structure from becoming a thermal short. Decide what gets to be warm, what must stay stable, and what can swing through orbital day-night transients without walking itself out of calibration.

In a dawn-dusk orbit, you buy continuous solar availability, but you do not buy a waiver from geometry. Solar arrays want one orientation. Radiators want another. Antennas and optical links want line of sight. Drag wants attention. Radiation shielding wants mass. Stationkeeping wants propellant or electrodynamic cleverness. The thermal design is coupled to everything else.

The radiator is not downstream of the business plan. It is in the first spreadsheet , or Claude conversation if you are not that good at maths.

Light moves from power source to processor

Orbital thermal constraints give optical computing a practical job beyond producing laboratory headlines.

The ENGtechnica write-up on Microsoft’s analog optical computer points at a useful direction: micro-LED arrays, spatial light modulators, and photodetectors doing vector-matrix operations in analog form, paired with a digital twin. The article claims promising results across image classification, nonlinear regression, MRI reconstruction, and financial modelling tasks, with Microsoft estimating up to 100x the energy efficiency of leading GPUs for suitable workloads.

The prototype is a specialised machine for particular AI inference and optimisation workloads, not a general-purpose replacement for digital computers. That narrow scope suits orbital compute, where general-purpose waste carries a physical penalty. Every joule that enters the box becomes a thermal export problem. Every ADC, DAC, memory movement, SerDes hop, and cache miss consumes both time and radiator area.

If you can do part of the workload optically, with fewer electrical conversions and less heat per useful operation, you change the orbital equation. Not enough to abolish thermodynamics, but enough to move a design from absurd to maybe annoying. That is a respectable improvement category in space engineering. Optical compute has the advantage of moving in fun new directions , couplint with the advances in quantum photonics or directly to laser comms module to get data off compute nodes to earth or between compute nodes in orbit.

There is a broader lesson here too. We keep treating compute as abstract symbol manipulation, then acting surprised when the factory bill arrives as heat. Landauer’s principle gives the tiny theoretical floor, kTln 2 per irreversible bit operation, but modern compute lives many floors above that. The practical losses are in data movement, leakage, switching, conversion, cooling overhead, and the human decision to train another model because the previous benchmark table had one unclaimed column.

Bits are not immaterial. Organising them has a thermodynamic cost.

Training, inference, and the ugly logistics of usefulness

Orbital AI training sounds glamorous, but inference is probably the more immediate fit.

Training wants enormous, tightly coupled clusters, rapid hardware refresh, dense networking, fault tolerance, huge data ingestion, and painful amounts of memory bandwidth. It wants technicians nearby when the expensive thing throws an unhelpful error. It wants the newest accelerators, because model economics are tied to performance per watt and performance per dollar, and both curves move quickly.

Inference near sensors is cleaner. Detect the smoke plume, ship the alert. Find the illegal fishing vessel, ship the track. Segment the flood boundary, ship the polygon. Compress the asteroid candidate stream before the downlink becomes the bottleneck. Turn imagery into geospatial events and only send raw data when the event deserves it.

Training still has a role, especially for models tuned to orbital sensors, onboard compression, anomaly detection, or autonomous operations. But the training argument has to survive hardware obsolescence. A terrestrial data center can swap GPU generations with forklifts, technicians, and a recycling vendor who may or may not deserve the word recycling. An orbital data center has a worse version of the same problem: the expensive solar arrays, radiators, pointing systems, communications hardware, and orbital slot may still be useful after the compute payload becomes embarrassingly old.

That is the stranded asset problem in orbit.

The answer cannot be to throw away the whole platform every time NVIDIA, AMD, Intel, Cerebras, Groq, Google, Microsoft, or some optical-compute startup moves the Pareto frontier. The more plausible architecture separates relatively static infrastructure from rapidly ageing compute: power bus, radiator fields, optical comms, attitude control, and docking interfaces on one side; replaceable compute, memory, and storage modules on the other.

In other words, orbital data centers need boring maintainability before they need cinematic scale like on the cover of latest IEEE Spectrum.

Close the silicon loop or don’t build it

A space data center that cannot be serviced is just delayed debris making a business case for companies like Paladin Space. When a territory politician asks over dinner why she should care about space, I can scare her by saying if she doesn’t orbital data centers will be falling through her roof. A risk that is uninsurable since the probabilities cannot be measured, and not suable against because the counter party is in China.

If we build orbital compute infrastructure, the end-of-life story has to be in the architecture from the beginning. Decommissioned compute modules need capture, removal, refurbishment, recycling, or controlled disposal. Radiator booms and solar arrays cannot be left as heritage sculpture in useful orbits. A failed training cluster should not become a fragmentation risk with tensor cores.

This is where orbital manufacturing stops being a novelty and starts looking like maintenance infrastructure. Silicon manufacturing in space has been discussed for reasons that range from microgravity crystal growth to vacuum processing and contamination control. Some of those benefits may turn out to be niche. Some may be overwhelmed by launch logistics and process control. But if the orbital asset base grows, the question changes from “can space make a perfect wafer?” to “which parts of the silicon, packaging, repair, and recycling loop are worth closing off Earth?”

Maybe the first useful loop is not wafer fabrication at all. Maybe it is inspection, module swap, radiator cleaning, connector replacement, board-level refurbishment, propellant top-up, or recycling aluminium structures. Maybe high-value semiconductor steps stay terrestrial while bulky static infrastructure stays orbital. Maybe microgravity does matter for particular crystals, optical components, or thermal interface materials. The answer should be found by process engineers, not by a launch animation.

But the principle is clear enough: keep the slow, massive, durable assets in place; rotate the fast-moving compute through them. Do not strand the power, cooling, networking, and orbital mechanics every time the accelerator roadmap changes.

The Earth is in space too

The tempting conclusion is that orbital data centers are exotic because they have thermodynamic problems. I think the opposite is true. They are interesting because they make the thermodynamic problems visible.

Earth-based data centers are also bounded by energy, entropy, materials, and waste heat. They just have more ways to export the consequences. A conventional data center can dump heat into air, water, district heating loops, neighbouring property, local weather, or a permitting process. Private property law lets the operator draw a fence around the useful part of the machine and call the heat someone else’s problem.

Physics does not recognise the cadastral boundary.

At planetary scale, Earth cools the same way an orbital data center does: by radiation to space. There is no cosmic cooling tower waiting outside the atmosphere. Add energy demand, trap more outgoing infrared, move heat around with clever plumbing, and the final balance still ends at the top of the atmosphere.

The terminator dual-meaning feels like a meme pill I have given myself. A data center on the dawn-dusk line cannot hide behind convection, and a data center in a suburb cannot hide behind property law forever. In both cases the question is the same: how much useful order can we create per joule, what materials make that possible, where does the waste heat go, and who is responsible for closing the loop when the hardware ages out?

Orbital data centers may one day make sense for particular AI inference, Earth observation, autonomy, and maybe specialised training workloads. But they will only be serious if they are designed as thermodynamic systems: power, compute, radiators, shielding, networking, servicing, debris removal, and silicon loops all in one ecosystem.

The Earth is in space too, and radiation is the only way the planet pays its final cooling bill.

Finding the Edge in Japan

2026-07-22T17:00:00.000-07:00

Tsukiji Market smells like grilled seafood, wet stone, soy, tourists and logistics. Somewhere between the fish stalls and the slow current of people carrying breakfast on sticks, I walked into a knife shop and found a wall of edges pretending to be a retail display.

The commitment was not subtle. Rows of Santoku knives sat behind glass with the calm seriousness of lab instruments. Not the Don Quijote souvenir version, but the shops where the object is expected to do work for decades if you respect it. Around USD 250 felt like the going entry point for a decent Santoku in the places we wandered into, and it rose very quickly from there.

Then in Kyoto, the scale changed. A sword shop had a katana listed at 7 million yen. At that price you are no longer buying a long sharp object. You are buying metallurgy, lineage, polishing, paperwork, scarcity, and a whole cultural machine for deciding that a particular edge has survived long enough to deserve reverence.

Knives in Tsukiji Market

We visited two knife shops in Tsukiji, a sword shop in Kyoto, and another knife shop near the Pokemon Center in the Takashimaya building in Shibuya. The geography helped the idea settle. Tsukiji had the working-market feeling, knives as seafood infrastructure. Kyoto carried the ceremonial weight of blades as history. Shibuya had the polished department-store version, still serious, but now sitting in the same urban stack as Pokemon, fashion, food halls and escalators.

Japan does this particularly well: the ordinary tool is allowed to become obsessive without becoming kitsch. A kitchen knife can remain a kitchen knife while also being a small thesis on steel, heat treatment, hand skill, regional specialisation, and daily maintenance.

Sharpening as Culture

The part that caught me most was the sharpening. The edge is the temporary part of the knife. It is the part you create, lose, recreate, neglect, rescue and eventually learn to feel before you can see it. A beautiful blade with a dull edge is mostly theatre. A modest blade with a good edge is useful.

That is a very Japanese thing to make visible in a shop: the maintenance is not a sad afterthought after the purchase. It is the point. Stones, angles, pressure, slurry, patience. The object keeps working because someone returns to the boundary between metal and material and restores it.

The one time I got into trouble in Japan, amidst mostly friendly people, was in Tsukiji when I accidentally bumped someone sharpening a knife. There are mistakes that can be fixed with a smile, and there are mistakes where the animal part of your brain notices the blade first. I walked backwards slowly, muttering “gomen” with every available ounce of tourist sincerity.

Santoku knives behind glass

There is a pleasing physics to this. Cutting is not brute force. Cutting is pressure concentrated along a thin enough line that the material gives up locally. The wedge finds a gap, opens it, and moves through. When the edge rolls or blunts, the same hand and the same steel suddenly need more force, more damage, more mess.

That is where the metaphor starts doing actual work instead of sitting on the mantelpiece.

Sharp Words, Sharp Tools

We use knife language everywhere when we talk about thought. Smart people are sharp. Arguments have an edge. Teams lose their edge. We sharpen our axe when we build up skills before attacking the next task. We cut scope. We carve out time. We slice a problem into pieces. We decompose something hard into smaller pieces so the first incision becomes possible.

Doing new things often feels like pushing a blade into something that does not yet have a natural opening. You do not smash the whole surface. You look for the gap. You test the grain. You find the weak plane, the softer layer, the place where a small movement changes the geometry of the rest of the problem.

Software architecture has this hidden inside it. Debugging has it even more. A failing build, a strange sensor trace, a dataset that refuses to behave: none of these usually yield to force. You find the narrowest question that can split the problem. Does the voltage sag under load? Does the request fail before authentication or after it? Does the geospatial reprojection break before tiling or during the write path? A good diagnostic question is a sharp tool.

The wrong question is a dull knife. It still consumes effort, but mostly by bruising the work.

The Robot With an Old Edge

Then, at the space forum, I met Abhay, who has been working with a 1980s knife-sharpening robot.

That sentence felt like Japan had followed me home with a punchline.

The theme was suddenly not just knives as craft objects, but knives as an interface between craft and automation. A sharpening robot from the 1980s is old enough to predate most of our current robotics hype cycles, old enough to belong to the world of industrial control cabinets, teach pendants, custom jigs, limit switches, and stubborn mechanical repeatability. It has to keep the angle, pressure and abrasive contact honest while the steel pushes back.

Sword display in Kyoto

That is the useful tension. Sharpening looks simple until you try to automate it. The robot has to hold an angle, manage pressure, follow a changing profile, deal with abrasive wear, heat, burr formation and the awkward fact that every blade has a history. The edge is not a CAD line. It is a lived boundary on a piece of steel. The exact usage of the knife accumulates and determines its fate, reality is a stochastic process.

This is why robotics remains humbling. The easy demo is motion. The hard part is contact. Once a robot touches the world, the world starts negotiating back. Knives make that negotiation visible because the result is brutally honest. The knife is either sharp, or it is not. The tomato knows, as does the audience watching a Unitree humanoid lose the dance floor, or the Qualcomm Dragonwing IQ10 robotics platform show up in an ET NOW clip of a mid-presentation collapse.

Tools That Teach Hands

There is a lazy version of automation where machines replace skill, and a better version where machines expose what skill actually contains. A knife-sharpening robot is interesting because it cannot escape the craft problem. It has to encode something a human sharpener may feel through wrist, sound, drag and accumulated failure.

That makes it a useful object to think with. Japan’s knife shops showed the cultural end of the spectrum: reverence for the edge, the maker, the steel and the ritual of maintenance. Abhay’s robot showed the industrial end: repeatability, fixtures, kinematics, control loops and the stubborn desire to mechanise a subtle hand skill.

Between those two ends sits most serious engineering.

We do not build new things by being generally clever. We build them by maintaining edges. The edge of a discipline. The edge of a toolchain. The edge of a team that still remembers how the machinery works beneath the interface. The edge of a question sharp enough to enter a hard problem without shattering it.

The holiday version of Japan gave me knives behind glass, stones on counters, a 7 million yen katana, and Santoku knives priced like serious instruments rather than kitchen accessories. The space-forum version gave me a 1980s robot still trying to make an edge repeatable.

That is the thought I wanted to keep: sharpening is not preparation for the work.

Sharpening is part of the work.

EO Platforms, Science Platforms, and the Apartment Between Binder and Earth Engine

2026-06-25T16:30:00.000-07:00

When you think of running code in the browser, you probably think of Binder or Google Colab. If you are at an AWS house, you think of Sagemaker.

That is the visible end of the digital science platform story: a notebook appears in a tab, Python imports something, a chart lands on the screen, and for a moment the whole research infrastructure problem looks solved. If you are fan of a different paradigm, Python can be replaced with R, or a programming language replaced by a chat session which has its side effects as Science/Engineering plans and some code.

Then the real workflow arrives. The data is 100 TB of imagery from a High-Altitude Balloon and it is sitting at the HPC; you will have to wait in the queue for the job to submit, but the proposal deadline is tomorrow and you really need preliminary exploration graphics to illustrate your story, since humans are inherently visual creatures. Do you fake it till you make it using Nano Banana, or do you commit to a fraction of the actual work you are applying for the funding to do in full?

The HPC credentials are institutional. The analysis needs Dask workers near cloud object storage. It will take two hours for Globus to haul the data over to the nearest AWS Region. The output, when the grant is completed, should become a tile service, a STAC collection, a policy brief, a model run, and a thing someone can repeat six months later when the next sensor collection lands after that much needed CMOS and Lens upgrade. For now you really need that plot showing how a thermal mosaic align with the visible spectrum satellite imagery.

At that point the browser notebook is the doorway, not the platform. The doorway opens onto a howling void of needs and unfilled expectations.

The apartment, the hotel room, and the empty block

The continuum feels a bit like the Django versus Flask argument, except measured in satellite scenes, climate arrays, grant budgets, and institutional risk. If you don’t know what Django or Flask is, perhaps opinionated vs minimalistic are more understandable.

A managed science namespace is the Flask or FastAPI option. You get a well-tested core, working access controls, shared storage, a scheduler, a route to scalable compute, and enough freedom to design your own universe. It is like buying an empty apartment in a good part of town. The walls are sound. The plumbing works. The rooms are yours, you will not get robbed if you step outside.

A highly managed platform is closer to a hotel room at the Ritz. Everything is convenient, polished, and serviced. You get a desk, a bed, a view, and a button to press when something breaks. You probably cannot knock down a wall.

Then there is the empty block of land: Kubernetes, object storage, IAM, container registries, workflow engines, data catalogs, and a team with enough scars to make them all behave. Maximum control, maximum blast radius.

The interesting work sits between those extremes. Digital science platforms are not one product category. They are a gradient of convenience, control, reproducibility, and institutional memory.

The first split: interactive science versus production exploitation

Earth observation platforms and broader science platforms grew from different pressures.

Science platforms usually start with researchers. They need notebooks, environments, collaboration spaces, package installation, teaching material, shared compute, and a path from experiment to publication. The center of gravity is interactive work: Jupyter, RStudio, Dask, documentation, executable papers, and a community that can teach the next person how to reproduce the result.

EO exploitation platforms usually start with data gravity. They need catalogs, tiling, pixel access, area-of-interest filtering, atmospheric correction, process graphs, long-running jobs, and application packaging. The center of gravity is operational work: find the scenes, run the process near the archive, publish the output, and make the result portable across platforms.

In practice those two worlds are merging. The Pangeo stack wants cloud-native scientific computing. EOEPCA wants interoperable EO exploitation building blocks. Digital Earth platforms want curated national-scale data cubes. NASA-style platforms such as MAAP and VEDA want science teams to move from discovery to analysis to communication without rebuilding the pipes each time.

Choosing a winner is not the problem. Knowing which layer you are buying is.

A rough map of the ecosystem

The ecosystem makes more sense sorted by the job each platform does rather than by brand.

Layer	Main job	Examples	What you gain	What you give up
Ephemeral notebooks	Launch a runnable environment from code	Binder, repo2docker-backed demos	Low friction, great teaching surface	Weak persistence, weak governance, limited scale
Hosted notebooks	Put compute in a managed browser workspace	Google Colab, Kaggle notebooks, SageMaker Studio Lab-style services	Fast start, familiar interface, low admin load	Runtime limits, cloudy provenance, platform-specific behavior
Community hubs	Give a research community a managed shared workspace	2i2c hubs, JupyterHub deployments, Pangeo hubs	Shared identity, shared environments, scalable Dask, community ownership	Still needs platform stewardship and cost discipline
Open science practice layers	Teach teams how to work reproducibly	OpenScapes, Project Pythia, Jupyter Book, MyST	Social machinery for reproducible science	Not a compute platform by itself
Cloud-native geoscience stacks	Make large arrays workable near object storage	Pangeo, Xarray, Dask, Zarr, Kerchunk, Intake	Open, composable scientific computing	You assemble and operate the stack
Data cube and EO libraries	Organize analysis-ready EO collections	Open Data Cube, xcube, stackstac, odc-stac	EO-shaped data abstractions	Needs catalog, storage, and operations around it
Managed planetary platforms	Bring data catalog and compute together	Google Earth Engine, Microsoft Planetary Computer	Convenient data proximity and APIs	Platform dependence and varying escape hatches
Standards-based EO exploitation	Package processing for portability	openEO, OGC API - Processes, EOEPCA, CWL-style application packages	Repeatable jobs across back ends	More ceremony and specification work
Science mission platforms	Align infrastructure to a mission community	NASA MAAP, NASA VEDA, AquaWatch, ESA EarthCODE-style efforts	Domain-specific data, workflows, and communication	Mission boundaries shape what is easy
Institutional digital earths	Curate national or sectoral data products	Digital Earth Australia, Africa, Pacific, and related platforms	Trusted local data products, policy relevance	Sustained funding and data governance required

That table is too tidy, of course. Real platforms straddle rows. A Pangeo hub can be a community hub, a cloud-native geoscience stack, and the working surface for a Digital Earth program. Earth Engine is both a managed planetary platform and a workflow language. EOEPCA is both architecture and open-source building blocks. A Digital Earth platform can host notebooks, APIs, dashboards, STAC catalogs, and products built from Open Data Cube.

The taxonomy still helps because “we need a science platform” can mean a teaching hub, a production EO processing system, a mission workbench, a public communication portal, or a Kubernetes tenancy model for research groups. Those are different purchases.

Pangeo is a community before it is a stack

Pangeo is often described through its software: Xarray for labelled N-dimensional arrays, Dask for parallel compute, Zarr for chunked cloud-friendly storage, Jupyter for interactive work, Kerchunk for virtualizing old file formats, Intake for catalogs, hvPlot for quick visualization, XPublish for serving datasets, and newer bounded-memory or serverless experiments such as Cubed and Xarray-Beam.

What goes into the requirements / constraints file of a notebooks repo is all and good, but like all code its value really depends on the people who read it and run it, Anne Frank’s diary is no good, if it is never found and read. The written word of any form be is a notebook or a painful diary, reaches its true potential when it transmits the ideas contained there in to the minds of thousands of others. Pangeo describes itself as a community for open, reproducible, scalable geoscience. Project Pythia is its education working group and a training hub for the geoscientific Python community. The infrastructure, documentation, showcase talks, working groups, cookbooks, and foundation tutorials are part of the platform. Project Pythia can be found, it has a lot of words, but no concrete infrastructure to run on, no printing press to replicate the proposed patterns and keep publishing them across planning cycles.

Many institutional platform programs miss this. You can install JupyterHub and Dask Gateway in a week. You cannot apt install a culture of reproducible workflows, documented environments, data access patterns, and examples that match the local science questions.

2i2c and OpenScapes occupy this gap. 2i2c turns the hub into managed open infrastructure with access control, custom environments, scalable compute, and a “Right to Replicate” philosophy, which is a useful antidote to the managed-service trap. OpenScapes works on the human operating system: habits, onboarding, documentation, and shared practice. The difference shows up when one champion leaves and the platform either survives or becomes a quiet monthly cloud bill, that everyone questions every quarter and decides to shutdown in a much “killed by Google” fashion. It have shut down a service or two in my time as well.

Earth Engine is the polished hotel room

Google Earth Engine remains the reference point for convenient planetary-scale EO analysis. It combines a multi-petabyte public catalog, server-side geospatial computation, JavaScript and Python APIs, and a browser code editor that made remote sensing feel weirdly immediate.

The trade-off is the same one that makes it attractive. You get the hotel room. You do not own the building.

For many users that is exactly right. If the job is land-cover change detection, trend mapping, rapid prototyping, teaching, or humanitarian analysis, Earth Engine collapses a brutal infrastructure problem into an API, and it has unlocked a large amount of science and public-interest work.

A national agency or regulated enterprise eventually asks harder questions. Can we run this against restricted internal data? Can we reproduce the process outside the platform? Can the cost model survive operational use? Can the workflow be audited? Can the output become an internal service rather than a script in someone else’s editor? What if Google does not like the economics and GEE gets “Killed by Google” ?

Those questions mark the boundary between convenience and control.

Planetary Computer, STAC, and the cloud-native middle

Microsoft Planetary Computer pushed a different shape: open catalogs, STAC metadata, cloud-hosted environmental datasets, Hub-style analysis environments, and APIs that fit the broader Python geospatial ecosystem. It did not only put data in the cloud; it made the catalog and assets legible to ordinary tools.

That middle ground is where STAC, Cloud Optimized GeoTIFF, Zarr, GeoParquet, and object storage conventions earn their keep. STAC calls itself a common language for geospatial information, with a core JSON structure for describing and cataloguing spatiotemporal assets. That sounds dull until you have inherited five satellite archives, three naming conventions, and one research assistant who left for a postdoc in Bremen.

These are the unglamorous freight standards of cloud-native geospatial work. Once data is published in these shapes, platforms compete on experience and operations instead of trapping users at the file boundary.

The same applies inside Digital Earth programs. Open Data Cube, odc-stac, xcube, stackstac, and Pangeo tooling become more powerful when the data model is boring and inspectable. Boring is good. Boring is how a wet-season flood product, a crop monitoring workflow, and a methane detection experiment can reuse the same infrastructure without pretending they are the same science.

openEO and EOEPCA are the uncomfortable middle

At the other end from free-form notebooks are workflow and process standards.

openEO gives users a unified API for Earth observation cloud back ends. It is a process graph world: define operations, submit work, let the back end execute near the data, and avoid rewriting the same analysis for every platform. In 2026 openEO became an OGC Community Standard, which is a useful signal that the idea has moved beyond a single project vocabulary.

OGC API - Processes handles a related need: wrap computational tasks as executable processes that a server can offer through a JSON-over-HTTP Web API. It is the unromantic contract that lets processing become a service instead of a notebook cell someone has to rerun by memory.

EOEPCA then packages a broader exploitation platform architecture around this world: reusable building blocks, open standards, federation between EO cloud platforms, application packaging, identity, data access, processing, and publication. Its mission is practical: enhance interoperability between cloud-based EO platforms and unify the fragmented cloud ecosystem for ground segment, EO science, R&D, and applications.

This middle is uncomfortable because it asks scientists to describe workflows with more ceremony than a notebook while asking infrastructure teams to expose more flexibility than a fixed portal. That ceremony earns its keep only when portability, audit, repeatability, and multi-platform execution are real requirements.

MAAP, VEDA, and AquaWatch show mission-shaped patterns

NASA’s Multi-Mission Algorithm and Analysis Platform (MAAP), Visualization, Exploration, and Data Analysis (VEDA), and AquaWatch point at another category: mission-shaped science platforms.

MAAP is built around collaborative algorithm development and analysis for biomass, carbon, and related mission data. It combines data, algorithms, and cloud computation for Earth science work at scale, including GEDI, ICESat-2, BIOMASS, NISAR, and related mission products. Its centre is not a generic notebook. It is a community of practice around specific data products, algorithms, and validation work.

VEDA is more communication and exploration oriented: science data, dashboards, stories, APIs, and applications that help users move from dataset to visible public insight. Its public dashboard talks about cloud-enabled analysis without writing code or downloading data, plus stories for science communication. In the geospatial agent stack draft I have been calling this the presentation and evidence layer: not just compute, but a way to package outputs for humans without losing provenance.

AquaWatch is the mission model closer to my own recent work: water quality intelligence rather than a generic EO workbench. Its platform shape is dictated by the problem. Satellite products, in-situ observations, calibration and validation workflows, data services, state-government users, and operational water decisions all have to sit in one fabric. Nobody wants an elegant notebook if the harmful algal bloom window has already closed and the river manager is still waiting for a plot.

Mission platforms should not be embarrassed by their specificity. A platform that knows its science domain can make strong defaults: data already mounted, examples already relevant, units already sensible, common plots already nearby, and metadata that matches the decisions people actually make.

Digital Earth platforms are institutional memory

Digital Earth platforms are a cousin of science platforms, but with more institutional weight.

Digital Earth Australia, Digital Earth Africa, Digital Earth Pacific, and related efforts are not merely places to run notebooks. They are long-running data product factories, national or regional knowledge systems, and trust infrastructure. They turn raw satellite archives into analysis-ready data and derived products that policy, environment, agriculture, water, and disaster teams can use.

The local flavours carry the point. Digital Earth Australia talks about trusted imagery and freely available analysis-ready datasets for researchers, land managers, agriculture, and emergency management. Digital Earth Africa frames the work as decision-ready products for sustainable growth across social, environmental, and economic challenges. Digital Earth Pacific is explicitly about decision-making for Pacific peoples at EO scale, with regional products, dashboards, an analytical hub, data, community, and governance in the same frame. This is not just a different skin on the same JupyterHub.

If you need a global flavour you can check out EASI from CSIRO under the AquaWatch program delivery, it flips the regional data cube approach to provide a global datacube from STAC using cloud native primitives. Almost a build your own Google Earth Engine approach, with associated DevOps and Platform Engineering burdens which you have to pay for to get.

That is a different job from a research notebook service. The value is not that one scientist can compute something, but that many people can rely on the same corrected, versioned, documented product line. Open Data Cube sits under many of these programs as an open-source way to manage analysis-ready data and has been used for continental-scale products such as Australian land-cover mapping from decades of Landsat imagery.

Here the apartment analogy changes. A Digital Earth platform is not an apartment you furnish. It is an apartment building with strata rules, fire exits, a maintenance schedule, and residents who will complain if the lift stops working, and I have been a super on such apartment buildings across 3 platforms now.

What must be portable?

Most platform debates get stuck on user interface preference. Notebooks versus portals. Python versus JavaScript. Managed service versus Kubernetes. Proprietary convenience versus open stack purity. The relevant debate is around what must be portable.

If only the result is needed, a polished managed platform may be perfect. If the method must move between agencies, clouds, and missions, then STAC, openEO, OGC API - Processes, CWL-style packaging, and open data formats become non-negotiable. If the community must outlive one grant, then docs, examples, governance, and training are platform features. If restricted data is involved, then identity, audit, and network controls are part of the science platform, not enterprise up-sell (don’t get me started on the SSO premium most SaaS platforms charge).

There are at least five kinds of portability hiding inside this one word:

Data portability: can assets move or be read by standard tools?
Workflow portability: can the analysis run on a different back end?
Environment portability: can the software stack be rebuilt?
Evidence portability: can another person inspect how the result was produced?
Community portability: can another team inherit the practice without oral tradition?

The best platform choice depends on which of those you actually need.

Agents make the old platform boundary fuzzier

Agentic tooling is the next complication.

Once agents can call a STAC API, resolve a place name, submit an openEO job, monitor an Argo Workflow, query a Digital Atlas, open a notebook, and draft a VEDA-style story, the visible platform recedes behind the contracts underneath it.

Portals do not disappear. The durable value moves down into services, catalogs, workflow engines, identity boundaries, and audit trails, and the UI becomes one client among many.

This is where EO and enterprise AI rhyme. A user should be able to ask a normal question:

Show me whether wet-season surface water around this catchment is outside the last ten-year range, and give me the data sources and caveats.

The system behind that question might need a gazetteer, a STAC catalog, a data cube, Dask workers, an openEO process graph, a workflow engine, a notebook artifact, and a short written summary. Calling all of that a chatbot misses the point. It is a digital science platform with a language-shaped front door.

Science Platform advantage

There is a less glamorous question hiding under the agent story: who operates and pays for the tooling and compute fabric the agents call?

If a user asks the wet-season surface-water question and an agent fans out across a slew of tools, somebody owns every link in that chain. Somebody pays for object storage requests, scheduler time, network egress, GPU or CPU minutes, model tokens, human review, and the maintenance of the small apis that make the orchestration possible.

A serious science platform needs weight propagation in both directions. Backward propagation attributes each output to the tools, datasets, people, grants, teams, and compute fabric it consumed. Forward propagation estimates what the next run will cost before the button is pressed. Without both, the presenter of the agentic tooling with customer touch point gets all the glory and gets to keep all the revenue, while the sustaining ecosystem underneath withers on the vine and the agentic tool ends up being a toy in the long run.

This is where the phrase “Science Platform advantage” starts to be useful, in the same way people talk about quantum advantage. The claim is not that a platform is clever. The claim is that a connected, orchestrated chain of task-specific tooling, assembled on demand and cost-tracked properly, delivers more value than doing the same work any other way.

For the utilitarians in the room, that claim needs a benchmark. Pose a real problem. Solve it fully the old way first: portal clicks, downloads, hand-written scripts, email threads, queue waits, manual plots, confused provenance, and the final slide dropped into a proposal at 11:47 pm. Then build the agentic way and run the same class of problem a thousand times, so the cost of building the new fabric is amortized across many analyses rather than charged to the first heroic demo.

The trouble is that no one was keeping clean accounts while we did it the old way. We rarely know how much the portal-clicking cost once staff time, failed runs, duplicate downloads, local storage, interpretation drift, and meetings are counted. So we cannot honestly promise cheaper or faster without measuring both sides. We can be more certain about the energy bill: silicon currently delivers far less useful scientific reasoning per watt than the human brain fed on caffeine and croissants.

That does not weaken the platform argument. It makes the accounting part of the platform. The useful question is not “can an agent do this?” The useful question is whether the platform can tell us what it did, who paid, who benefited, and whether the next thousand runs beat the old way by enough to justify the extra silicon. Will the environmental solution we are building preserve the environment or become yet another nail in the anthropogenic coffin we are putting around our planet.

My current bias

I lean toward the apartment model.

Give science teams a managed namespace with identity, storage, standard catalogs, scalable compute, observability, and a small number of blessed paths to production. Keep the walls movable. Let the team bring Pangeo, Open Data Cube, xcube, DuckDB, GDAL, QGIS plugins, or specialist models where the science demands it. Make the durable contracts boring: STAC, COG, Zarr, GeoParquet, openEO, OGC APIs, container images, reproducible environments, and documented workflows.

Use the hotel room when speed beats ownership. Use the empty block only when the institution is ready to become a platform operator. Everything else is a furnishing choice.

EO platforms and science platforms are not rival tribes. They are different answers to the same infrastructure question: how much freedom do you need, how much responsibility can you carry, and what parts of the science must still make sense when the browser tab is closed?

Sources and threads to pull next

Pangeo and Project Pythia for the open, reproducible, scalable geoscience community and its education layer.
2i2c and OpenScapes for managed open infrastructure, community hubs, right-to-replicate thinking, and reproducible research practice.
STAC, OGC API - Processes, openEO, and EOEPCA for catalog, process, workflow, and platform-interoperability contracts.
Google Earth Engine as the canonical managed planetary-scale EO analysis environment.
Microsoft Planetary Computer as the open-catalog, cloud-hosted environmental data, and scientific API pattern.
NASA MAAP, NASA VEDA, and AquaWatch as mission-shaped platform patterns around collaboration, analysis, visualization, communication, and operational water-quality intelligence.
Digital Earth Australia, Digital Earth Africa, Digital Earth Pacific, CSIRO EASI, and Open Data Cube for institutional data product platforms and analysis-ready data infrastructure.

We Don't Have Hooves: AI Logos, Cognitive Shedding, and the Exoskeleton Economy

2026-05-29T17:00:00.000-07:00

I spent a weekend writing a small Rust tool that composites AI company logos onto a Redbubble full-print t-shirt canvas. Not because anyone asked. Halfway through tuning the layout I noticed something faintly absurd: every logo I was tiling looked like a tyre tread. OpenAI’s swirl is a directional rain pattern. Gemini’s four-pointed star is a Bridgestone Blizzak winter sipe. Anthropic’s ringed A is a sidewall stamp. Nvidia’s eye is a hub cap. Lay them next to each other on a 7000-pixel canvas and the whole thing reads less like an industry portrait and more like a Beaurepaires catalogue page.

Logos From Loops

The visual grammar of AI companies has converged on a small family of forms: radial symmetry, interlocking arcs, gradients from cool to warm, the occasional offset chevron. They are tread patterns. They are tread patterns because tread patterns are what you draw when you want to suggest motion, grip, and continuous contact with a surface you are moving over – which is exactly the brand promise of a model that turns prompts into outputs without you needing to know how the road underneath works.

The tool generates 21 of these tread-pattern logos – the leaders, challengers, niche players, and visionaries from the April 2026 Gartner Magic Quadrant for Enterprise AI Coding Agents, plus the dozen companies tracked by isaiprofitable.com – and tiles them across the canvas under four layouts: a jittered grid, a hex pack, an arabesque mask built from Lissajous curves under a wallpaper symmetry group, and a Voronoi packer that wraps cleanly at the edges. The arabesque mode is the one that looks most like a real design, which is fitting because the brand identities themselves are arabesques of the same underlying motif.

Nothing in the code is novel. What is striking is how little code it takes to produce something that passes for a branding agency deliverable. Spend a few minutes tweaking curve density and symmetry group and you can cover most of the Fortune 500’s recent AI rebrand portfolio without engaging a designer. None of it requires aesthetic judgement. All of it requires a loop.

That is worth sitting with before moving on to the money.

The Gartner Thermometer and the Sequoia Deficit

The 2024 and 2025 Gartner Hype Cycles placed generative AI at or near the Peak of Inflated Expectations, the top of the curve before the long slide into the Trough of Disillusionment. That topological framing is rough but useful. More arresting is the arithmetic underneath it.

Sequoia Capital published a piece in mid-2024 estimating that AI infrastructure – primarily Nvidia GPU clusters – required roughly $600 billion in annual revenue to justify the capital being committed, while observable AI-attributable revenue across the industry was a fraction of that figure. The gap has a name in finance: “productive deployment lag.” In plain language: the shovels are being sold, the mines are being dug, and the gold has not appeared at the rate the tunnel length implies it should. On isaiprofitable.com, the only company with a green checkmark next to its name is Nvidia, the one selling the shovels.

The 21-tread-pattern t-shirt is a small monument to this gap. Nvidia at the hub, eleven others on the sidewall burning through capital at rates that make the GPU revenue arithmetic interesting, one wallpaper symmetry group holding them all together on a cotton substrate.

The more interesting question is not whether AI is profitable yet. It is why AI is already affecting how we think, even before the P&L sheets catch up.

The Fur Analogy and Cognitive Shedding

Around 1.5 million years ago, Homo erectus started losing body hair. The proximate cause was almost certainly thermoregulation: an upright, running hunter on the African savannah generates core heat faster than a fur coat can dissipate it under midday sun. So the fur went. But heat still needed managing. The solution was clothing, an external thermal regulation artifact that did the job the body had previously handled internally.

This is a pattern that runs through the entire history of human technology. We replace innate biological functions with external artifacts. We then build manufacturing chains around those artifacts. We build economic systems around those chains. The original biological capacity atrophies from disuse or is simply never developed in the first place, because the artifact is already there when the individual arrives.

We do not have hooves. We have shoes. We do not have gills. We have submarines. We do not have the magnetoreceptors that birds use on migration routes. We have GPS. In each case the external artifact is not merely a substitute; it is a platform. It invites elaboration. The shoe becomes the boot, the boot becomes the orthotic, the orthotic becomes a running shoe industry, and the running shoe industry eventually funds the biomechanics research that feeds back into the next generation of the artifact.

I spent six months in late 2022 and early 2023 as an Enterprise Architect contractor at Bridgestone Australia in Adelaide, wiring up the interfaces between SAP, Salesforce, and the mobile mechanic booking platforms that schedule a van to a customer’s driveway to swap a set of tyres. The work was unglamorous integration plumbing – canonical data models, idempotent message contracts, the usual distributed-systems hygiene – but it gave me a tourist’s window into how a tyre and car parts business actually operates. Every node in the system was eventually a physical thing: a passenger radial sitting on a pallet in a Wingfield warehouse, a wheel alignment slot at a Beaurepaires bay, a truck retread coming back from its second life on a B-double. The software was a thin coordination layer over a very heavy industrial substrate.

The history of the company that owns that substrate is the part that stuck with me. Bridgestone – ブリヂストン – started in 1931 not as a tyre company but as a spinoff of a 久留米 (Kurume), 九州 (Kyushu) maker of 足袋 (tabi) split-toe footwear. The founder, 石橋正二郎 (Shojiro Ishibashi), had been adding rubber soles to cotton tabi since 1923 under the Asahi brand, and rubber soles turned out to be the gateway material to rubber tyres. The name Bridgestone is a literal English inversion of his surname: 石 (ishi, stone) and 橋 (hashi, bridge), reversed and translated, picked partly because it sounded export-ready and partly to echo Firestone, the American tyre company he admired. (Bridgestone would eventually acquire Firestone in 1988 for $2.6 billion, which is the kind of historical loop that only happens in long-lived industrial companies.) The katakana spelling ブリヂストン preserves an archaic ヂ where modern Japanese would write ジ, a piece of pre-war orthography the brand keeps the way Mitsubishi keeps its three diamonds.

What I took away from those six months is that Bridgestone is not really a tyre conglomerate that used to make shoes. It is a rubber compounding and supply-chain organisation that noticed rubber was useful for several different external human exoskeletons – first feet, then cars, then conveyor belts, hoses, seismic isolators under buildings, and now sustainably-sourced guayule and dandelion-derived natural rubber to hedge against Hevea brasiliensis geopolitics. The material is the through-line. The product is whichever exoskeleton the century happens to need.

I always jokingly refer to BridgeStone as - “Big Rubber”.

The Car Was Always an Exoskeleton

The automotive vocabulary has colonised AI discourse so completely it has become invisible. We “steer” language models. We apply “guardrails.” We talk about “autopilot” and “co-pilot” and “lanes” and “on-ramps.” There is a vehicle for sale right now whose entire value proposition is that the steering wheel becomes optional once the AI is mature enough. Almost nobody stops to ask why this particular metaphor landed so naturally.

Here is one answer: cars and AI are both exoskeletons.

A car is not a vehicle in the narrow sense. It is a kinetic exoskeleton that extends the range, speed, load-bearing capacity, and weather tolerance of a soft-bodied primate who would otherwise top out at 5 km/h on a good day. The car does not move instead of the human; it extends the human’s movement by several orders of magnitude at the cost of an elaborate support infrastructure: roads, fuel networks, insurance markets, traffic law, emissions regulation, and, yes, tyre companies.

AI is doing the same thing to cognition. The language model does not think instead of the human; it extends the human’s language-generation and pattern-retrieval capacity by several orders of magnitude at the cost of an equally elaborate support infrastructure: GPU clusters, training pipelines, safety teams, regulatory frameworks, and eventually a Gartner hype cycle or two. The car vocabulary arrived so naturally because the thing being described was already familiar. We built the kinetic exoskeleton first. The cognitive one came later, but the shape of the dependency relationship – human plus artifact plus infrastructure – is identical.

What Gets Shed

The cognitive functions most visibly affected are the ones that involve generation under uncertainty: drafting a first sentence, naming a variable, composing an email, sketching a logo. These are precisely the tasks that feel slow and effortful without assistance, and precisely the tasks that AI handles smoothly. The friction is real. The assistance is real. The atrophy risk is also real.

Musicians who practise scales develop neural pathways that do not form if the scales are skipped. Writers who struggle through first drafts develop a tolerance for ambiguity that does not develop if a model always resolves it for them. The logos from the code above look entirely plausible. They look like every other AI logo in the current design vocabulary. They were generated without any cognitive friction whatsoever, which is both the point and the problem. The friction was where the design decision lived.

This does not mean AI assistance is bad, any more than wearing shoes is bad. It means the physiotherapy question matters: are you also doing the cognitive equivalent of the exercises that keep the underlying capacity from going the way of the body hair or your quads since you have spent 2 days chasing the zebra you speared and are waiting to run out of steam and die.

The Shoe Company Principle

Ishibashi’s insight in 1931 was not that feet needed covering; the Asahi tabi side of the business already had that locked up. His insight was that rubber was a platform material for exoskeletal extension, and that the same compounding science, vulcanisation know-how, and distribution muscle that made a good rubber-soled tabi could make a good passenger tyre. He followed the material from one form of human locomotion extension to the next, and a hundred years later that bet is still paying out across tyres, conveyor belts, and seismic isolators.

The AI companies following the GPU stack from language generation to reasoning to planning to embodied robotics are doing exactly this. The H100 is the rubber: a platform material whose applications are not exhausted by the first product built on it. The chat interfaces, the co-pilots, the logo generators – these are the 足袋. What comes next on that manufacturing chain – the 自動車 (jidousha, automobile) tyre equivalent for cognition – is the part worth watching, and worth surviving the trough of disillusionment for.

Whether the current crop of AI businesses is profitable by the time the next Gartner report drops is the wrong question. Whether the infrastructure being built can outlast the hype cycle long enough to produce the genuine platform shift is the question Ishibashi would have asked, probably while measuring rubber tensile strength in a factory that used to make footwear. Japanese has a phrase for that patient industrial cultivation: 改善 (kaizen, continuous improvement). It is not the same word as ハイプ (haipu, hype), and the two have rarely been observed in the same building.

The last shoe company that asked the wrong question about its own product is still making pneumatic rubber cylinders for the machines that replaced the horses we no longer needed to breed. Every AI logo on the t-shirt is a tread pattern drawn by people who do not yet know which century of exoskeleton they are designing for. That beats a PowerPoint ecosystem built around the profitability question, and it is the bet I would rather be inside of when the trough arrives.

The code is at ai_logos_shirt – a small Rust crate, MIT licensed, build instructions in the README. Ping me if you have strong opinions about cognitive atrophy, the Sequoia revenue gap, or split-toe rubber footwear.

From Digital Atlas to Gazet: Building a Geospatial Agent Stack for Enterprise AI

2026-05-23T16:30:00.000-07:00

The same thought keeps coming back whenever I look at geospatial AI: too much useful work is still trapped in slow, manual interfaces.

Someone opens a portal, clicks through search forms, downloads files, copies a bounding box, jumps to a geocoder, then checks an internal approvals tool. None of this is mysterious; it is just fragmented.

That is why MCP matters here. Not because every problem needs another chatbot, but because a composed agent stack can turn UI choreography into a question, a traceable tool chain, and a defensible answer.

The shape of that stack is getting clearer:

A discovery layer for authoritative datasets and services.
A place-resolution layer for turning ambiguous human geography into valid geometry.
A retrieval and processing layer for actual data operations.
A governance layer so this can survive contact with enterprise reality.
An orchestration layer that routes work to the right model and the right tool at the right time.

Themes carried from the earlier LinkedIn post

The LinkedIn thread that led to this draft had four recurring ideas, and they still hold:

Most geospatial AI pain is not in model quality. It is in UI choreography.
Narrow components with explicit contracts beat one giant, vague “do-everything” agent.
Portals are temporary. Durable value sits in services, workflows, and evidence trails.
The end state is not one-shot answers. It is persistent monitoring plus governed recomputation.

This post extends those ideas with concrete stack boundaries, standards, and orchestration patterns.

Digital Atlas of Australia as the discovery layer

In daa_mcp, I wrapped the Digital Atlas of Australia search API as a Go MCP server. The target is the Digital Atlas endpoint at https://digital.atlas.gov.au/api/search/v1, with a very narrow tool surface:

atlas_list_collections
atlas_search_items
atlas_answer_query

That narrowness is deliberate. Discovery tools should be boring, explicit, and provenance-heavy. If an agent cannot reliably name the dataset, map, or document it found, and where it came from, it should not improvise higher-order spatial reasoning.

This is the first important pattern for enterprise AI engineering: do not ask the model to remember the catalog. Ask the catalog.

In practice, a user should be able to ask “what hazard datasets are relevant around this place?” and get source URLs, approximate bounds, and the inventory used to produce the answer. That alone is a major improvement over portal-clicking.

DevSeed’s Gazet is the missing place-resolution layer

The more interesting development lately is not another catalog wrapper. It is the rise of narrower geospatial AI components that do one thing well.

Development Seed’s Gazet is a very good example. It is a lightweight natural-language geocoder and GIS operations system over Overture and Natural Earth parquet datasets. The point is not to sound impressive. The point is to turn a phrase like “the northern half of India” or “districts along the Ganges” into valid geometry and spatial SQL.

That matters because geospatial systems usually break before the raster math starts. They break at the human-language boundary: fuzzy place names, overlapping regions, ambiguous jurisdictions, rivers that matter politically but not administratively, borders people describe one way and datasets encode another.

DevSeed’s public write-up on Gazet is refreshing because it avoids frontier-model mysticism. They fine-tuned a Qwen 3.5 0.8B model, generated around 70k validated training pairs, and built a system that runs on CPU, packages into a single container, and emits valid geometry. The model card shows an 812 MB Q8 GGUF artifact. That is software engineering.

The surrounding design is exactly the structure geospatial AI needs:

fuzzy candidate search against Overture divisions and Natural Earth
DuckDB-backed in-memory spatial SQL
GeoJSON output rather than vague textual summaries
a path toward execution-feedback loops and CodeAct-style tool use

Gazet is not an MCP server today, but it is exactly the kind of component that should sit behind one. A Digital Atlas MCP answers “what data exists?”. A Gazet-style place tool answers “what geometry did the user mean?”. Keeping those jobs separate reduces hallucination at both layers.

Skills and MCPs are complementary, not interchangeable

Development Seed’s public skills repository reinforces the same architectural idea from another angle. Skills package workflow knowledge and output structure; MCP servers package live tool access.

That distinction matters. A skill can tell an agent how to build a VEDA story, export an issue to markdown, or set up a Python repo. An MCP server lets that same agent call live APIs, inspect datasets, or resolve fresh operational context.

Enterprise AI work needs both. One gives you procedure. The other gives you reach.

STAC, GDAL, and the rest of the operational path

Once discovery and place resolution are handled, the rest of the geospatial chain looks familiar again.

STAC remains the right interoperability layer for item and asset selection. GDAL, QGIS-style tooling, OSM connectors, Earthdata accessors, and similar MCPs remain the right place for the sharp-edged operational work: clipping, reprojection, vector joins, raster access, metadata inspection, and export.

This is why I still do not buy the idea of a single universal geospatial MCP. Real systems are layered:

Layer	Job	Example
Discovery	Find authoritative datasets, maps, and documents	`daa_mcp`
Place resolution	Turn language into geometry	Gazet-style geocoder / gazetteer
Catalog handoff	Resolve items and assets	STAC MCP
Workflow orchestration	Schedule and monitor long-running jobs	Argo Workflows via MCP
Spatial compute	Transform and analyze data	GDAL / QGIS / OSM style MCPs
Presentation	Package outputs for humans	VEDA-style skills, notebooks, apps

The protocol is the plumbing. The quality comes from giving each part of the stack a narrow responsibility.

This is the same systems lesson in geospatial form: trying to make one component do discovery, place semantics, heavy compute, and governance usually creates brittle architecture. Small explicit layers look less magical, but they survive production.

EO processing standards are now part of the same stack

This is where real operations begin. Once a question needs heavy lifting, you need a standards-aligned processing path, not just another API call.

I think of it as three aligned pieces:

eoAPI as the service substrate (STAC, tiles, PostGIS-adjacent query surfaces)
openEO UDFs for custom analysis logic that does not fit built-in processes
OGC API - Processes as a portable execution contract for geospatial jobs

This matters because of execution portability, not standards purity. If a workflow is represented through openEO process graphs, UDF hooks, and OGC process semantics, you are no longer trapped in one vendor UI or one internal notebook convention.

The openEO side is especially useful because it is explicit about chunked execution and long-running back-end jobs. run_udf and related process patterns (apply_dimension, apply_neighborhood, reduce_dimension) are built for data-cube scale operations where synchronous request/response is the wrong abstraction.

OGC API - Processes complements this by giving a common HTTP interface for process descriptions, inputs, execution, and result retrieval. That is the bridge between geospatial compute and enterprise service design: you can expose processing as a governed capability, not just a script somebody knows how to run.

eoAPI then provides a practical packaging layer around STAC-first infrastructure (pgSTAC, stac-fastapi, titiler-pgstac, tipg). Data discovery and data services stay first-class while compute orchestration grows around them.

Long-running analysis needs workflow orchestration, not prompt retries

When EO analysis becomes asynchronous (tiling, temporal composites, model inference over large extents, zonal stats at scale), agent architecture must move from “call tool, get answer” to “submit job, monitor, collect artifacts, summarize result”.

This is exactly where Argo Workflows patterns fit, and why Argo-flavored MCP servers are interesting. The MCP Market Argo server by jakkaj exposes workflow submission/status/result retrieval through JSON-RPC. That maps naturally to agent behavior:

build a workflow payload from resolved geometry + selected assets
submit to workflow engine
poll status with bounded retries and timeout policy
fetch outputs and register artifact locations
return a grounded summary with links to outputs and provenance

Critically, outputs here are not only text. They can be Cloud Optimized GeoTIFFs, STAC Items, GeoParquet extracts, vector tiles, QA reports, or model feature embeddings. For geospatial agents, the right response class is generated artifacts plus explainable metadata, not just a paragraph.

If you combine this with gateway policy (auth, quotas, audit) and model routing, you end up with a more durable operating model:

language models interpret intent and coordinate tools
workflow systems execute expensive, long-running spatial jobs
standards-based APIs keep outputs portable across teams and platforms

That is much closer to “EO production system” than “chatbot with map screenshots”.

Why enterprises should care: this is really about UI replacement

The bigger opportunity is not just geospatial search. It is enterprise task conversion.

Most corporate and government knowledge work still lives in forms, portals, dashboards, policy systems, and browser tabs. Much of it is manual API orchestration done by humans through UIs.

We should be blunt about portal economics. Most are disposable visual sales tools, one level up from a PowerPoint deck. They are useful while an initiative is being socialized or funded, but once that phase passes, maintenance becomes hard to justify.

The exception is when the interface itself is the monetized product. If the UI is a booking or experience engine, like Airbnb, Uber, or Netflix, the interface is the business and ongoing investment is obvious. If it is only a discovery surface, the long-term value is usually in the underlying services and automations, not the glossy front end.

That is where agent stacks become interesting: not “replace staff with a bot”, but “replace repetitive navigation with question answering backed by orchestrated tools”.

For example:

use a Digital Atlas MCP to find relevant public datasets
use a Gazet-style tool to resolve the place phrase into geometry
use STAC or downstream data APIs to locate assets
use GDAL or analytics tools to extract what matters
use an internal engineering or policy system to fetch approvals, tickets, or previous decisions
return one answer with links, bounds, intermediate evidence, and the exact tools invoked

That is a much more defensible enterprise AI story than a generic assistant sitting on top of a screen scraper.

The enterprise layer above the tools: gateway patterns

Of course, the moment you try to productionize this, you run into the boring parts that actually matter: auth, quotas, observability, data leakage, audit, policy, and cost.

At that point, the conversation should move from product selection to gateway patterns. Martin Fowler’s gateway pattern framing is useful because it centers controlled mediation between clients and backend capabilities, which maps directly onto agent-to-tool and agent-to-service boundaries.

The first pattern is identity mediation. Agent calls need to carry a clear identity boundary between user, agent, and downstream systems, with policy checks at each boundary.

The second pattern is protocol normalization. Teams will mix chat payloads, tool calls, MCP traffic, and legacy APIs. The gateway has to normalize those flows so routing and controls are consistent.

The third pattern is cost and budget governance. Token and request spend cannot be an afterthought. Per-key, per-team, and per-tier budgets need to be enforceable in the traffic path, not in a monthly spreadsheet.

The fourth pattern is safety policy composition. Guardrails are rarely one thing. In practice you need layered controls: authentication, content moderation, prompt-injection checks, data-loss prevention, and explicit deny behavior.

The fifth pattern is telemetry with replayable evidence. A useful gateway record is not just “request succeeded”. It is who called what, with which policy version, routed to which provider or tool, at what cost, with what outcome.

I still mention Kong mainly as a signal of category maturity, not as a single implementation path. The same pattern language also shows up in Otoroshi’s AI gateway model: provider abstraction, OpenAI-compatible route surfaces, API-key enforcement, metadata-driven budgets, moderation and regex guardrails, plus extension-based control points.

Whether a team uses Kong, Otoroshi, or another gateway stack is less important than whether these patterns are explicit. Enterprises want MCPs and agent flows, but they need them behind durable control planes instead of scattered sidecars with weak visibility.

Once agents start calling internal systems, this layer becomes unavoidable. Someone will want to know who called what, with which identity, against which policy, and how many tokens or requests that answer burned to produce.

Small models are not a concession, they are often the right architecture

The other part of this conversation that is becoming more honest is model size.

Development Seed’s Gazet makes the point very clearly: many geospatial tasks are structured enough that a small model, grounded to the right schema and surrounded with the right tooling, is better engineering than throwing a giant model at the problem.

That also lines up with what NVIDIA has been highlighting around agentic workloads. Their recent MiniMax M2.7 material is not about a universal oracle; it is about efficient agentic harnesses, tool calling, and orchestration-friendly serving with explicit tool-call parsers, auto-tool-choice, and a runtime story through vLLM, SGLang, NIM, and NemoClaw.

MiniMax M2.7 is not “small” in total parameter count, but it is architected in the direction the market is moving: sparse activation, efficient serving, orchestration hooks, and infrastructure that treats tools as first-class. The important lesson is not the specific model; it is the pattern.

The pattern looks like this:

small or moderate specialist models where the task is structured
larger reasoning models only where the ambiguity genuinely demands them
explicit tool routing rather than one giant model pretending to be every subsystem

For geospatial and enterprise workflows, that is just common sense.

The real end state

The end state here is not “chat with your maps”.

It is something more useful:

take a question expressed in normal language
resolve the place semantics properly
discover the authoritative data sources
retrieve or compute what is needed, including expensive long-running analytics when required
enforce enterprise controls on the path
return an answer that is fast, inspectable, and grounded in tools

Beyond one-shot questions, run persistent agents that watch for change over time. For hazard operations, that means continuously tracking weather, flood signals, fire risk, and related event streams, then escalating when deviation from steady state crosses defined thresholds.

The same pattern applies to slower domains such as seasonal wheat growth. A persistent agent should infer baseline behavior from historical and current EO catalogs, run heavy analysis when confidence drops, and react when observed trajectories diverge from expected seasonal dynamics.

This is where long-running workflows and persistent agents work together: the workflow engine handles expensive catalog-scale computation, while the agent handles temporal monitoring, trigger logic, and action routing.

That operational loop was the key point in the earlier LinkedIn post, and it is worth repeating: in enterprise geospatial systems, value compounds when agents watch state over time and selectively escalate computation, not when they generate one more static summary.

That is what turns manual UI work into operational question answering.

For geospatial work, it means less tab-hopping and more reproducible spatial reasoning. For enterprise AI more broadly, it means agents stop being a decorative chat layer and start looking more like a disciplined API and workflow orchestration fabric.

Better than another dashboard, frankly.

If you are building in this direction, the interesting design questions are no longer “which model is smartest?” They are:

where should discovery stop and reasoning begin?
which place-resolution tasks deserve their own narrow model?
which tools should be exposed directly as MCPs versus hidden behind workflow skills?
what sits at the gateway for auth, quotas, audit, and A2A telemetry?
which monitoring loops should run continuously, and what deviations should trigger expensive recomputation or human escalation?

That is engineering, not theatre.

References

DAA MCP repo: https://github.com/whatnick/daa_mcp
Development Seed Gazet blog: https://developmentseed.org/blog/2026-04-29-gazet/
Development Seed Gazet repo: https://github.com/developmentseed/gazet
Development Seed Gazet model: https://huggingface.co/developmentseed/gazet-model
Development Seed skills repo: https://github.com/developmentseed/skills
Development Seed skills directory: https://github.com/developmentseed/skills/tree/main/skills
Kong AI Gateway: https://konghq.com/products/kong-ai-gateway
Otoroshi AI gateway tutorial: https://www.otoroshi.io/docs/tutorials/setup-ai-gateway-with-otoroshictl?_highlight=ai
Otoroshi AI gateway topic: https://www.otoroshi.io/docs/topics/ai-gateway
Martin Fowler, gateway pattern: https://martinfowler.com/articles/gateway-pattern.html
Geospatial MCP search: https://mcpmarket.com/search?q=Geospatial
eoAPI: https://eoapi.dev
OGC API - Processes overview: https://ogcapi.ogc.org/processes/
openEO UDFs: https://openeo.org/documentation/1.0/udfs.html
openEO process definitions (run_udf, run_ogcapi, etc.): https://processes.openeo.org/
Argo Workflows docs: https://argo-workflows.readthedocs.io/en/latest/
Argo MCP listing (jakkaj/mcp-argo-server): https://mcpmarket.com/server/argo
NVIDIA MiniMax M2.7 agentic workflows: https://developer.nvidia.com/blog/minimax-m2-7-advances-scalable-agentic-workflows-on-nvidia-platforms-for-complex-ai-applications/
STAC MCP listing: https://mcpmarket.com/server/stac

AI-assisted PCB design for energy monitoring (and scaling up don't trust the autorouter)

2026-05-13T17:00:00.000-07:00

There is a specific smell to late-stage PCB work. A mix of coffee, solder mask anxiety, and the faint optimism that one more route pass will magically make analog behave. The poking around with oscilloscope probes and poring at datasheets to find out that you failed to pull reset low, out comes the magnet wires for a bodge.

Over the last week I have been building out an ADE9000 breakout with an AI-assisted workflow wrapped around KiCAD scripting. The results are useful, occasionally impressive, and still very far from “hands-off” for precision energy monitoring.

If you grew up with “don’t trust the autorouter” as muscle memory, good news: the saying still holds. We just scaled the blast radius from one menu click to an entire AI pipeline.

What changed on the board, not just on a slide deck

The recent commit sequence in ADE9000_Breakout tells the story better than any marketing copy:

d1e3352 (2026-05-07): “Initial AI Creation”
043161c (2026-05-08): “Complete routing”
b8c8132 (2026-05-09): “Reroute with JST connector for SPI”
a469246 (2026-05-09): “Move decoupling closer”
84b86e7 (2026-05-12): “Relayout using connectors for analog inputs”
7d4b420 (2026-05-16): “Rerouted with Groundplanes and new skills”
c40e40a (2026-05-16): “Added License”
9b2d97e (2026-05-17): “Add project-local 3D CAD assets”
cf8ba2c (2026-05-17): “Fix CT jack STEP orientation”
f5a11b2 (2026-05-17): “Remove CT jack pad overlays”

This was not a single-shot generation. It was iterative engineering with scripting and machine assistance in the loop:

Placement and connector mapping in scripts like place_pcb.py.
Deterministic route passes in route_pcb.py.
Critical route seeding and patching in seed_critical_routes.py and patch_remaining_routes.py.
Mechanical/silk cleanup with apply_board_markings.py and move_refs_to_silkscreen.py.
Continuous ERC/DRC artifacts (erc.json, drc.json) committed as hard checkpoints.
3D STEP model handling pushed into the reusable layout and size-shape skills so mechanical review can keep pace with PCB edits.

That pattern matters. The AI and automation stack gave speed and repeatability, but the engineering value came from repeated correction passes driven by board physics.

The same iteration loop is now visible in whatnick-energy-monitor-skills:

335a718 (2026-05-16): Initial whatnick energy monitor skills
78bfc44 (2026-05-17): Document STEP model workflow
79d925d (2026-05-17): Clarify connector STEP model matching
af57c2d (2026-05-17): Document connector CAD alignment workflow
32adeff (2026-05-17): Correct ADE9000 jack CAD guidance
d4167f6 (2026-05-17): Remove ADE9000 jack overlay guidance

That repo now captures reusable circuit, layout, routing, and board-shape guidance, plus the ADE9000 project overlay. The 3D model side was not a first-pass success either. STEP exports and connector models took multiple iterations before the project-local paths, model matching, and export workflow were stable enough to trust in a mechanical review. The later commits are the best evidence: first add local CAD assets, then discover the CT jack STEP orientation is wrong, then remove pad overlays that made the model look plausible while hiding the fact that the footprint and mechanical story needed to line up properly.

This is where AI-assisted hardware feels different from AI-assisted code. In software, a bad abstraction usually fails in tests or production logs. In PCB work, a bad abstraction can become a connector body floating neatly above the wrong pads in the 3D viewer. It looks professional right up until the part, enclosure, or cable says otherwise.

NotebookLM research: good at methodology, weaker on analog edge cases

I started on this adventure with posts from Samuel Beek promoting AI PCB design. I blended in my research background and curated a set of academic papers on AI/Algorithm usage in PCB place-n-route. I queried my pre-created NotebookLM notebook “PCB Place and Route Algorithms” specifically for mixed-signal energy-monitoring constraints.

The useful part of that synthesis:

Current AI placement/routing research still optimizes mostly for geometric proxies (wirelength, overlap, congestion).
Newer approaches improve constraint capture and collaboration, but are explicitly human-in-the-loop.
Model quality degrades on low-frequency pattern classes and novel interface combinations (cold-start behavior).

The important caveat from the same query was even more telling: source coverage was strong on AI placement methodology, but thinner on precision metering specifics like safety creepage strategy, return-current choreography around split references, and “what actually ruins ENOB on a real board at 2am”.

That gap is exactly the point.

Schematik, SnapMagic, and the new hardware workflow stack

Samuel Beek’s Schematik story is almost too perfect as an origin myth for this moment. An AI-generated wiring guide for a home-grown door opener took out the fuses in his apartment, which is a fairly direct way for physics to reject your prompt. Schematik has since raised $4.6 million from Lightspeed Venture Partners and is positioning itself as a “Cursor for Hardware”: describe a device in plain language, get a bill of materials, purchase links, and assembly instructions.

The interesting engineering choice is the safety boundary. Schematik is deliberately aiming at low-voltage circuits, typically 3-5 V IoT and maker projects, because that is where the promise is large and the downside can still be bounded. That is a sensible line. It is also a reminder that “AI for hardware” is not one market. The assistant that helps someone build an MP3 player is not the same system I would trust near mains metering, isolated sensing, or a DIN rail enclosure without a lot more constraint machinery around it.

SnapMagic is attacking a nearby but different layer of the stack. Its pitch is an AI copilot for electronics design, built on the huge CAD model base that started as SnapEDA: symbols, footprints, 3D models, part discovery, BOM optimization, supply-chain substitution, and integration with existing EDA tools including KiCad. That matters because a surprising amount of PCB time is not heroic analog insight; it is finding the right model, importing it cleanly, checking whether the footprint is sane, and making sure the thing still exists at Mouser or Digi-Key.

Schematik is closer to natural-language project generation. SnapMagic is closer to CAD-data and component-selection acceleration. My little ADE9000 workflow sits in the garage between them: NotebookLM for research synthesis, KiCad Python for deterministic changes, Freerouting for a first pass, project-local skills for repeatable domain knowledge, and human review for the parts that still smell like physics.

That is the practical opening for individual designers. You do not need to wait for one perfect vendor platform. You can build a small, opinionated workflow from pieces you already control:

Put your design rules and recurring board-family choices into a local skill or checklist.
Use AI for datasheet digestion, net naming, script generation, and alternative exploration.
Keep KiCad, ERC, DRC, STEP export, and git history as the accountability layer.
Treat external libraries like SnapMagic/SnapEDA as accelerators, then verify footprints, symbols, pin numbers, and 3D models against datasheets and mechanical reality.
Let autorouters and AI propose routes, but hand-check return currents, decoupling loops, creepage, clocks, testability, and enclosure fit.

That sounds less magical than “hardware Cursor”, but it is much closer to how individual designers can safely get leverage today.

Why this hurts more on energy monitoring boards

An ADE9000-style board is not just “digital plus some analog”. It is a negotiated peace treaty between:

tiny differential analog signals,
noisy clocks and digital SPI edges,
shared ground structures,
high-voltage interfacing constraints,
and assembly realities.

AI can route what it can score. Physics punishes what you forgot to score.

For energy monitoring, the failure modes are often subtle first and expensive later:

A seemingly short route with a terrible return path becomes an EMI antenna.
“Close enough” decoupling in XY turns into high loop inductance in 3D.
Digital fanout convenience leaks noise into the measurement front end.
Clearance passes in one view while creepage silently fails along real surfaces.

The scaled-up autorouter adage

Classic autorouter distrust was about ugly traces and cleanup effort.

AI-assisted distrust is about false confidence.

You now get cleaner visuals, plausible routing, and confidence scores. The board can look more “engineered” while still violating analog intent. That is worse than obviously bad output, because it delays the moment when a human gets suspicious.

In the ADE9000_Breakout commits, that showed up as repeated topology and placement adjustments:

decoupling moved closer,
connector strategy revised,
critical nets explicitly seeded,
remaining logical gaps patched after freeroute import,
then relayout for analog input connector realism.

None of that is anti-AI. It is pro-accountability.

A practical review checklist I now treat as mandatory

Before I trust an AI-assisted pass on a precision board, I manually review:

Return current continuity under each critical signal path.
Analog/digital ground interaction at the exact stitch points, not just net names.
Decoupling loop geometry (pin, cap, via topology), not just nearest-component distance.
Clock and fast digital net proximity to high-impedance analog channels.
Creepage and clearance across real isolation boundaries and along surfaces.
Testpoint access and assembly risks (reworkability, tombstoning, awkward probe points).

If any of those rely on “the model probably understood that”, I assume it did not.

What AI is genuinely good for in this workflow

After using it in anger, the wins are real:

Faster exploration of placement variants.
Deterministic scriptable edits to keep design intent reproducible.
Constraint management that catches obvious misses early.
Better ergonomics for repeated board evolutions.
Mechanical and export workflows, including STEP models, can be standardized, but they still need several passes to reconcile footprint libraries, connector variants, and CAD export paths.

This is similar to CNC in machining. You still need a machinist mindset. You just get to fail faster and with better logs.

The part that still needs engineers

The physics does not care whether a trace came from a human, an RL policy, or a nicely branded copilot.

Energy metering boards live or die on the details that are hardest to encode as generic reward functions. The edge where analog integrity, EMC, and safety overlap is still mostly tacit knowledge earned through measurements, bring-up scars, and post-mortems.

So yes, use AI aggressively for PCB work. I certainly am.

Just keep the old sign above the bench.

Don’t trust the autorouter.

Now it applies to systems, not just traces.

If you are building similar mixed-signal boards, I would love to compare review checklists and failure cases that escaped DRC but showed up on the bench.

Using Aluminium Instead of Carbon to Make Silicon

2026-05-01T17:00:00.000-07:00

I have been thinking a lot about sovereign capability lately, not in the abstract flag-waving sense, but in the boring physical sense of what materials sit near each other, what energy sources are nearby, what ports exist, and what loops can actually close.

That line of thought clicked into a more concrete shape after reading the paper Carbon-Neutral Silicon via Aluminothermic Reduction? Exploring Industrial Symbiosis through Life Cycle Assessment, the Australian Silicon Action Plan, and then updating my Aluminium + Silicon Sovereign ecosystem slide deck to reflect it.

The core idea is simple enough to explain to a high-school chemistry class: we normally reduce quartz to silicon with carbon. What if, in the right industrial setting, we used aluminium to reduce silicon instead of carbon?

The Conventional Route Uses Carbon

Silicon does not come out of the ground in neat shiny wafers. It starts as silica or quartz, and the conventional metallurgical route is carbothermic reduction: take quartz, add a carbon source, add a lot of heat, and accept a pile of carbon dioxide as part of the bargain.

That bargain made sense when the objective function was mostly “make silicon cheaply”. It makes less sense when we also care about carbon intensity, geopolitical fragility, and whether a country with abundant ore, sunshine, and smelting know-how can turn those endowments into a durable manufacturing base.

The Alternative Route Uses Aluminium

The paper explores aluminothermic reduction, using an aluminium source as the reductant material instead of carbon. More specifically, it looks at aluminium dross as an industrial symbiosis input rather than a pristine, purpose-made feedstock. That detail matters. This is not a fantasy process that assumes some magical zero-cost aluminium stream falls from the sky. It starts from a messy industrial byproduct and asks whether a better loop can be built around it.

The headline result is strong enough to justify attention: the authors find that the aluminothermic route can reduce global warming impact and cumulative energy demand by up to 80% relative to the reference route.

That is the part that makes you sit up.

The useful thing about the paper is that it does not stop at the good news. Some impacts get worse, especially if the aluminium scrap would otherwise have displaced something valuable elsewhere, and because this route still needs extra input materials. So this is not free decarbonisation. It is a real industrial trade-off.

That makes the paper more useful, not less. Serious policy should be built on “this looks promising, but here are the hotspots” rather than on conference-hall hydrogen hallucinations.

Why This Starts To Look Real

In updating that slide deck, I kept coming back to the same question: what happens if you stop treating aluminium, silicon, and solar panels as separate industries?

That is the framing I find compelling, because it turns this from a chemistry curiosity into an engineering and logistics problem.

Around 80% of a typical solar panel by mass is aluminium plus silicon. If a country is serious about energy sovereignty, it should be thinking not just about installing more panels, but about building the material loops that sit behind them. Solar farms are not merely generators. They are future material stockpiles sitting in the sun.

Once you see that, a different policy picture appears:

quartz becomes not just a mining input but a strategic silicon feedstock.
bauxite and aluminium refining become adjacent to solar manufacturing rather than unrelated heavy industry.
end-of-life panels become future reductant, frame stock, and silicon feed instead of landfill problems.
smelters, ports, and renewable generation start to look like parts of the same machine.

This is where aluminium reducing silicon instead of carbon stops being an isolated chemistry trick and starts looking like something you could build an industry around.

If I Had To Put Pins On A Map

The notebook I pulled together on domestic solar manufacturing helped sharpen this. Once you stop talking in continent-sized blobs and start naming actual places, a few candidates jump out.

1. Kemerton and south-west WA

This is the least speculative option because Simcoa at Kemerton already exists and is still Australia’s only operating silicon manufacturer. The Silicon Action Plan notes Simcoa is producing about 52,000 tonnes of metallurgical silicon a year, mining its own quartz and running an established smelter operation.

That matters because south-west WA also has the Darling Range bauxite mines, alumina refineries at Wagerup, Pinjarra and Worsley, the SWIS grid, and Bunbury port infrastructure all in the same broad industrial neighborhood. If you wanted to trial aluminium-assisted silicon reduction somewhere in Australia, starting near the one place that already knows how to make silicon seems less heroic than starting from a blank paddock.

2. Townsville and the Lansdown precinct

If Kemerton is the incumbent, Townsville is the “someone is actually trying to draw the whole supply chain in one industrial estate” option. The major project write-up and Solar Sunshot coverage point to the Lansdown Eco-Industrial Precinct near Townsville as the proposed site for a quartz-to-metallurgical-silicon campus plus silicon ingot and wafer manufacturing.

What I like about Townsville is not that it is magically complete today. It is that the logic is visible. There is Queensland quartz, there are nearby solar resources, there is port access, and CopperString plus the Northern Queensland REZ story gives you a plausible path to much more electricity than the region has today. It is easier to imagine an aluminothermic pilot piggybacking on a place already trying to integrate quartz, silicon and wafer production than on a site that only knows one piece of the story.

3. Gladstone

Gladstone feels like the heavy-industry answer. It already has alumina, aluminium-adjacent infrastructure, deep-water port capability, and a lot of people thinking about how to decarbonise industrial heat without hollowing out the place. The Climateworks work on Gladstone is interesting here because it frames the region not just as a load, but as a flexible industrial node that could soak up and shape renewable power.

Gladstone is weaker than Kemerton on current silicon capability, but stronger on industrial mass. If you needed somewhere that already thinks in terms of furnaces, refineries, export terminals and gigawatts rather than artisan clean-tech vibes, Gladstone is on the shortlist.

4. Mourilyan and Weipa as upstream feedstock pieces

I would not put the whole chain in one place just to satisfy a PowerPoint aesthetic. Sometimes the better answer is a linked corridor rather than one mega-site.

The Mourilyan silica sands project is interesting because it gives Far North Queensland a high-purity silica input close to road and port infrastructure. Pair that with Cape York bauxite and alumina flows coming through Weipa and Yarwun and you start to see a north-to-central Queensland materials story, even if the final smelting and wafering steps land further south.

That sort of arrangement is less neat on a map, but a lot more believable in real life.

I have spent enough time around electronics, energy monitoring, and hardware supply chains to be skeptical of national capability claims built on nothing more than a minister at a lectern. Sovereign capability usually comes from embracing the mess: furnaces, scrap streams, slag reprocessing, aging solar farms, logistics yards, and the boring people who know how to keep them running through summer.

The paper explicitly highlights recirculating carbonation gases, reprocessing byproduct slags, and using surplus aluminium scrap as some of the most important improvement levers. Those are exactly the kinds of details that separate a sovereign ecosystem from a PowerPoint ecosystem.

The Fallen Leaves Analogy Is Better Than the Circular Economy Cliche

One line from the slides stuck with me: every 10 years or so, as panel efficiency degrades or silicon technology advances, you recycle the aluminium and silicon into a new panel. Build enough installed capacity and after 30-35 years you have not just electricity generation, but a meaningful stockpile of reusable material.

That feels less like a recycling slogan and more like a forest floor. Fallen leaves are not waste. They are deferred structure. The same could be true of first-generation solar deployments if we design the industrial loop ahead of time rather than pretending recycling will somehow organize itself later.

This is also where the sovereign-policy lens improves the climate-policy lens. A circular loop that produces domestic industrial feedstock, manufacturing resilience, export optionality, and lower carbon intensity is politically sturdier than one justified only as moral sacrifice.

What I Would Actually Like To See Next

If this idea is to move from interesting paper to something testable, I would want to see a few things next:

a serious Australian material flow analysis for quartz, aluminium scrap, aluminium dross, solar panel retirements, and metallurgical silicon demand.
a location-based study around Kemerton, Townsville-Lansdown and Gladstone rather than a placeless national average.
explicit comparison against the alternative use of the aluminium scrap streams, because the paper shows this assumption drives a lot of the environmental trade-off.
a pilot framed as industrial symbiosis infrastructure, not just as a decarbonisation demonstration.

The real question is not “can we make a greener tonne of silicon?” It is “can we build a self-reinforcing aluminium-silicon-energy system that compounds capability over decades?”

Final Thought

I like this idea because it is neither purely green-tech optimism nor old-school extractive nostalgia. It says something more interesting: a country with abundant sun, bauxite, quartz, and engineering talent should be able to turn one generation of solar build-out into the feedstock for the next.

Using aluminium to reduce silicon instead of carbon will not solve everything. The paper is clear about the trade-offs, and that honesty is part of why it is worth reading. But as a way of connecting chemistry, recycling, heavy industry, solar deployment, and geography into one coherent story, it has teeth.

That is usually a sign the idea deserves a prototype.

From SaaS to Serviced Software -- the code was never the hard part

2026-04-03T15:00:00.000-07:00

Sometime in the last two years the bottleneck shifted. I used to spend most of a greenfield week staring at an empty editor, scaffolding routes, arguing with ORMs, and coaxing CSS into something that did not look like a government form from 2004. Today I can prompt my way to a working CRUD app with auth, a reasonable schema, and even half-decent styling before lunch. The code, it turns out, was never the hard part. The hard part was–and still is–keeping the thing alive once real users touch it.

The SaaS mental model

“Software as a Service” trained an entire generation to think the value lives in the application layer. Build a clever feature, wrap it in a subscription, ship a landing page. The implicit promise: we write the software, you pay monthly, everyone wins. That framing put the spotlight squarely on creation–new features, new integrations, new UI polish.

But anyone who has operated a SaaS product past the euphoric launch week knows where the hours actually go:

Rotating secrets and patching CVEs at 11pm on a Friday
Chasing down why the invoice PDF lambda timed out in ap-southeast-2 but not us-east-1
Fighting Terraform drift after someone clicked through the console “just this once”
Explaining to a customer why their data export is 48 hours stale because the Celery worker OOM-killed itself

The ratio of build-time to keep-it-running-time was already lopsided. AI just made it more obvious.

Enter the vibe-coded prototype

Large language models have compressed the “zero to working prototype” phase from weeks to hours. Cursor, Copilot, Aider, v0, Bolt – pick your weapon. The scaffolding phase that used to justify a two-pizza team for a quarter now fits in a solo weekend sprint. I have experienced this first-hand: prompting out a FastAPI backend with DynamoDB tables, an SPA frontend, and a deployment pipeline that mostly works. The code is not elegant. It does not need to be. It is structurally correct enough to demo and iterate.

This is genuinely magical. It is also genuinely dangerous, because it creates an illusion of completeness. The prototype works on your laptop, passes the happy path tests the LLM also generated, and looks great in the demo. Then production happens.

Production is where software goes to get serviced

Here is where the mental model needs updating. We are not really selling Software as a Service anymore. We are selling Serviced Software – and the distinction matters.

In the SaaS framing, the software is the product and the service is the delivery mechanism. In the Serviced Software framing, the service is the product and the software is just the substrate it runs on. Customers do not care that your backend is FastAPI or Express or Rails. They care that:

It is up when they need it (hosting, redundancy, failover)
Their data is safe (encryption at rest and in transit, access controls, backups that actually restore)
It stays current (dependency updates, OS patches, framework migrations)
It costs a predictable amount (no surprise egress bills, no runaway autoscaling)
Someone answers the phone when it breaks at 3am

None of those are code problems. They are operational problems. And they are the problems that AI is worst at solving, because they require sustained human judgement over months and years, not a one-shot generation pass.

The maintenance asymmetry

There is a well-known asymmetry in software engineering: building version 1.0 is perhaps 20% of the total lifetime cost. The remaining 80% is maintenance, evolution, and eventual decommissioning. AI has dramatically reduced the cost of that first 20%. But it has done almost nothing for the other 80%.

If anything, AI makes the maintenance problem worse. When code is cheap to produce, people produce more of it. More repos, more microservices, more side projects that “just need a small server.” Each one becomes a maintenance liability. Each one needs patching, monitoring, log rotation, certificate renewal, database vacuuming. The open-source maintainer burnout problem I wrote about after PyConAU 2019 is now everyone’s problem, because everyone is now a maintainer of their own vibe-coded fleet.

I keep thinking about the analogy to 3D printing. When desktop printers got cheap, everyone printed trinkets for a month. Then the printers gathered dust because the hard part was never fabrication–it was design, finishing, and material science. The bottleneck moved upstream and downstream simultaneously, leaving the newly-cheap middle step feeling oddly irrelevant.

What “Serviced Software” looks like in practice

If you accept that the value has shifted from code creation to code stewardship, a few things follow:

Platform engineering matters more than feature engineering. Internal developer platforms (Backstage, Port, Humanitec) that abstract away infrastructure and enforce guardrails are more valuable than another AI code assistant. The companies investing in paved roads for deployment, observability, and incident response will win over those investing in faster code generation.

Managed services eat custom code. Every line of custom infrastructure code is a future maintenance burden. I learned this the hard way running microservices from folders on EC2 – cron jobs, plain-text .env files, manual venv management. It worked, but every operational incident was my problem. The appeal of managed Postgres over self-hosted, or Vercel over hand-rolled CI/CD, is not laziness. It is recognizing where your scarce operational attention should go.

Security becomes the primary differentiator. When everyone can generate a working app, the ones that survive are the ones that do not get breached. Supply chain attacks, dependency confusion, credential leaks in AI-generated code that helpfully hardcoded an API key – these are the failure modes of the vibe-coding era. Security is not a feature you bolt on; it is the service layer that justifies the subscription.

Cost modeling is a core engineering skill. I wrote about EKS baseline costs being $70/month for personal projects back in 2020. That sensitivity to operational cost used to be a niche concern. Now that anyone can spin up infrastructure with a prompt, understanding what it costs to keep running is table stakes. Cloud bills are the new technical debt – invisible until they are catastrophic.

Tactical tornados vs strategic maintainers

Every engineering org has its archetypes. The tactical tornado is the superstar feature developer who can bang out a new module in a weekend, leave a trail of impressed stakeholders, and move on to the next shiny thing. They are celebrated in sprint reviews, promoted quickly, and held up as the template for “10x engineers.” Product managers love them because they turn roadmap dreams into demo-able reality at terrifying speed. AI amplifies the tornado: give them Copilot and a weekend and they will generate an entire product surface.

Then Monday arrives. The tornado has moved on to the next feature. Someone else inherits the code–no tests beyond the happy path, no runbooks, secrets in environment variables that nobody documented, an autoscaling policy copied from a blog post that assumed a different traffic shape. The tactical tornado created value in a burst. The strategic maintainer captures it over months.

Strategic maintainers are the people who go deep on the two or three features that actually drive revenue. They understand the edge cases customers hit at 2am. They know which database index is holding the query plan together and what happens when the table crosses 50 million rows. They are the ones who turn a flashy demo into a reliable product–incrementally tightening error handling, adding observability, negotiating with product managers about which “small” feature requests would actually require rewriting the payment flow.

Product managers sit in the middle of this tension. A good PM dreams up the features that will deliver the most value and sequences them so the team can ship without drowning in maintenance debt. A less experienced PM treats every sprint as a feature factory, stacking new work on top of un-serviced foundations, because the roadmap rewards visible output over invisible resilience. In the Serviced Software framing, the PM’s job is not just “what should we build next?” but “what is costing us the most to keep running, and is it worth it?”

The industry has historically rewarded tornados and feature-shipping PMs disproportionately. Promotions go to the person who launched the thing, not the person who kept it alive for three years. AI will sharpen this imbalance unless orgs explicitly revalue the strategic maintainer. When code generation is cheap, the scarce skill is not writing new software–it is understanding existing software deeply enough to keep it healthy.

The human layer

There is a deeper point here that goes beyond tooling. The shift from SaaS to Serviced Software is really a shift from building to caring. Building is exciting, creative, dopamine-rich. Caring is routine, patient, often invisible. Our industry has always undervalued the people who keep the lights on relative to the people who ship new features. AI will widen that gap unless we consciously correct for it.

The sysadmin, the SRE, the on-call engineer, the person who actually reads the CVE advisories and patches before the exploit drops – these roles are becoming more important, not less. The code is becoming commodity. The care is becoming scarce.

Where this goes

I do not think SaaS as a business model disappears. But I think the honest version of what customers are paying for will increasingly sound like: “We run and maintain this software so you do not have to.” That is not a new idea – managed hosting has existed forever. What is new is that the software layer itself is becoming thin enough that the operational layer dominates the value proposition.

We are entering an era where the question is not “can you build it?” but “can you keep it running, secure, updated, and affordable for the next five years?” The answer to that question has never been a one-shot prompt. It is a sustained commitment – and that, for now, remains stubbornly human.

Two Agents, One Codebase: An F1 Race Team Approach to Porting ACOLITE to Rust

2026-03-07T16:00:00.000-08:00

In Formula 1, every team fields two drivers. Not as a backup plan – as a strategy. One driver pushes the pace, forcing rivals to respond. The other holds position, manages tyres, and covers the alternative strategy. They share telemetry, they share a garage, but they are running different races on the same track. The team wins when both cars score points, not when one driver tries to do everything.

Porting a scientific Python codebase to Rust feels remarkably similar. You need the aggressive driver – the one who charges into unfamiliar code and lays down fast laps of Rust implementation. And you need the calculating driver – the one who reads the data, watches for degradation, and calls out when the numerical precision is drifting. Two AI coding agents, paired like Norris and Piastri, sharing a codebase but operating on different parts of the problem.

The Starting Grid: Why Rust for ACOLITE?

ACOLITE is RBINS’ atmospheric correction toolkit for aquatic remote sensing. It handles everything from Landsat and Sentinel-2 to hyperspectral sensors like PACE OCI (286 bands) and PRISMA (239 bands). The Dark Spectrum Fitting (DSF) algorithm is elegant – image-based, no external atmospheric inputs – but in Python, processing a full PACE scene involves reading 291 NetCDF variables, interpolating multi-dimensional LUTs, and correcting each pixel’s reflectance through a chain of gas transmittance, Rayleigh scattering, and aerosol models. On a decent machine, this takes around 230 seconds.

The seed was planted at FOSS4G 2025 in Auckland when Leo Hardtke ran a tutorial on Earth Observation processing with Rust. It was plagued by Nix environment issues (as I noted in my conference write-up), but when the code ran, it was fast. Zero-cost abstractions and fearless concurrency are not just slogans at that point – they are wall-clock seconds you are not spending waiting for your atmospheric correction to finish.

I had also been watching Rob Woodcock’s acolite-mp branch, which tackled the same performance problem from within Python. His approach was clever: per-band parallelism with memory budgets tuned to cloud CPU-to-RAM ratios (2, 4, or 8 GiB per core), replacing NumPy’s interpolation with the multithreaded pyinterp, and carefully managing the GIL contention that Python’s threading model inflicts on you. He got Sentinel-2 from 791s down to 197s and Landsat from 312s to 99s on a 24-core i9 – roughly a 3-4x speedup.

But the GIL is still there. The memory model is still Python’s. And as Rob himself noted, “further performance improvements are possible but require more extensive changes to the file handling” and “there is a fair amount of GIL contention which limits threading being caused by some structural choices in the implementation.” At some point, you are fighting the language rather than the problem.

Rust sidesteps all of this. No GIL. No garbage collector. Rayon gives you data-parallel iterators that map across bands or tiles with work-stealing. Memory usage is deterministic and known at compile time – you can profile it statically before deploying, which is a sentence that makes no sense in Python-land but is table stakes in systems programming.

The Pit Crew: Two Agents via ACP

Here is where the teammate analogy really kicks in. In F1, a team with only one driver is not half a team – it is no team at all. You cannot run a split strategy with a single car. You cannot use one driver to hold up a rival while the other pulls a gap. The performance of the pair exceeds the sum of the individuals because they create options that a solo driver simply cannot.

Porting 40,000+ lines of scientific Python to Rust is the same. A single AI agent writing Rust will drift – the implementation slowly diverging from Python’s numerical behaviour until your reflectance values are off by just enough to be scientifically useless. You need the second driver to keep it honest.

The solution I landed on was a multi-agent orchestration harness using the Agent Client Protocol (ACP), a JSON-RPC 2.0 protocol over NDJSON stdio that lets coding agents communicate in a structured way:

Agent	Role	F1 Equivalent
Kiro	Executor – writes Rust code, runs tests, reads files	Lead driver – pushes the pace, sets fast laps
Copilot	Proposer – reviews output, suggests next steps, cross-checks Python	Second driver – covers the strategy, watches the gaps
Human	Approver – filters proposals before dispatch	Team principal – makes the call on when to pit

The workflow per sensor port looks like this:

Human provides --task to the orchestrator (tools/agent_harness.py)
Kiro receives the task via ACP session/prompt and starts writing code
Kiro streams output via session/update chunks
Output goes to Copilot for review against the Python source
Copilot proposes ACTION: lines – “fix the gas transmittance interpolation order”, “the Rayleigh LUT needs pressure stacking”
Human approves or rejects
Approved actions go back to Kiro
Repeat until regression tests pass or maximum cycles reached

This is not vibe coding. This is a two-car team running a split strategy.

Think about how McLaren or Red Bull operate. The lead driver qualifies on pole and sets the pace in clean air. The second driver starts on a different tyre compound, runs a longer first stint, and emerges from the pits into a different part of the field. They are solving complementary problems – one optimises for raw speed, the other for strategic coverage. Neither is redundant.

Kiro is the lead driver. It attacks the Rust implementation aggressively – writing loaders, porting DSF algorithms, wiring up rayon parallelism. It sets fast laps. It also occasionally bins it into the gravel trap by hallucinating a NumPy broadcasting rule that does not exist in ndarray.

Copilot is the second driver. It reads the Python source, cross-references the Rust output, and spots where the gap to parity is growing. “The gas transmittance interpolation order is wrong” is exactly the kind of radio call a second driver makes – not flashy, but it prevents a DNF.

The human is the team principal. You do not override the drivers on every corner, but you make the strategic calls: do we pit now and fix this RMSE regression, or do we push on and address it in the next stint? Is a 0.002 RMSE difference in Sentinel-2 reflectance acceptable? (It is – that is within float32 precision.) When do we switch from tiled DSF to fixed DSF mode for this sensor?

Together, they converge faster than either alone, for the same reason that two cars gathering tyre data in free practice gives the team more information than one car doing twice as many laps.

The Telemetry: Regression Tests Against Real Data

In F1, both drivers generate telemetry. The team overlays their data – braking points, throttle application, cornering speed – to find where one is faster and why. The overlay is the truth. Not the driver’s feeling, not the engineer’s simulation, but what the car actually did on the track.

Regression tests are our telemetry overlay. The Python ACOLITE output is Driver 1’s trace. The Rust output is Driver 2’s. We overlay them pixel-by-pixel, band-by-band, and look at the delta. When the traces diverge, something real has changed and we need to understand whether it is a genuine improvement or an error we need to correct.

There are currently 141 Python regression tests that compare Rust output against Python output pixel-by-pixel across real satellite scenes:

Landsat 8/9: 13 regression + 13 Rust-vs-Python + 7 benchmark tests
Sentinel-2 A/B: 19 regression + 15 Rust-vs-Python + 9 benchmark tests
PACE OCI: 17 regression + 14 Rust-vs-Python + 12 DSF comparison + 12 ROI + 10 full-scene tests

The tolerances are tight. Sentinel-2 achieves RMSE < 0.002 (physics-equivalent). Landsat gets RMSE < 0.02. PACE full-scene (1710 x 1272 pixels x 291 bands) hits mean RMSE of 0.004 with 100% of pixels within 0.05 of Python. Correlation coefficients are R > 0.999 across all sensors.

These are not toy tests on synthetic data. They run against actual L1 scenes downloaded from USGS and NASA. When the tests break, something real is wrong.

The Performance Gap: Where the Seconds Go

Sensor	Scene Size	Rust	Python	Speedup
Landsat 8	62M px x 7 bands	66s	180s	2.7x
Landsat 9	62M px x 7 bands	56s	180s	3.2x
Sentinel-2 A	30M px x 11 bands	52s	182s	3.5x
Sentinel-2 B	30M px x 11 bands	64s	173s	2.7x
PACE OCI (full)	1710 x 1272 x 291 bands	84s	230s	2.7x

The PACE result is particularly satisfying. The key optimisation was switching from 291 per-band NetCDF reads to 3 bulk detector reads, then applying rayon-parallel atmospheric correction across tiles. Load is 12 seconds, AC is 34 seconds, write is 35 seconds. That write phase for a 291-band hyperspectral cube goes to GeoZarr V3 with gzip compression – try doing that in a Python event loop without your memory allocator throwing a tantrum.

Energy Efficiency: The Fuel Strategy Nobody Talks About

Here is the part where I get philosophical – and where the F1 analogy turns from metaphor into mirror.

Formula 1 underwent a fuel efficiency revolution in 2014. The FIA introduced hybrid power units, capped fuel flow at 100 kg/hour (monitored 2,200 times per second), and forced teams to extract maximum performance from minimum fuel. The result was not slower cars – it was faster cars that used less. The 2026 regulations go further: fossil carbon is prohibited entirely, the MGU-K will deliver three times the electrical power (350kW vs today’s 120kW), producing up to 1,000 horsepower while burning sustainable fuel. Less fuel, more power. That is not a trade-off – it is an engineering constraint that drives innovation.

The same constraint applies to scientific computing, we just pretend it does not. Cloud computing bills are denominated in dollars, but the underlying unit is energy. Every CPU cycle your atmospheric correction burns is a watt drawn from a power grid somewhere. When you are processing continental-scale Sentinel-2 archives or the full PACE ocean colour mission, those watts add up. Python is the V10 era of scientific computing – glorious, unrestricted, and profligate with resources.

Rust is the hybrid power unit. Its advantage is not just speed – it is energy per unit of work. A 3x speedup roughly translates to using a third of the compute time, which means a third of the energy, a third of the carbon footprint, and a third of your AWS bill. The Rust Foundation and others have pointed to studies showing compiled languages like Rust and C using an order of magnitude less energy than interpreted languages for equivalent workloads. Just as F1 teams discovered that fuel efficiency constraints forced them to build fundamentally better engines, switching to Rust forces you to think about memory layout, allocation patterns, and data flow in ways that Python’s garbage collector lets you ignore – until the bill arrives.

And here is the irony that would make an F1 sustainability officer wince: Earth observation processing is meant to monitor the planet’s health. Burning excess energy to do it is like running your emissions-monitoring car on leaded fuel. F1 recognised that the sport’s 20-car grid is only 1% of its total carbon footprint, but pursued fuel efficiency anyway because the technology trickles down. The same logic applies to EO processing pipelines. The individual savings per scene are modest, but at continental archive scale they compound – just like how F1’s hybrid innovations now power road cars from Ferrari’s SF90 to the electric components in every modern turbo engine.

Static memory profiling makes this tangible. In Rust, I can tell you at compile time that a Sentinel-2 full-scene atmospheric correction will peak at approximately N gigabytes of memory, because the allocations are deterministic. In Python, you find out at runtime – usually when the OOM killer visits your pod. F1 teams know their fuel load to the gram before the formation lap. Rust gives you the same certainty for compute.

Kubernetes 1.35 and Vertical Pod Autoscaling

This deterministic memory behaviour dovetails nicely with Kubernetes 1.35’s improvements to Vertical Pod Autoscaler (VPA). VPA watches your pod’s actual resource usage and adjusts CPU and memory requests/limits accordingly. When your workload has predictable resource usage – as Rust workloads tend to – VPA converges quickly to the right allocation instead of oscillating between OOM kills and wasted headroom.

For a processing pipeline that ingests satellite scenes of varying sizes (a Landsat scene is 62 million pixels across 7 bands; a PACE scene is 2.2 million pixels across 291 bands), VPA can right-size pods per sensor type. Rust’s static memory profile means the VPA recommendations stabilise fast, which means tighter bin-packing, which means more scenes processed per node, which means lower cost per scene.

Compare this to Python pods where memory usage is non-deterministic, garbage collection spikes are unpredictable, and the VPA has to overprovision to avoid OOM. The 2 GiB/core cloud ratio that Rob’s acolite-mp was carefully designed around becomes less of a constraint when your language does not waste half of it on interpreter overhead.

Out-of-Band Development: Preventing Merge Conflicts with Upstream

One design decision I am particularly happy with is keeping the Rust port on a separate feature branch (feature/rust-port) and treating it as out-of-band from the Python codebase. ACOLITE upstream is actively maintained by Quinten Vanhellemont at RBINS, with regular additions of new sensors, algorithm refinements, and bug fixes. A traditional “rewrite in Rust” approach would create an immediate fork that diverges with every upstream commit.

Instead, the Rust code lives in src/, benches/, and tests/ directories that do not exist in upstream Python ACOLITE. The Python code in acolite/ stays untouched. The regression tests are the synchronisation mechanism – they import both the Python ACOLITE modules and the compiled Rust binary, run the same scene through both, and compare outputs.

When upstream adds a new sensor or changes a gas transmittance coefficient, the regression tests fail in the Rust port. That failure is the trigger: it goes into the agent harness as a --task, Kiro investigates the numerical difference, Copilot cross-references the upstream commit, and the fix lands in Rust without touching a single Python file. No merge conflicts. No rebasing nightmares. Just tests that enforce parity.

This is how you keep an acceleration layer in sync with a moving target – you do not try to merge them. You test them against each other.

What Is Next: The Gap to Full Sensor Parity

The roadmap has the current state at 48 Rust tests, 141 Python regression tests, and three sensors fully validated (Landsat 8/9, Sentinel-2 A/B, PACE OCI). The architecture – loader, AC, writer – is clean and extensible. But three sensors out of 30+ is a qualifying lap, not a race win. Here is what closing the gap to full ACOLITE parity actually looks like.

Sensor Coverage: 3 down, 30+ to go

Python ACOLITE supports a sprawling constellation of sensors. The Rust port has ticked off the three highest-priority ones but the remaining fleet breaks into tiers:

Tier 1 – Near-term (shared loader patterns exist):

Sensor	Bands	Loader Type	Blocker
Sentinel-3 OLCI	21	NetCDF	Sensor def exists, needs full pipeline
PRISMA	239	HDF5	Shares pattern with PACE
DESIS	235	HDF5	Shares pattern with PACE
EnMAP	224	HDF5	Shares pattern with PACE
EMIT	285	NetCDF	Similar to PACE OCI

These are the low-hanging fruit. The PACE port proved out the NetCDF and hyperspectral GeoZarr writer path; PRISMA/DESIS/EnMAP share the HDF5 loader pattern. Each is a well-scoped --task for the agent harness – Kiro writes the loader and wires up the AC pipeline, Copilot validates against Python output on a reference scene.

Tier 2 – Medium-term (new loader work required):

Sensor	Bands	Notes
Landsat 5 TM / 7 ETM+	7-8	Older calibration metadata formats
PlanetScope (Dove/SuperDove)	4-8	Commercial format, GeoTIFF based
WorldView-2/3	6-29	Multi-resolution, pan-sharpening
Pleiades	5-7	DIMAP format
QuickBird-2	5	Legacy but still used
VIIRS (NPP/J1/J2)	22	Swath-based HDF5, three platforms
Aqua/Terra MODIS	36	HDF4/HDF-EOS
GOCI-2	12	Korean ocean colour mission

Each of these needs a dedicated loader – different metadata formats, different calibration approaches, different file layouts. The atmospheric correction core (DSF, gas transmittance, Rayleigh, aerosol models) is shared, but getting the radiometrically calibrated top-of-atmosphere reflectance array into the pipeline is the per-sensor work.

Tier 3 – Geostationary and niche (lowest priority for aquatic applications):

GOES ABI, Himawari AHI, MTG-I FCI, SEVIRI, Sentinel-3 SLSTR, AMAZONIA-1 WFI, CHRIS, HYPERION, HICO, HyperField, HYPSO, Tanager. Some of these (HYPERION, HICO) are decommissioned but their archives are still processed. Others (Tanager at 420 bands, HYPSO at 120) are newer hyperspectral missions that would benefit most from Rust’s performance advantage.

Beyond Loaders: The Algorithm Gap

Sensor parity is not just about reading files. Python ACOLITE has several processing features the Rust port does not yet implement:

ROI subsetting: Limit processing to a bounding box or polygon – critical for operational workflows that do not need a full scene
Ancillary data retrieval: NCEP ozone, pressure, and wind speed from NASA OBPG; currently the Rust port uses default values
DEM-derived pressure: Copernicus DEM at 30/90m for surface pressure estimation in mountainous coastal regions
Glint correction: Sun glint removal for low-latitude ocean scenes
RAdCor adjacency correction: The physics-based adjacency effect correction developed under the STEREO program
TACT thermal processing: Surface temperature from Landsat thermal bands via libRadtran – this one is architecturally interesting because it requires calling an external Fortran radiative transfer code
Interface reflectance (rsky): Sky reflection correction at the air-water interface
L2W water products: Chlorophyll-a (OC algorithms), TSS (Nechad, Dogliotti), turbidity, Secchi depth – the derived products that downstream scientists actually use

The L2W gap is the most consequential. Most ACOLITE users do not care about surface reflectance per se; they want chlorophyll maps or turbidity time series. Until the Rust port can produce L2W outputs, it remains a fast atmospheric correction engine rather than a complete aquatic remote sensing toolkit.

The Realistic Path

Closing this gap is not a sprint, it is an endurance race – appropriately enough. The agent harness makes each sensor port a repeatable, testable unit of work. The pattern is established: write loader, wire to AC pipeline, run regression tests against Python, fix deltas, validate on real data. Each sensor port takes the agents a day or two of focused work plus human review.

At the current pace, Tier 1 sensors are within reach in the near term. Tier 2 will follow as the loader library matures. The algorithm features (ancillary data, glint, TACT) are orthogonal to sensor coverage and can be developed in parallel. L2W is the final milestone – when the Rust port can ingest a Sentinel-2 scene and produce a chlorophyll-a map that matches Python to within measurement uncertainty, the port will be race-ready for production.

Each of these is a --task for the agent harness. Two drivers, one constructor’s championship. The lead driver pushes into unfamiliar sensor territory, the second driver validates against Python, and the telemetry overlay catches every divergence before it compounds into a retirement.

If the intersection of Rust, Earth observation, and AI-assisted development interests you, the code is all on GitHub. Feel free to ping me with ideas, bug reports, or competing approaches – especially if you have a cleverer way to handle the N-dimensional LUT interpolation. That one was a fun 3 days of Rapid Rust Rewrite fuelled by AI Amphetamine Analogs.

Copilot + Arduino CLI + Saleae: Driver Development for V93XX

2026-02-27T16:00:00.000-08:00

I have been spending time building out the V93XX_Arduino library for Vangotech energy-monitoring ASICs, and this is one of those projects where tooling makes or breaks momentum.

The pairing that worked best for me was:

GitHub Copilot for repetitive protocol code, tests, and workflow glue
arduino-cli for deterministic compile and upload loops
Saleae Logic Analyzer captures to confirm what is actually on the wire

The short version: Copilot gets you to a plausible driver quickly, but the logic analyzer gets you to a correct driver.

V93XX driver workflow walkthrough

Why this trio works

Developing protocol drivers is usually a game of “spec says one thing, silicon does another thing”.

If you rely only on serial logs, you can miss timing and framing issues. If you rely only on captures, you can waste hours writing boilerplate and one-off scripts. Using these three together gives a tighter loop:

Copilot proposes code and tests from your intent
Arduino CLI compiles and flashes quickly from scripts
Saleae confirms framing, parity, baud behavior, and CRC bytes
Copilot helps refactor after you learn what the bus is doing

In practice, this turned the V9381 UART ChecksumMode and waveform/FFT work from a stop-start activity into a repeatable pipeline.

Project context: V93XX_Arduino

Repository: whatnick/V93XX_Arduino

The library currently covers:

V93XX_UART for V9360 and V9381 UART modes (including address pins and checksum modes)
V93XX_SPI for V9381 SPI paths (faster acquisition, up to MHz clocks)
Examples for baseline comms, waveform capture, and FFT

Useful docs in the repo:

Practical workflow

1) Start with a machine-checkable baseline

Before touching hardware, run the local CRC and framing tests.

python tools/test_checksum_mode.py

If this is red, do not flash anything yet.

2) Use Copilot for structured implementation work

I get better results when prompts are specific and constrained. Example prompt:

Implement V9381 UART read path with explicit ChecksumMode behavior.
Keep Clean mode strict and Dirty mode permissive.
Add unit tests for CRC-8 edge cases and frame parsing.
Do not change public method names.

Copilot is strongest here at:

Building test scaffolding quickly
Keeping repetitive register access code consistent
Producing incremental refactors after captures reveal protocol edge cases

3) Consume datasheets with PDF MCP and keep them in-repo

Before generating more driver code, I now ingest the vendor datasheet with pdf-reader-mcp so Copilot can work from extracted register tables and command framing notes.

My workflow is:

Commit the original datasheet PDF to the repository, for example under docs/datasheets/v93xx/.
Use PDF MCP to extract the high-value sections (register map, UART/SPI timing, checksum/CRC rules, waveform buffer details).
Save extracted artifacts back into the repo as text or markdown notes, for example docs/datasheets/v93xx/register_notes.md.
Prompt Copilot using both code and extracted notes so generated changes are tied to concrete datasheet language.

This gives you a versioned paper trail from silicon docs to driver behavior, which is very useful when reconciling captures from the logic analyzer.

4) Compile and flash with Arduino CLI

For ESP32-S3 targets, this keeps the build deterministic and scriptable:

arduino-cli board list
arduino-cli compile --fqbn esp32:esp32:esp32s3 examples/V9381_UART_DIRTY_MODE
arduino-cli upload --fqbn esp32:esp32:esp32s3 --port COM3 examples/V9381_UART_DIRTY_MODE

PowerShell users can run the project helper script:

.\tools\run_automated_tests.ps1 -Port COM3

That pipeline can run unit tests, compile, upload, serial verification, and optional capture/analysis phases.

5) Validate on-wire behavior with Saleae

The logic analyzer phase is where protocol assumptions get tested.

For UART work:

Confirm bus settings (baud, parity, stop bits) match the target mode
Verify frame boundaries and inter-frame timing
Check CRC byte placement and value against expected payload sum/complement logic

If using the helper scripts in this repo:

python tools/capture_v9381_uart.py
python tools/analyze_checksum_captures.py [capture_dir]

This gives a concrete report of expected vs observed CRC and whether behaviour matches Clean vs Dirty semantics.

What changed in my debugging habits

Old approach

Edit sketch
Upload
Stare at serial output
Guess

New approach

Ask Copilot for narrow, test-backed change
Build and flash with Arduino CLI from script
Capture with Saleae and compare against expected frame math
Feed mismatch details back into Copilot for targeted fixes

It feels closer to hardware TDD than trial-and-error firmware hacking.

Trade-offs and cost notes

Arduino CLI is free and excellent for repeatable CI-style local loops
Copilot is a paid productivity tool, but pays for itself when protocol code churn is high
Saleae hardware is not cheap, but it can save days when parity/timing/CRC bugs are subtle

If you are doing serious serial or SPI driver work, a logic analyzer is not optional for long.

Troubleshooting notes that keep coming up

Wrong serial configuration (especially parity) can look like random CRC failure.
Upload success does not imply runtime protocol correctness.
Dirty mode can hide issues by design; keep a Clean mode path for strict validation.
Keep datasheet extracts in-repo so Copilot prompts reference real tables, not memory.
A scripted workflow (pdf-reader-mcp + arduino-cli + capture + analysis) beats ad-hoc manual steps every time.

Closing

For this V93XX work, the winning combination was AI-assisted coding plus old-school instrumentation.

Copilot helps me move faster, Arduino CLI keeps the loop reproducible, and Saleae captures keep me honest.

If you are building protocol drivers, treat those as complementary tools, not alternatives.

Loading FPGA Bitstreams with OpenOCD and Raspberry Pi - From Captain DMA to NeTV2

2026-01-27T16:00:00.000-08:00

A Hot Adelaide Afternoon and a Pile of FPGAs

I was recruited to wire together the NeTV2 to the Raspberry Pi 5 PCI bus by some means. Tim donated a bunch of hardware to make it work. I picked them up from his place, while he sorted through the contents of his container. The ultimate aim is to get them going as part of fpgas.online.

The box contained a treasure trove: an NeTV2 boards from bunnie’s (Andrew Huang) video overlay project, and lots of Raspberry Pi’s. As I went down the rabbit hole I acquired some more hardware for myself, Captain DMA development boards, and various other Artix-7 based experiments (Acorn, LiteFury). These boards share a common trait – they’re designed to appear as network devices on a PCI Express bus, which makes them interesting for all sorts of applications from video processing to… let’s say “creative” system analysis and game hacking using auto-target features.

The challenge with inheriting random FPGA boards is getting new gateware onto them. The original programming cables might be lost, proprietary software might be Windows-only and in Chinese, and documentation can be scarce. This is where OpenOCD and a humble Raspberry Pi come to the rescue. As long as you figure out which pins are the JTAG, and can get a PR merged.

Pi4 and Pi5 with NeTV2

The Problem: DMA Cards and Their Programming Woes

Captain DMA cards (and similar Artix-7 based DMA devices) typically come with a CH347 USB JTAG interface. The vendor provides a GUI tool, but it’s entirely in Chinese and only runs on Windows. For those of us who prefer reproducible, scripted workflows on Linux – or want to program boards headless on a Raspberry Pi – this is less than ideal.

The workflow the GUI implements is actually straightforward:

Connect to the CH347 JTAG interface
Detect the FPGA/JTAG chain
Load a BSCAN SPI bitstream for JTAG-to-SPI access
Program the SPI flash with the target FPGA image
Refresh the FPGA to load the new bitstream

The key insight is that we’re not programming the FPGA directly – we’re using JTAG to access the SPI flash chip that stores the bitstream. The BSCAN primitive acts as a bridge between JTAG and the SPI flash.

OpenOCD Scripts for Captain DMA

I’ve put together a set of scripts at https://github.com/whatnick/fpga-dma-scripts that replicate the Chinese GUI’s functionality using pure OpenOCD commands. The scripts support Linux/macOS and Windows, and handle:

Automatic BSCAN bitstream download from quartiq’s prebuilt collection
CH347 interface configuration installation
Version checking for OpenOCD (needs 0.12.0+dev for CH347 support)
Cross-platform flashing with proper error handling

Captain DMA clone attached to Raspberry Pi

Basic Usage

On Linux/macOS:

./flash.sh 75t /path/to/pcileech_75t484_x1.bin

On Windows:

flash.cmd 75t C:\path\to\pcileech_75t484_x1.bin

The scripts support three Artix-7 variants: 35T, 75T, and 100T. The underlying OpenOCD command is:

openocd -c "set BSCAN_FILE bscan_spi_xc7a75t.bit" \
        -c "set FPGAIMAGE pcileech_75t484_x1.bin" \
        -f flash.cfg

The flash.cfg orchestrates the JTAG chain detection, BSCAN loading, and SPI programming sequence.

Example Output

A successful flash looks like this:

Open On-Chip Debugger 0.12.0+dev-02377-gb9e401616 (2026-01-25-09:47)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
xc7_program
jtagspi_program
Info : CH347 USB To UART+JTAG from vendor wch.cn with serial number 0123456789 found. (Chip version=5.44, Firmware=0x45)
Info : clock speed 7500 kHz
Info : JTAG tap: xc7.tap tap/device found: 0x13632093 (mfg: 0x049 (Xilinx), part: 0x3632, ver: 0x1)
Info : [xc7.proxy] Examination succeed
Info : Found flash device 'win w25q64fv/jv' (ID 0x1740ef)
Info : sector 0 took 234 ms
Info : sector 1 took 250 ms
...
Info : sector 32 took 259 ms
Info : sector 33 took 256 ms

Each sector takes about 230-260ms to program. A full 75T bitstream covers around 34 sectors, so expect 8-10 seconds for the flash operation.

NeTV2 Wired over PCI-e

NeTV2: Bunnie’s Video Overlay Board

The NeTV2 (Network Television version 2) is bunnie’s open-source HDMI video overlay board. It uses the same Xilinx Artix-7 family (XC7A35T or XC7A100T depending on variant) and was designed for real-time HDMI signal processing. The board features:

Artix-7 FPGA (35T or 100T variant)
DDR3 SDRAM (K4B2G1646F)
PCIe x4 interface
RMII Ethernet
HDMI input and output
SD card slot
SPI flash for bitstream storage

The original NeTV2 scripts from bunnie’s repo (https://github.com/bunnie/netv2mvp-scripts) were designed for Raspberry Pi 2/3 using the older sysfs GPIO interface. I’ve submitted a PR (https://github.com/AlphamaxMedia/netv2mvp-scripts/pull/1) that updates these for Raspberry Pi 4 and upstream OpenOCD.

Raspberry Pi 4 GPIO JTAG Configuration

The key changes for RPi4 compatibility involve the bcm2835gpio driver configuration:

# Boilerplate setup for Rpi on Alphamax configuration

adapter driver bcm2835gpio

transport select jtag

set _CHIPNAME xc7a35t

# Raspi4 uses a different peripheral base address
# RPi 2/3: 0x3F000000
# RPi 4:   0xFE000000
bcm2835gpio peripheral_base 0x3F000000

# Speed coefficients tuned for RPi 3B+
bcm2835gpio speed_coeffs 100000 5

# GPIO pin assignments for JTAG
adapter gpio tck 4
adapter gpio tms 17
adapter gpio tdi 27
adapter gpio tdo 22
adapter gpio srst 24

reset_config none

adapter speed 10000

The GPIO pins map directly to the Raspberry Pi’s header:

Signal	GPIO	Physical Pin
TCK	4	7
TMS	17	11
TDI	27	13
TDO	22	15
SRST	24	18

SPI Flash Programming with Raspberry Pi

The cl-spifpga-rpi4.cfg config file handles the actual programming:

# Burn an FPGA image onto the SPI ROM

source [find interface/alphamax-rpi.cfg]

source [find cpld/xilinx-xc7.cfg]
source [find cpld/jtagspi.cfg]

init

jtagspi_init xc7.pld $BSCAN_FILE
jtagspi_program $FPGAIMAGE 0

virtex2 refresh xc7.pld

exit

The command to flash a NeTV2 from a Raspberry Pi:

/opt/openocd/src/openocd \
  -c "set BSCAN_FILE /home/k8s/netv2mvp-scripts/bscan_spi_xc7a100t.bit" \
  -c "set FPGAIMAGE /home/k8s/pcileech_netv2_top.bin" \
  -f /home/k8s/netv2mvp-scripts/cl-spifpga-rpi4.cfg

The output shows the Macronix SPI flash being programmed:

Info : BCM2835 GPIO JTAG/SWD bitbang driver
Info : clock speed 10000 kHz
Info : JTAG tap: xc7.tap tap/device found: 0x13631093 (mfg: 0x049 (Xilinx), part: 0x3631, ver: 0x1)
Info : [xc7.proxy] Examination succeed
Info : Found flash device 'mac 25l6405' (ID 0x1720c2)
Info : sector 0 took 286 ms
Info : sector 1 took 282 ms
...

Note the device ID difference: 0x13631093 for the 100T part versus 0x13632093 for the 75T. The flash chip is a Macronix MX25L6405 (ID 0x1720c2) versus the Winbond W25Q64FV (ID 0x1740ef) on the Captain DMA boards.

LiteX Builds for NeTV2

If you want to run something other than the original gateware, LiteX has excellent support for the NeTV2. The board definition lives at https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/kosagi_netv2.py.

Building a Basic LiteX SoC

Install LiteX following the standard setup, then:

cd litex-boards/litex_boards/targets

# Build for XC7A35T variant with Ethernet
./kosagi_netv2.py --variant=a7-35 --with-ethernet --build

# Or for the 100T variant with PCIe
./kosagi_netv2.py --variant=a7-100 --with-pcie --build

The target supports:

DDR3 SDRAM: Using the K4B2G1646F module with LiteDRAM
Ethernet: RMII PHY via LiteEth
PCIe: x4 Gen2 via LitePCIe
SD Card: Both SPI mode and native SDIO
LED Chaser: For the obligatory blinky demo

Platform Definition Highlights

The platform file defines the IO constraints:

_io = [
    # 50 MHz clock input
    ("clk50", 0, Pins("J19"), IOStandard("LVCMOS33")),
    
    # User LEDs
    ("user_led", 0, Pins("M21"),  IOStandard("LVCMOS33")),
    ("user_led", 1, Pins("N20"),  IOStandard("LVCMOS33")),
    # ...
    
    # SPI Flash for bitstream storage
    ("spiflash", 0,
        Subsignal("cs_n", Pins("T19")),
        Subsignal("mosi", Pins("P22")),
        Subsignal("miso", Pins("R22")),
        Subsignal("wp",   Pins("P21")),
        Subsignal("hold", Pins("R21")),
        IOStandard("LVCMOS33")
    ),
]

The platform also defines the programmer configuration, which now supports both OpenOCD and OpenFPGALoader:

def create_programmer(self, programmer="openocd"):
    if programmer == "openfpgaloader":
        return OpenFPGALoader(cable="digilent_hs2")
    elif programmer == "openocd":
        bscan_spi = "bscan_spi_xc7a100t.bit" if "xc7a100t" in self.device else "bscan_spi_xc7a35t.bit"
        return OpenOCD("openocd_netv2_rpi.cfg", bscan_spi)

Provisioning for fpgas.online

The ultimate destination for these NeTV2 boards is fpgas.online – a community project that provides free remote access to FPGA development boards. The service currently has around 10 FPGA boards connected, each with a webcam pointed at the LEDs so you can see your bitstream running in real-time.

Each board is attached to a Raspberry Pi that handles:

JTAG programming via the GPIO bitbang interface
Webcam streaming for visual feedback
Web-based terminal access for interactive development
Queue management so multiple users can share the hardware

This is where the Raspberry Pi JTAG programming approach really shines – you can reflash gateware remotely without any physical access to the boards. Combined with LiteX’s rapid build cycle, it becomes practical to iterate on FPGA designs from anywhere in the world.

If you want to try FPGA development without buying hardware, head over to https://ps1.fpgas.online/fpgas/ and claim a board. There’s usually one free and ready to use.

Why This Matters

These techniques apply to any Artix-7 board with SPI flash storage:

Reproducibility: Scripts in version control beat clicking through GUIs
Automation: CI/CD pipelines can flash test firmware automatically
Headless operation: A Raspberry Pi in a lab can program boards remotely
Cross-platform: The same OpenOCD configs work on Linux, macOS, and Windows
Tool preservation: When vendors disappear, open source tooling remains
Remote labs: Services like fpgas.online can offer FPGA access to anyone

The BSCAN SPI approach is particularly elegant – it uses a small “helper” bitstream that turns the FPGA’s JTAG interface into a SPI master, allowing you to program the flash chip that normally stores the FPGA configuration. Once programmed, a power cycle or refresh command loads the new bitstream.

Resources

fpgas.online Remote Lab: https://ps1.fpgas.online/fpgas/
Captain DMA Scripts: https://github.com/whatnick/fpga-dma-scripts
NeTV2 RPi4 Scripts: https://github.com/AlphamaxMedia/netv2mvp-scripts/pull/1
LiteX NeTV2 Target: https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/kosagi_netv2.py
BSCAN SPI Bitstreams: https://github.com/quartiq/bscan_spi_bitstreams
PCILeech FPGA Project: https://github.com/ufrisk/pcileech-fpga
Video Walkthrough: https://youtu.be/CNQnafoOTCM

The next time you inherit a box of mystery FPGA boards, you’ll know what to do. OpenOCD and a Raspberry Pi can breathe new life into almost any Xilinx 7-series board – no proprietary tools, no Windows VM, no Chinese language skills required.

SQRL Acorn boards

Feel free to ping me with other interesting DMA or video processing boards you’ve encountered. There’s always more gateware to explore.

Seven Years to Kubernetes: A Turing Pi 1 Christmas Miracle

2025-12-24T16:00:00.000-08:00

The Seven Year Itch (For Hardware)

Often I buy more hardware than I need. ~~Actually, strike that~~ – I always buy more hardware than I need. It’s a disease, really. This particular affliction manifested in 2018 when I pre-ordered a Turing Pi 1 because I had convinced myself that building a Raspberry Pi cluster would be the perfect way to learn Kubernetes.

It was not the perfect way.

Little did I realize that it would take me seven years to gather all seven Raspberry Pi Compute Module 3+ boards and finally bootstrap a k3s cluster. In that time:

Kubernetes went through approximately 47 major versions
The Raspberry Pi 4 and 5 came out (and experienced their own chip shortages)
I discovered my Turing Pi board had a faulty ethernet switch
I aged visibly, just look at my github profile and videos from recent conference presentations.

The Recipe for Homelab Kubernetes Suffering

In this write-up, I’ll outline what it actually takes to set up a Raspberry Pi 3+ cluster in 2025. Consider this a cautionary tale wrapped in a tutorial. I’ll probably resell this now-functioning cluster to another masochist – er, enthusiast – and use the recouped capital to buy something newer that will sit on my shelf for another seven years.

Step 1: Acquire the Base Board

The Turing Pi 1 was a great option back in the day. It’s a mini-ITX form factor board that accepts up to 7 Raspberry Pi Compute Modules in SODIMM slots. The on-board gigabit ethernet switch was supposed to be the killer feature – no external networking required!

Pro tip: Make sure the on-board switch actually works. Test it before you commit to this path. Mine didn’t, which I discovered approximately 6 years too late.

Step 2: Collect Your Compute Modules (Like Pokemon, But Expensive)

You’ll need Raspberry Pi Compute Module 3+ boards. The Turing Pi 1 can handle up to 7 of them. I sourced mine from Mouser Electronics, though availability has been… variable… over the years.

I really wish there were more alternatives in the SODIMM compute module format. If you’re in the business of making one with a newer processor and more RAM, let’s talk. Seriously. My DMs are open.

Step 3: Flash the OS (The Easy Part, They Said)

The Compute Modules have onboard eMMC storage, which is the preferred boot device. Trying to use SD cards will lead to disappointment, inconsistent boots, and existential questioning of your life choices.

Here’s the gear you’ll need:

A Compute Module IO Board - Something like the Waveshare CM3/CM3+ IO Board or the official Raspberry Pi IO Board to put the module in USB mass storage mode
rpiboot/usbboot - The tool that makes the eMMC appear as a USB drive
Raspberry Pi Imager - The official tool for flashing OS images

Critical step: Bake in your SSH public key during the imaging process. This will save you from having to find 7 spare HDMI cables and keyboards. The Pi Imager has a settings gear icon that lets you configure hostname, SSH keys, and WiFi – use it.


# Generate an SSH key if you don't have one

ssh-keygen -t ed25519 -C "kubernetes-cluster"

Step 4: Network Configuration (Here Be Dragons)

Plug in all the modules and fire up the on-board Turing Pi ethernet. If you’re lucky, the on-board network works and you can access all the nodes. Marvel at how easy this was.

If you’re me, you’ll discover the switch is dead and enter the seven stages of homelab grief:

Denial: “It’s probably just a loose connection”
Anger: Unprintable + emails to Turing Pi support and learning that the board is End-of-Life and unsupported.
Bargaining: “Maybe I only need 4 nodes anyway”
Depression: stares at pile of unused compute modules
Acceptance: “I guess I’m buying USB ethernet adapters”

The Workaround: Get a bunch of USB-to-Ethernet adapters like the TP-Link UE300 and wire them into an external switch.

Unfortunately, only 4 of the compute modules have their USB ports exposed on the Turing Pi 1. For the other 3, you’ll need to do some creative soldering to expose the USB D+/D- and power lines. That’s just 12 more flying wires on the board. What could go wrong?

USB Ethernet Adapters and Working cluster

Step 5: The Case Mod (Optional But Satisfying)

I got a nice acrylic case to put it all in. It has a fan connection on top for cooling, which you’ll need because 7 Pi’s generate surprising heat.

There were no extra slots for the 3 additional USB connections I needed. But I have a Dremel, two weeks of Christmas holidays, and absolutely no fear of voiding warranties.

Step 6: Actually Installing Kubernetes (The Easy Part, For Real This Time)

With SSH keys baked in, installing k3s is delightfully straightforward using k3sup (pronounced “ketchup”, because of course it is).


# Install k3sup

curl -sLS https://get.k3sup.dev | sh

sudo install k3sup /usr/local/bin/



# Bootstrap the first node as the server

k3sup install --ip 192.168.1.101 --user k8s



# Join additional nodes as agents

k3sup join --ip 192.168.1.102 --server-ip 192.168.1.101 --user k8s

k3sup join --ip 192.168.1.103 --server-ip 192.168.1.101 --user k8s

# ... repeat for remaining nodes

k3sup SSHes into each machine, downloads the necessary bits, and bootstraps a low-resource-friendly cluster with etcd (or SQLite) as the datastore. It’s genuinely magical compared to kubeadm.

Reality check: After the k3s install, the Pi 3 doesn’t have much headroom left for actually running applications. We’re talking about 1GB of RAM shared between the OS, kubelet, and your workloads. It’s a great testbed for learning the k3s API and running ARM binaries natively, but don’t expect to run your company’s microservices on it.

K3Sup based cluster setup

The Final Result

After seven years of procrastination, hardware hunting, debugging dead ethernet switches, creative soldering, and Dremel work, I finally have a working 7-node Kubernetes cluster.

It also serves as a rather festive Christmas decoration with the green PCB and red blinking LEDs. Very on-brand for the holidays.

What’s Next?

Hopefully I’ve been a good boy this year and Santa will bring me some newer clustering hardware to play with. The Turing Pi 2.5 looks tempting with its support for CM4, Jetson, and the Turing RK1 modules.

But knowing me, I’ll buy it in 2025 and finally get it working by 2032.

Hardware Shopping List

For anyone brave enough to follow this path, here’s what you’ll need:

Item	Link	Notes
Turing Pi 1 Board	Turing Pi	Check if ethernet works!
Raspberry Pi CM3+ (x7)	Mouser	8GB/16GB/32GB eMMC options
CM IO Board for flashing	Waveshare	Or official RPi IO Board
USB Ethernet Adapters	Amazon	Just in case
Ethernet Switch	Your choice	8+ ports recommended
Acrylic Case	Various	With fan for cooling

Software & Tools

Raspberry Pi Imager - OS flashing tool
rpiboot/usbboot - For eMMC flashing
k3s - Lightweight Kubernetes distribution
k3sup - k3s installer via SSH
etcd - Distributed key-value store

Feel free to ping me with your own homelab Kubernetes horror stories. Misery loves company.

Refactoring the Austender Scraper: From Colly to OCDS

2025-12-12T00:53:47.054-08:00

The AusTender analyser started life as a straight HTML scraper built with Colly, walking the procurement portal page by page. It worked, but it was always one redesign away from a slow death: layout shifts, odd pagination edges, and the constant need to throttle hard so I could sleep at night.

Then the Australian Government exposed an Open Contracting Data Standard (OCDS) API. That changed the whole game. Instead of scraping tables and div soup, I can treat the portal like a versioned data feed.

Part of why I care: I am kind of fascinated by government spending as a system. Budgets read like a mixture of engineering constraints and political storytelling, and I keep wanting to trace the thread from “budget line item” to “actual contract award” without hand-waving. The Treasurer’s Final Budget Outcome release (2022-23, “first surplus in 15 years”) is exactly the sort of headline that makes me want to drill down into the mechanics: Final Budget Outcome shows first surplus in 15 years.

So the redesign in austender_analyser does three things differently:

Fetch via OCDS, not HTML: Reduce breakage by consuming the API’s canonical JSON, not scraped pages.
Persist to Ducklake: Store releases, parties, and contracts in Ducklake so you can query locally without rerunning the whole pipeline. This does not quite work yet; I am treating it as a learning exercise with Ducklake. It is much easier to learn on a real problem than on toy demo datasets.
Treat caching as optional: Counterintuitively, the local cache is sometimes slower than pulling fresh data. Ducklake’s startup and query overhead can outweigh a simple, parallelized upstream call. The new design keeps the cache but makes it opt-in and measurable.

If you prefer Python, the upstream API team ships a reference walkthrough in the austender-ocds-api repo (see also the SwaggerHub docs and an example endpoint like findById).

Why move off Colly?

Scraping HTML is like doing accounting by screenshot. OCDS is the ledger export.
Less breakage: OCDS is documented and versioned; DOM scraping is brittle.
Faster iteration: You model on structured data immediately, not after a fragile extraction layer.
Clear rate behavior: You can respect API limits without guessing at dynamic page loads.

Why keep Ducklake in the loop?

Ducklake is the reproducibility knob. It lets me freeze a snapshot, replay transforms, and run offline queries when I am iterating on analysis (or when the upstream is slow, or when I just do not want to be a bad citizen).

But caches are not free. Ducklake has startup and query overhead, and that can be slower than simply pulling fresh JSON in parallel. So the pipeline treats Ducklake like a tool, not a religion: measure the latency, pick the faster path, keep an escape hatch when you need repeatability.

Current flow

Pull OCDS releases in batches, keyed by release date and procurement identifiers.
Normalize the JSON into Ducklake tables (releases, awards, suppliers, items).
Emit lightweight summaries for quick diffing between runs.

Lessons learned

A stable API beats heroic HTML scraping almost every time. Even in times of AI and (firecrawl)[https://www.firecrawl.dev/].
Caches are not free; measure them. Sometimes stressing the upstream lightly is faster and still acceptable within published rate limits.
Keep exit hatches: allow forcing cache use, bypassing it, and snapshotting runs for reproducibility.

Next steps: Going deeper : tighten validation against the OCDS schema, add minimal observability (latency histograms for API vs cache), and ship a “fast path” mode that only hydrates the fields needed for high-level spend dashboards. Going broader : find sites and build API and Web aggregators for Australian state tender sites (e.g. VicTender and international ones.

Solar Ceilings and Compounding Dreams

2025-12-06T03:36:26.069-08:00

It is fashionable to wave away physical constraints with vague references to solar abundance and human ingenuity. Yet every balance sheet eventually meets a balance of energy. Solar photons may shower Earth with roughly 170,000 terawatts, but financial markets expect growth that compounds on top of itself forever. The math linking those stories rarely appears in the same paragraph—so let’s put them together.

Setting the Stage

I keep coming back to Tom Murphy’s dialogue in Exponential Economist Meets Finite Physicist. In Act One, Murphy plots U.S. energy use from 1650 onward and it traces a remarkably straight exponential line at ~3% per year. Economists in the conversation shrug; after all, 2–3% feels modest. But compounding at that pace means energy demand multiplies by ten every century. Our economic models implicitly assume something even more optimistic : 8–10% returns in equity markets, pension targets, and venture decks; without asking what energy supply function supports that.

Thermodynamic Guardrails

Murphy distills the second law of thermodynamics into plain language:

“At a 2.3% growth rate (conveniently chosen to represent a 10× increase every century), we would reach boiling temperature in about 400 years… Even if we don’t have a name for the energy source yet, as long as it obeys thermodynamics, we cook ourselves with perpetual energy increase.”

That thought experiment matters less for the literal 400-year timer and more because it shows energy growth must decelerate to avoid turning Earth into a heat engine. Solar panels, fusion, space mirrors … pick your technology. The waste heat still has to radiate away. We cannot spreadsheet, app and AI our way around Stefan–Boltzmann and Black Body radiation.

Solar Arithmetic vs Demand Curves

Let’s grant the optimists a heroic build-out: cover 5% of Earth’s land area with 20%-efficient photovoltaic arrays, assume a generous 200 W/m² average output, and we net roughly 20 TW—about the entire human primary energy demand today. That is fantastic news for decarbonization, but it is not a blank check for compounding GDP. If demand keeps growing at 3%, we would need 20 TW × (1.03)ⁿ in perpetuity. Within 250 years we’d be trying to harvest thousands of terawatts—orders of magnitude more land, materials, storage, and transmission than our initial miracle project. Solar abundance is real; solar infinity is fiction.

Finance Is an Energy IOU

Money is a claim on future work, and work requires energy. When pensions assume 7–8% annual returns, when startups pledge 10× growth, and when national budgets bake in permanent productivity gains, they are effectively promising that future societies will deliver 2–3 doublings of net energy per century. If we instead hit a solar plateau—because land, materials, or social license cap expansion—those financial promises become unmoored. We can pretend that virtual goods, algorithmic trading, or luxury desserts (to borrow Murphy’s Act Four anecdote) deliver infinite utility without added energy, but the chefs, coders, and data centers still eat, commute, and cool their CPU’s , GPU’s and Tensor processors. The intangible economy rides on a very tangible energy base.

Rewriting the Business Plan

Accepting a solar ceiling does not doom us to stagnation. It just forces different design constraints:

grow quality, not quantity—prioritize outcomes per unit energy … do proof of useful work rather that roll the dice and gamble.
align finance with expected energy supply rather than mythical exponentials … and I am not talking of wasting energy on crypto.
treat efficiency gains as buying time, not as a perpetual motion machine … if you learnt enough physics in high school to reject the perpetual motion machine, but have been lulled into perpetual 8% returns from the finance markets, there is a serious schizophrenia issue.
embed thermodynamic literacy in economic education so debates start from the same math.

Murphy ends his essay noting that growth is not a “good quantum number.” It is not conserved. Our job is to craft institutions, portfolios, and narratives that can thrive when net energy flattens, because physics already told us that day will arrive long before our spreadsheets hit overflow errors.

Darwin 2022 - Ruminations Compendium

2025-12-06T03:05:00.000-08:00

Collected reflections from the July 2022 Darwin trip, a narrative of adaptation, organisational change, and expansion can live in a single place.

July 19 – Lemmings And Launchpads

There is no exception to the rule that every organic being naturally increases at so high a rate, that if not destroyed the earth would soon be covered by the progeny of a single pair. Even slow breeding man has doubled in twenty five years, and at this rate in a few thousand years there would literally be no standing room for his progeny. – Charles Darwin

Like the lemming marching and diving into the ocean to self‑regulate, humanity plunges itself into vices of its own creation: alcohol, drugs, violence, and greed. Perhaps the next plunge is into the real ocean or into the vacuum of space, chasing more room in which to stand or float. Failure in harsh environments creates room by removing weaker individuals, or greater resilience by rewarding the most adaptable. Colonial Australia itself was founded on such selection—the most adaptable individuals and the strictest rule enforcers reshaped an unforgiving frontier.

July 20 – Organisational Evolution In Flight

Seeing that a few members of such water-breathing classes as the Crustacea and Mollusca are adapted to live on the land, and seeing that we have flying birds and mammals, flying insects of vast diversified types, and formerly had flying reptiles. It is conceivable that flying fish, which now glide far through air, slightly rising and falling by the aid of their fluttering fins, might have been modified into perfectly winged animals. – Charles Darwin

The ability to skim over water for a few metres comes from external tweaks, but the ability to cross the Pacific like a Godwin Tern comes from internal rewiring: hollow bones, high metabolism, and a brain with a built‑in compass. Organisations face the same distinction. A brief digital-transformation spasm can bolt on an app or a website, yet sustaining that flight demands internal metamorphosis and a sense of direction from leadership. Caterpillars become butterflies through wholesale change—so must companies that aspire to be more than flying fish.

July 23 – Questions For The Corporate Naturalist

Where are the transitional forms?
Organisations with no lines on the org chart operate as pure adhocracy. Hidden behind corporate veils, they are like pupae in cocoons, waiting to emerge in a more defined shape.
How can specialised organs evolve?
Marketing machines, technology muscle, sales teeth, enterprise-planning backbone, analyst frontal lobes—each department is an organ honed for a specific survival task.
Is behaviour or instinct inheritable?
Culture answers this. The rituals, stories, and incentives that survive layoffs and leadership changes become the genetic code of the firm.
Why are some species sterile when crossed, while others are fertile?
Some mergers and acquisitions thrive; others fail because the two organisational genomes cannot integrate and diverge instead of hybridising.

July 24 – Conquering New Lands

He who believes in the struggle for existence and in the principle of natural selection, will acknowledge that every organic being is constantly endeavouring to increase in numbers; and that if any one being vary ever so little, either in habits or structure, and thus gain an advantage over some of that inhabitant, however different it may be from its own place, it will seize on the place of that inhabitant. – Charles Darwin

International expansion is a contest for ecological niches. Bringing hard‑won optimisations from one country to another is a bid to displace incumbents. The organisations that vary—by process, by product, by mindset—claim new ground first.

FOSS4G 2025 Auckland: Hazy Hops and Geospatial Heavy Lifting

2025-11-24T16:00:00.000-08:00

Every year, there is an international gathering of geospatial software geeks working in open source somewhere in the world. This has become known as the FOSS4G conference–with the “F” preferentially standing for “Freedom” rather than just “Free”. The last one of these I attended in person was a long while ago in Sydney (2009!). Since then, I have attended a couple of the local Oceania editions and even served on the board of the non-profit that organizes them, but this was my first global one in a long time.

It was great to see the community still alive with a lot of energy, attracting around 400 people from around the world to Tāmaki Makaurau.

Travel and Weather: The Metallica Tax

November in Auckland is very pleasant. Perhaps that is what drew so many events there at once. Accommodation was expensive to unaffordable due to a Metallica concert and an Indigenous education conference happening simultaneously. There were, of course, amazing shows and entertainment on offer as a result, but my wallet definitely felt the “Metallica Tax”. I ended up staying a 30-minute bus ride away from the city centre in Takapuna, which gave me some nice morning views of the Hauraki Gulf before diving into the windowless conference rooms.

Tutorials: Rust, AI, and No-Code Flows

EO with Rust - Leo Hardtke

Leo Hardtke’s tutorial was all about making Earth Observation (EO) processing faster with Rust and without the Python overhead. It was, however, plagued by Nix vagaries and the classic environment distribution issues that seem to follow “modern” build systems everywhere. When it worked, it was blazing fast. The code is available on GitHub for those brave enough to venture into the world of memory safety and zero-cost abstractions.

AI in the Frontend - Felix Palmer

Felix Palmer showed off a cool Claude-enabled frontend to make Deck.gl do things in response to free-text commands. Zooming and searching is just the beginning; we can also do custom frontend processing using the frontend equivalent of GEOS, Turf.js.

I hit a snag here: I could not get my account activation SMS from Claude while in NZ (roaming issues, I suspect). So, in true maker fashion, I ported the code to work with AWS Bedrock instead. It’s a good reminder that in the world of LLMs, being tied to a single API is a recipe for frustration.

Re:earth Flow - Kyle Waite

This was perhaps the most interesting tutorial from an open-source and national policy perspective. I often insist that science developers learn programming, but sometimes for adoption, this is simply not a feasible change. This is what creates room for numerous commercial and open-source tooling for GUIs and no-code/low-code workflow systems like FME, ESRI, and even the QGIS toolbox.

As part of the Plateau project, the Japanese government is putting together the “Flow” framework for prefectures to process and convert their own 3D models. The intro to the UI tools was amazing, and in time, it seems like it can challenge FME. The state management is done by y.js, and we had a fun time getting multiple users to modify the flow until it broke in spectacular fashion.

Presentations: The ones that stuck with me among many excellent ones

For the full roll of presentations check out the 150+ youtube playlist.

The Java Geospatial Ecosystem - Jody Garnett

Jody Garnett gave a deep dive into the state of the art behind the libraries doing the heavy lifting in the Java Geospatial ecosystem: JTS, GeoTools, and the long journey behind getting the Java Imaging library ported to an open-source equivalent supported by the Eclipse foundation. It’s a reminder that while Python gets the hype, a lot of the world’s spatial data still moves through Java pipes. I ended up also going to his talk on Geoserver, which keeps being a force to reckon with in serving GIS data at scale.

GPU Accelerated Zarr Loading - Wei Ji

Wei Ji’s presentation on GPU-native Zarr was a highlight. Optimizing data throughput for large-scale geospatial ML workflows is the new frontier, and moving the bottleneck from the CPU to the GPU is where the real gains are.

Is Zarr the new COG? - Jarrett Keifer and Julia Signell

An excellent presentation dispelling the hype around Zarr. The takeaway? Given the same input and compression, the resulting content is the same size. It’s the usage pattern and the “tyranny of chunking” that really affects performance, along with the fine points about using multiple files, shards, and inodes.

Navara Web3D Engine - Keiya Higuchi

The Re:Earth folks returned with Navara, a re-imagined 3D engine for the web using a modern stack including WebGPU. It aims to create more options for realistic rendering and a functional separation between GIS functions and visualization functions–which are often messily intertwined in current offerings.

STAC Adventures with Matt Hanson

One of the most fun presentations was Matt Hanson’s “Choose Your Own Adventure” session. He let the audience pick the path in D&D fashion to learn more about the STAC ecosystem. I threw in my two cents on the fact that “not all STAC is made equal” and the desperate need for conformance checks on hosting services that claim to be STAC compliant but fail in subtle ways that break downstream libraries.

ESA’s Zarr Foray - James Banting (SparkGeo)

ESA is restructuring the Sentinel satellite imagery archive into DGGS / Healpix Zarr format and publishing the EOPF toolkit. The SparkGeo presentation showed the state of implementation of the user-facing libraries and how they are being integrated into various tools.

After Parties: Hazy Hops and Community

The conference after-parties at the pub are where most of the real action happens. I spent my evenings hanging out with a mix of old friends and new faces, trying the famous New Zealand hazy hops. There’s something about a good IPA that makes discussing coordinate reference systems much more bearable.

Drop shipping products from PCBWay

2022-11-05T06:23:00.006-07:00

Drop shipping products from PCBWay

For a while I have been ordering PCB’s from PCBWay and parts from Mouser and Digikey, then hand assembling them at home. These have been very small scale cottage industry style runs and ultimately time consuming as I focus more on design and evaluation of new energy monitor ASIC’s such as the V9261F. When PCBWay started offering to stock and drop ship my PCB’s directly from the factory using their extensive clout with DHL, I promptly signed up for the service.

Recently I have been getting my ATM90E36 Devkit PCB’s assembled there. The service has been excellent with concierge like parts choice and purchase. Followed by extremely helpful consultation on assembly progress and correctness.

I received the following images after the first stage and confirmed the crystal and LED’s.

Then I received some more inspection photos to allay any doubts.

Looking forward to the stock appearing on the shop front.

NOTE: This is a paid promotion of PCBWay services

Trucks vs Trains as an analogy for Microservices vs Monoliths

2020-12-27T13:51:00.002-08:00

2018 and 2019 was mostly spent obsessing over containers, trucks, trailers and hand written paper invoices for me. I was helping build out the technology stack and engineering team for Lori Systems. Early in 2019 we made our first DevOps hire, getting Clive from Safaricom and getting started on migrating our handrolled Django monolith from EC2 to EKS. We would make jokes around shipping containers using containers. Clive even had a container shaped stressball with the EKS logo on it. This set me thinking on the parallels between shipping code and shipping goods, perhaps also led to the foundations of this post.

Intermodal Shipping in the real-world and in software

Over the almost 2-years of work in Logistics I learnt a lot about how the global logistics system works. Almost like the life-blood of the planet. Large container ships abstract away contents and ship things from Taiwan to Timbukutu. The seminal book on this topic is perhaps, The BOX. Watching global shipping lanes in Marine Traffic and scraping ships arriving in Mombasa from the KPA Sharepoint became a daily ritual. I digress, back to the original point on the importance for containerization in shipping code or machinery.

Docker uses the ubiquitous whale/ship logo, most containers arrive at ports this way from the oceans of developers. I don't quite have an analogy here for the massive ships that land the containers at ports, some 500 or 1000 TEU's at a time. The analogy here covers land transport aspects, somewhat related to how code runs in production and is typically served via datacenters / public clouds to users.

Containers themselves make transfer of goods/code from development (ships) to production (trains/trucks) easy. However even containerized applications can demonstrate tight coupling similar to what a train has, in effect being a distributed monolith, instead of a true suite of microservices. In my opinion, any system that requires a release train approach for new features is most likely to be a distributed monolith masquarding as microservices. The real flexibility comes from the low coupling between containers and the freedom to release each clearly delineated service at its own cadence on the roads.

Trains are awesome

My 5yo is currently obsessed with steam engines, even though they are from an era long gone. There is something magical about a powerful engine pulling everything along smoothly on a set of constraints (rails). It works nicely as long as no quick changes are needed in the carriages and everyone wants to get to the same destination. Trouble arises when something in the closely coupled chain of components goes awry and requires a quick change. I still don't understand the scene in snow piercer where a few wagons were dumped in a siding at speed. If we can do that one neat trick perhaps monoliths would become much more maintainable. In early stages of a product monoliths are a nice simple entry point, especially if the features are narrowly scoped and well coupled. On the reverse the monolith may be a very good idea for a mature product which is not changing rapidly and perhaps needs to be optimised for performance instead by reducing communication overhead between components by introducing tight coupling. In both cases a modular approach and service-oriented designs are still feasible, as long as the implementation and maintenance team is aware of the implications. People are still driving around in classic cars from the 1900's, where as steam locomotives from that era are languishing in museums.

Trucks are flexible

One of the killer advantages of trucks in the logistics business is their ability to deliver right to the factory or warehouse loading bay. It is simply not feasible to build train tracks to serve every address. Even in areas with great railway infrastructure, buffers (known as Inland container depots) have to be placed to cover the last few miles of transport from the rail to the industrial areas. This sort of mode can sometimes be seen in Microservices being layered on older monoliths to provide user facing services, especially in banking systems. The other great advantage trucks have is the ability overtake each other gradually along the road, this manifests itself in software systems as rolling deployment of new features. Such an approach requires careful management of the stateful parts of the system such as storage and database schemas. Otherwise it turns into a Fast and Furious game of stealing a container from a moving platform, aka the Romanian Rollover.

This analogy is not new

The logistics analogies are rife in software engineering, we ship code, we package things, we have release trains. The largest real world container orchestration organization Maersk uses a 7-point logo surprisingly similar to the most popular software container orchestration platform Kubernetes. I will continue updating this post as more ideas and links come together.

You can engage with article via comments or the twitter thread.

Random Roadtrip Rumination : Microservices vs Monotliths is Trucks vs Trains for shipping features.
— Tisham Dhar (@whatnick) December 26, 2020

Desktop Software API's in Python (KiCAD, FreeCAD, Blender, QGIS)

2020-10-04T15:19:00.004-07:00

Python wraps around everything

For the last couple of years I have mostly written Satellite Data Processing code in Python and plenty of Flask/Django web services. However Python is also an excellent automation tool for GUI based applications allowing custom plugins to be written and functionality provided out of the box extended by users.

The first desktop application I seriously looked at Python plugins for was QGIS. It was early days of learning how to wrap C++ code using SWIG/SIP etc. In the old mailing list you can find a much younger me making inane comments about mixing wrapper metaphors in QGIS with SWIG + SIP. We have come a long way since then and SIP based bindings are the mainstay of QGIS plugins.

QGIS

QGIS has so many Python plugins that they need a registry of their own. Occasionally QGIS Python gets twisted around itself due to multiple Pythons in the user enviroment. You can also flip the python API around and instead of building a plugin you can turn QGIS into a custom desktop application. Which is what I have done with my basic Airport Viewer demo.

QGIS being a fairly extensive and complex C++ application which takes hours to compile, being able to make small quick changes in python is invaluable.

KiCAD

At the time of writing KiCAD has an extensive Python API for processing the automating the PCB layout part of the workflow and this has lead to many innovations in automating traditionally laborious hand layout or even performing complex simulations / optimization to set trace lengths. For example Josh Johnson has one for laying parts out in a circle and Greg Davill has several for length matching and rendering file generation. My personal favourite among the KiCAD scripts is the one for generation of Interactive BOM.

I am really looking forward to Python script support in the Schematic Editor. Meanwhile programmatic Schematic generation tools like Skidl provide schematic oriented Python fun.

The rendering of the PCB's is often done in Blender. Which has its own set of Python nicities.

Blender

My first foray in creating a Blender API based application was during the Kinect USB protocol hacking days. The data stream had just been decoded and I wanted an easy pipeline to a commonly installed / open-source 3D display software. The Python API is mature enough for people these days to quickly put together motion capture plugins for Blender. This plugin however demonstrates the challenges for creating native plugins for blender, the .pyd files for Python have to be recreated for different versions of Blender for ABI compaitibility.

Getting the binaries working has had me thrashing about and posting in forums, then sticking to a working Blender build with Python 2.7 for about 5 years since I did not want to touch it and break it. My integration actually reversed the embedding process, i.e. instead of using additional modules in the Blender embedded python I embedded Blender in a 3D GIS automation.

Native plugin weirdness aside, Blender Python API is a really powerful tool for creating procedural objects from waves / fluid simulation to astrophysics with amuse.

FreeCAD

FreeCAD is sort of the third part of my physical electrical / mechnical design triumvirate. I occasionally design parts for KiCAD in FreeCAD, or bring multiple boards together to test enclosure fit. FreeCAD also has an extensive python library which is leveraged by KiCAD part library maintainers to parametrically generate parts.

The scripting in FreeCAD can be used much like the PCB layout scripts in KiCAD to create this with circular symmetry, like ball bearings which are difficult and repetitive to do by hand.

Final words

There are lots of other pieces of desktop software I have used that have started shipping with Python API's to address the never ending demand from users to easily automate repeated tasks. The live process for making this blogpost in somewhat recursive fashion can be found here.

Started a new blog post around #python wrapping around a bunch of desktop applications to ease automation / extension. Live updates here as I collate material to flesh it out. pic.twitter.com/zz8b29kHcO
— Tisham Dhar (@whatnick) October 4, 2020

I have even made videos withs a proprietary one, I will live that here for anyone interested in my attempts at a voiceover.

Compiling QGIS in MSVC in 2020

2020-08-30T00:12:00.004-07:00

Compiling QGIS on Windows in 200x

I don't quite remember when I decided to help compile QGIS on Windows. It was somewhere between compiling GDAL with ECW support for Photoshop on Windows and getting carried into Direct3D and C# land with NASA WorldWind. It was sometime in the 2000's while still working at Apogee Imaging in Lobethal.

At that point I was manually building a database of the footprints of satellite imagery that filled up a wall cabinet with CD's and DVD's. The technique was something like open up the image, go around edges and trace a polygon. This was days before mature boolean thresholding and reliable/easy raster-to-vector logic.

I hopped on IRC on #qgis in Freenode and chatted with luminaries like timlinux, frankw and gsherman. Listened to the automated notifications from sigq, the commits bot. Things were heating up and instead of a Linux cross-compile to windows using MingW, something native to windows say using MSyS+MingW instead of Cygwin was desired. A lot of GDAL and Qt worked in MingW, so presumably QGIS would too. So I set myself to put together an MSYS environment with all the third-party dependencies that could be used to happily build QGIS. Eventually I built a release in NSIS as well.

My MSYS environment got packed in a zip and shared via FTP/HTTP on a VPS I had back then to the rest of the community. I earned myself a pin in the QGIS core contributor map in Adelaide. Something I am very proud of to this day. Eventually the MingW build got deprecated and native MSVC builds were supported. That's how contributions work, nothing lasts forever. In my IRC days, I helped on-board Nathan Woodrow to QGIS, who in turn I believe helped on-board Nyall Dawson. Nyall has surpassed us all in feature contributions and work on QGIS.

Getting started on QGIS on MSVC blog article. Trying to add some historical perspective on my involvement. Found this acknowledgement in the docs. Thanks to @timlinux . Back in the days of sharing mingw zip files via FTP. pic.twitter.com/NTkQgU5ibN
— Tisham Dhar (@whatnick) August 29, 2020

Fast forward to 2020, compiling QGIS in MSVC

I am getting back into doing lots of Open-source work after long hiatus in private industry with Aerometrex and start-up land with Lorisystems. It is great fun working on mostly in the open at Geoscience Australia. There is actually a recently archived opendatacube + qgis repository here. Seeing that repo and speaking to Nathan and LinuxConfAu inspired me to have a go and getting back into actively working on the Qgis code base. It has sprawled out, with lots and lots of new features. The build system is still familiar via CMake and actually much easier now with MSVC. I cast around for a recent guide and found this. The guide mostly works, however I made some refinements.

Ditched bison and flex via Cygwin to using the one available via Msys2. These can be found here. Not needing the while Cygwin system helps in keeping the windows build system light. Simply download the binaries and add them to the Osgeo4W binaries directory.
Captured my CMakeCache.txt to make it easier to reproduce and debug the build environment for others.
Used Incredibuild in demo mode to use a few NUC's I have lying around to speed up the build. Recording while building failed the first time and worked the next. The whole build from scratch still tooks around 35minutes overall.

I am planning to throw some of my day to day DevOps skills towards the QGIS project and start helping again with Raster enhancements and windows release management. Perhaps getting Incredibuild in the hands of the windows maintainers will help tighten up the iteration cycle and make testing easier.

The twitter thread/ stream of consciousness edition of this is available as well.

Moving from babun to msys2 to get flex and bison required for compiling QGIS from source. https://t.co/W866AOM9Kt pic.twitter.com/fUmqpa9YHY
— Tisham Dhar (@whatnick) June 26, 2020

Microservices the hard way - folders in EC2 Instance

2020-07-22T00:12:00.006-07:00

For day to day work I wrangle containers in EKS these days. However when doing personal projects EKS is a luxury (baseline cost being $70 or so per month). So I decided to do microservice development for the rain radar project using no Docker, no Kubernetes but using:

multiple venvs
multiple service folders
environments in .env files (secrets in plain text)
web service start using @reboot in Cron
scheduled services using ... ya Cron

The whole thing started with noble intentions to use lambda's all the way however I got stuck in using S3-SNS to trigger the lambda and decided to scan the S3 bucket using timestamps to find latest files to process. More on the pitfalls of that later.

The major microservices handle are:

Raw radar data preparation using custom hand crafted algorithm, being ported to Rust here.
Inserting prepared data to DynamoDB as a sparse array and serving this via Flask.
Nowcasting using the timeseries of sparse array of rain observations also serving results via Flask.
Capturing rain events and nowcasts and creating text and gif to send to twitter.

Each of these applications consumes the other to some extent and is sort of separated in responsibility. I decided to deploy them with basic a folder per application on the /home/ubuntu directory, with a venv per folder.

I had it like this for a while. Then I got tired for sshing into the box and git pulling in each folder. So I decided to write a fabfile per application which would do this for me and created deployment keys which would be used to pull the code to this folder. Then I got tired of running multiple fabfiles and decided to setup a polled process which run the fabfiles and git synced the code from a master pipeline.

Eventually I got around to bootstrapping the whole VM using Packer + Ansible playbooks. The development work for it was done locally using Vagrant with Hyper-V as the VM provide to test the same Ansible playbooks. I will follow up on this with a few characters on twitter.

Wrangling VM's with Packer, Vagrant and Ansible all at the same time. Helped me figure out issues in Ansible docs and divergence between bento ubuntu box and AWS AMI. One calls ntp ntpd, the other ntp.service. pic.twitter.com/tSbaPuKtVB
— Tisham Dhar (@whatnick) July 18, 2020

Once the initial Packer AMI is established the choice is to either keep building this image or to move away from the whole VM based old-school stuff to a more modern/fun Kubernetes way.

Replicating Databases in RDS

2020-06-01T06:59:00.005-07:00

One Sunday in 2018 I sat for a whole day in Art Caffe at the ground floor of Yaya Centre in Nairobi on the phone to Norman at AWS Support in Cape Town discussing DMS for MSSQL servers. After a whole day of screen sharing and being on call we decided what we were trying to do was no achievable, but AWS was working on it. The next day AWS sent me an NDA (since expired).

Data replication from on-prem Database instances or between cloud database instances is an issue that comes up all the time. I have hands on experience doing this a couple of times now. This post summarizes my 3 or so attempts at doing this with different sources and targets and lessons learnt.

MSSQL to MSSQL using AWS DMS

At the start-up I was working at we adopted a pre-built mini ERP, it covered logistics workflows and finance / billing aspects. It was built in the 2000's in .NET Classic and ran on IIS and MSSQL server. Quickly the MSSQL became the single point of lack of scalability in the system. Since AWS does not natively support read-replicas for MSSQL RDS instances I looked at DMS to create these replicas. DMS did not quite work as expected and led to the conversation alluded to above with Norman. I ended up performing replication using CloudBasic as the SaaS provider for managing the Change Tracking and handling schema changes in the source tables and propagating them to the target replicas. The replication was fine for single replicas, but quickly bogged the source database down as I added more replicas.

As aside the same database was also being replicated to a Redshift cluster for BI usage using tooling provided by Periscope Data.

As part of this excercise I came to appreciate the advantage to write-only / append-only schemas in improving no-lock replication performance (at the cost of storage), also the need for timestamp columns such as update_time to perform incremental data transfer. I spent a lot of time reading the Schemaless articles by Uber Eng around building Schemaless DB's on top of RDBMS's like MySQL. I don't 100% agree with their design choices but it adds interesting perspective. The bottomline CRUD at scale is HARD.

RDS PostgreSQL to PostgreSQL using DMS

Fast forward a year or so, I am now working at Geoscience Australia, with the Digital Earth Australia. Everything runs on Kubernetes and is highly scalable. Single point of lack of scalability is again the database. A pattern seems to be emerging here. We were performing cluster migration in Kubernetes and I offered to investigate DMS again.

In the MSSQL scenario there is a small prologue, I had previously migrated around 1million cells from a massive Google Sheet to the MSSQL database at the start of my tenure at the startup, by the time we hit scalability issues in the single instance MSSQL we were at 10million rows in the largest append-only table. The PostgreSQL migration of the datacube tables featured 8-9 million rows in the largest table. However the database also has lots of indexes and uses PostGIS for some applications, particularly Datacube Explorer. DMS fell down in support for the Geometry columns, however I learnt a lot in setting up DMS using Terraform IAC and fine tuning for JSON Blob columns, which in Datacube design in 1.8.x series can be upto 2MB in size. DMS migrates standard columns separately from LOB columns.

Ultimately DMS was not feasible for datacube DB migration to a new RDS instance. However I believe core datacube can be migrated next time I try with applications depending on Materialized views and PostGIS setup afresh on new DB. Also by the time I try again Amazon may have better PostGIS support. For the cluster migration we ended up using a snaphot of the last DB.

On-prem PostgreSQL to RDS PostgreSQL

There is a Datacube PostgreSQL DB instance at the NCI which needs to be regularly replicated to RDS. It powers the Explorer Datacube application. However DB migration from one server without direct disk access to RDS where we also don't have disk access using pg_dump / pg_restore for a DB with largest tables being around 22 million rows and the compressed dump being around 11GB is a long running task. Ideally we sort something out that is incremental using update_time generated using triggers. The options explored so far are :

Using an Airflow DAG with Kubernetes Executors wrapped around pg_dump/restore with some application specific details.
Using COPY from S3 support for Aurora PostgreSQL, CSV's restored using the COPY command are generated incrementally.
Using PostgreSQL publish / subscribe and Logical Replication. Networking over the internet to maintain connectivity securely to the on-prem instance via SSH Tunnels and to the RDS instance via EKS port-forwarding.

ADE7816 Energy Monitor

2020-05-07T15:11:00.002-07:00

I have been meaning to try out the ADE7816 Single-phase 6 current channels energy monitor for a while. However time has been lacking for the last couple of years. Finally I have a working version with successful board bring-up and a semi-working Micropython driver, with an Arduino driver in the works.

PCB Design

The PCB design process for this was not easy mostly due to a footprint choice mistake on my part. I had placed the 5x5mm QFN part instead of the 6x6mm QFN part in KiCAD. This made the DRC fail everywhere in standard settings. However it ended up being a collaboration opportunity with Greg Davill who loves to practice and photograph bodging stuff. So I now have a work of art at hand instead of a non-functional board.

Practising my bodge skills

This is a QFN where the footprint on the PCB was incorrect.

Just a couple more bodge wires to add. pic.twitter.com/AmZ1CsWqC3
— Greg (@GregDavill) April 22, 2020

I am even debating whether to place the rest of the parts and possibly take away from the dead-bug awesomeness. Next time need to order parts in advance and make sure I do 1-1 prints to verify footprints before pulling the trigger on PCB's.

Energy Montor Details

Now to more about the energy monitor. This ASIC features 3 single-ended and 3 differential current inputs and a single-phase voltage input, all in very compact 40-pin 6x6mm QFN package. In fact the PCB is large on purpose to accomodate ease of use with stereo-jack type current clamps. The main usage would be in standard households where there are typically 3-4 lighting circuits, 1-2 socket circuits and dedicated Air Conditioning circuit. A single energy monitor could be built to monitor all channels using a single-ASIC and leave out fancy NILM stuff from worrying about the lights. The socket circuits could have anything plugged into them and can potentially have point-of-load monitoring instead of breaker board based monitoring. All this translates to more data being generated for IoT platforms and some sensible firmware work needs to be done to handle that.

ADE7816 Driver Development

This is still work in progress. I have done some initial exploration to find prior art. Nothing exists yet from Arduino however there is some register lists from a Javascript driver written for the now defunct Intel Edison.

Intel never quite had the maker market pinned right to market that board, it makes me sad to think of all the engineering ours sunk into a now defunct platform. Open-source software / hardware helps us salvage some of that. I also sped up the register listing by copy-pasting the ubquitous table from the ADE7816 datasheet and dropping it into a Jupyter notebook to parse all the registers, not as fancy as the Pdf parser I had built before, but much more reliable.

My @adafruit #ESP32 + Logic Analyzer test rig. Due in the background is just to keep cables tidy. The ESP32 runs micropython so that I can iterate on SPI modes quickly and use the excellent struct functions to pack/unpack bytes. https://t.co/0pQgAO1Cgl pic.twitter.com/R2imRukrpT
— Tisham Dhar (@whatnick) April 14, 2020

My driver development follwed the now tried and tested Micropython + Jupyter Notebook + Logic Analyzer path. I used an ESP32 feather as host processor with standard micropython loaded and probed the SPI bus with read-write packets for known registers until the protocol gave in and started responding with some values. The ASIC is super versatile in supported protocols - I2C Slave, SPI Master and SPI Slave modes are all viable. So developing a fully functional driver supporting all the possible modes will take a while. The initial work so far is on the SPI slave mode since all my other work in DIN rail and Featherwing formats is linked to the SPI bus, however the I2C mode can be really interesting for host-processors with fewer pins and flaky SPI support (while having solid I2C support - like the Onion).

If anyone is interested in driver development I am happy to send you a board or you can get one yourself from Oshpark, Aisler or PCBWay. Once the drivers mature I will list it for a wider audience on Tindie.

Distributed Locking from Python while running on AWS

2020-04-04T00:12:00.006-07:00

In the day and age of eventually consistent web-scale applications the concept of locking may seem very archaic. However in some instances attempting to obtain a lock and failing to do so within a limited window can prevent dogpile effects for expensive server side operations or prevent over-write of already executing long running tasks such as ETL processes.
I have used 3-basic approaches to create distributed locks on AWS with the help of built-in services and accessed them via Python which is what I build most of my sofware in.

File locks upgraded to EFS

File based locks in UNIX file-systems are very common. They are typically created using the flock command, avalaible in Python under os-specific flock API. Also checkout the platform independent filelock. This is well and good for a VM or single application instance. For distributed locking, we will need EFS as the filesystem on which these locks are held, Linux-Kernel and NFS will use byte-range locks to help simulate locally attached file system type locks. However if the client loses connectivity the NFS lock-state cannot be determined, better run that EFS with enough replicas to ensure connectivity.
File locking this way is very useful if we are using EFS for holding large file and processing data anyway.

Redis locks upgraded to ElastiCache

Another popular pattern for holding locks in Python is using Redis. This can be upgraded in the cloud-hosted scenario to Redis-Elasticache, This pairs well with the redis-lock library.
Using redis requires a bit of setup and is subject to similar network vagaries and EFS. It makes sense when using Redis already as an in-memory cache for accelration or as a broker/results mechanism for Celery. Having data encrypted at rest and transit may require running an Stunnel Proxy.

An AWS only Method - DynamoDB

A while ago AWS published an article for creating and holding locks on DynamoDB using a Java lock client. This client creates the lock and holds it live using heart-beats while the relevant code section executes. Since then it has been ported to Python and I am maintaining my own fork.
It works well and helps scale-out singleton processes run as Lambdas to multiple lambdas in a serverless fashion, with a given lambda quickly skipping over a task another lambda is holding a lock on. I have also used it on EC2 based stuff where I was already using DynamoDB for other purposes. This is possibly the easiest and cheapest method for achieving distributed locking. Locally testing this technique is also quite easy using local-dynamodb in a docker container.
Feel free to ping me other distributed locking solutions that work well on AWS and I will try them out.

Testing the OrangeCrab r0.1

2020-01-24T05:48:00.001-08:00

After hassling Greg Davill for a while on twitter and admiring his OrangeCrab hardware I managed to catch up with him in person in Adelaide. I have been away in Nairobi till October last year, then I spent a brief few days in Adelaide before coming over to Canberra to take up a position in Geoscience Australia. The new gig is much less time commitment than the start-up world and hopefully will allow more time for blogging and board bring-ups like this one.

I caught up with Greg at a Japanese restaurant in Rundle mall and was treated to his now trademark led cube and led icosahedron. They are insanely detailed pieces of work and deserve staring at. However I am most grateful for the care package he left me, an OrangeCrab v0.1. This is an ECP5 board in feather form factor with an ADC built in to respect the Analog In pins on the feather. My aim for this board is to host some energy monitoring code on the FPGA with a fast 4mbps or so ADC and perform power/energy calculation on some parts of LUT's/DSP and have a softcpu push data out.

Greg also left me a home-made FTDI based board to use a JTAG programmer. The whole setup requires 3 USB cables:

To plugin and power the FPGA board (eventually it should be alos programmable via this port)
To attach a USB-Serial converter and watch the console when the gateware comes up
To program the board over JTAG using an FTDI chip

Getting firmware compiled these days is getting easier, but Greg had done his initial testing with Lattice Diamond. I managed to installed it in WSL and promptly ran into a tonne of issues. The weirdest being close coupling to bash, Ubuntu actually uses dash as its default shell. You can get a Diamond licence and help support integration of diamond in litex-buildenv.

I was about to give up then Greg got it working with the opensource toolchain and [NextPNR-ECP5. I had by then setup litex-buildenv to support the orangecrab. So getting some gateware on was relatively easy. Then I got stuck on RAM timing bug in Litex till a few hours ago when I tested out some new gateware.

Also Checkout out Greg's foboot fork and help make programming over the USB possible and reduce 1 USB cable. More work on this including getting MicroWatt running coming soon.