So, I gave it a shot. I used it for probably 2-3 months. It’s a fine tool, as is Ghostty, but these things are mostly solving issues I don’t really have. I’m not someone who uses vim or emacs as my primary IDE, and I don’t need to run a command in 50 separate terminal windows simultaneously (er, anymore, ask me about my first beowulf cluster in 2003!). I also don’t switch my working context across terminal panes in a single window, I do it at the macos desktop level (using SpaceJump, which I recently found and is helpful for that). I have a desktop per project, and each desktop has the terminal windows, browser windows, documentation, and IDE windows for just that project (I have 12 desktops, currently).

Ultimately, I gave iTerm2 another shot after a long time (years, actually) just using the native macos Terminal application. It turns out that iTerm2 actually has vertical tabs, which is nice. Also, the TUIs for Codex and Claude Code both set pretty good window titles, and notifications happen exactly like they did in cmux. I had issues with the cmux in-terminal browser, so I never used it, so in all, I can’t say I’m really missing a whole lot from not using cmux.

This in no way is a knock on cmux, or anyone who uses it. I’m also not advocating that others not use whatever they want. I’m just recording my current mindset. At some point I’m sure it’ll change or I’ll go in another direction, which makes posts like this interesting in a historical chronology context.

Quick Historical Summary

I started using LLMs in 2022 with the release of ChatGPT. I was pretty quickly fascinated by it and my mind exploded with ideas and questions. By early 2023 I was training my own model, because I discovered the term “training” before I knew what “RAG” was. Lots of manual labor, embeddings, vector databases, laptops were crashed… I loved every minute of it. Improvements to every single part of the AI ecosystem seemed to happen overnight. Between 2023 and early 2026 I tried to build the same IOS app three times. The first time I was cut & pasting code from ChatGPT into XCode, there were no coding agents, the code was pretty bad, the training data cutoff was forever ago, etc. The third time, the training cutoff was just a couple months ago, the code was pretty good, everything from code implementation to pushing PRs was done by an agent, and that attempt actually tested out and I put it in the app store. In the same period, I went from AI being forbidden at one job, to it being required at another job, to now being at a place that trusts engineers to make things happen, which turns out to be the most effective approach I’ve seen. Maybe I should write about that.

Today

I am a software engineer who started their career focused on systems/network/data infrastructure. I co-authored a book about Linux system administration, and one about Python development, to give an idea about the sort of things I like to get into. I’m lucky to have been able to mold a career around not just being either a sysadmin or a developer, but landing in environments where I was able to wear lots of hats and add value in both of those spaces. My current environment allows for that, too.

I’m using AI in a lot of different contexts as a result. I use it in monorepos and small repos. I use it for infrastructure, software engineering, data engineering, research & ideation, and any other category of task that comes up. I use it with Golang, Python, Typescript, Javascript, Terraform (HCL), bash & zsh, awk & sed, regexes, SQL, Scala, and jq. I use it against various SaaS APIs, I use it to draft communication, create slide decks, etc.

I have also used it to obviate the need to use external vendors for various things, and I expect that to continue. I find that to be the most interesting result of using AI: the idea that it’s possible to skip having expense report line items for github repo backups or security awareness training, because you can now build those things using some terraform, or Google Workspace tools, in a matter of hours.

Tooling

TL;DR – I have used Claude Code since it came out. I started using Codex when that came out, but it hasn’t been able to pull me over from Claude Code as my daily driver. However, Sol 5.6 just came out a week or so ago, and it is pretty amazing. Just in time as Anthropic is about to (probably) make Fable prohibitively expensive. I use some other stuff for various non-daily tasks, but these are really my main tools.

The AI tooling landscape is just breathtaking. The speed with which things have appeared, disappeared, merged with other things, spawned whole new cottage industries, etc., is astounding. If everyone actually made use of only those tools specific to their role and using AI within it, we would run out of space on our hard drives for tools.

I use relatively few tools, and I think this is due to having used AI for so long and having built up a reasonably mature set of markdown files (hooks, rules, skills, etc) that dictate how AI tools will behave. I do try hard to keep up with the evolution of tooling, and I do adopt new tools for various one-off projects, PoCs, and small tasks, only to find myself thinking that they’re kind of redundant, don’t work very well, use up too many tokens, etc. I think I’ve gotten the most benefit from:

• Iterating on my local configurations (skills, rules, AGENTS.md, etc)
• Keeping up with the Claude Code and Codex changelogs
• Experimenting with open weight models in LM Studio
• Talking to friends in other industries, companies, with other side projects, etc about their own use of AI

As I’m writing, I guess I’m also concluding that an awful lot of the AI tooling space consists of noise from fledgling efforts to ride the wave, sell you something, and achieve an exit. For open source tooling, some genuinely want to solve a problem and sit on the same side of the table as their users, but others quickly evolved into commercial offerings that were then bought out by those whose motives are less clear and whose track records make them less trustworthy to me.

What I’m Missing

Two really critical things:

1. A really good way to manage many agents across many separate, independent projects that each require a diverse set of tasks (setting up a database, validating a schema, creating a new github repo, writing code across multiple repos, implementing terraform for new infrastructure, writing documentation, etc). I have a visual model for this in mind that I tried to implement with AI, but it didn’t go great. That was a year ago, though. It might be a no-brainer with today’s models, which is crazy to think about.
2. Infrastructure in particular is a challenge, because the model for infrastructure changes doesn’t naturally align with how code changes happen. Code can be implemented locally, and then have a suite of thousands of tests that can locally confirm that not only does your new code do what you expect, but it doesn’t cause any existing code to break. And those tests can run locally too. And typically if those tests all pass, surprises after deploying to prod should be exceedingly rare. And, if there is an issue, you can revert to the last known-good version of the code in a flash. The whole coding ecosystem is built around making this easy. Almost none of this is true for infrastructure, so I find that infrastructure tasks require more bot-sitting, I trust everything a lot less, I have to review every little thing, it’s very difficult to account for everything up front in the prompt, or even in my precious markdown rules/skills, etc.

I’m doing some work on both of these things, but they aren’t really my primary job, so time on them is limited. But even though there is tooling that seems to want to solve these problems, I just don’t find them to be very… desirable, for one reason or another.

For #1 what I really want is something that kinda looks like Linear but when I move a ticket from ‘Planned’ to ‘In Progress’, that triggers an agent to pick up the task and execute on it, and it moves the ticket to ‘ready for review’ when it’s done, so I can review it (for now – later I’d have the option of delegating review to a bot that could be authorized to go all the way through merging and deployment, potentially).

For #2, I shouldn’t really need something as heavy as Atlantis or SpaceLift, but at the same time, the current features of GitHub are an awkward fit — it’s just always obvious that GitHub was built for code and not infrastructure, so from that perspective I understand why these large, heavyweight tools exist. Terraform’s own quirks don’t help, either, though it’s very obviously light years beyond where Chef & Puppet left off, but now they’re putting some useful stuff in their cloud-only product, etc. OpenTofu is definitely in my future.

The Future

The rest of my year is probably going to be spent:

• Building AI infrastructure in AWS
• Building infrastructure that’s not AI-specific but is used for projects in the AI space
• Preparing for scale, mostly focused on the infrastructure space
• Preparing for frontier model pricing to become kind of insane (so, experimenting with replicating existing functionality using non-frontier models, which I’m already doing, it’s just lower on the priority list for the moment)

If anyone wants to connect on any of this let me know. If you just want me to write more about any of this, let me know. If you just think I should stop sleeping on your favorite tool, let me know that too! It’s certainly more than a small possibility that I don’t know about every tool and I’m happy to hear about more!

Yesterday morning, I priced out a solution for doing backups of the repositories in our GitHub organization. It was the classic cable TV problem: It was expensive, and the only way to get the features I wanted was to upgrade to a plan that would have me paying for all kinds of stuff I don’t need just to get the one thing I do. Yesterday afternoon, I started testing a homegrown solution that Claude helped me build, complete with a docker image, python code, terraform code to deploy it all, a README suitable for anyone either using or managing the service, etc.

It actually is already a new world. It’s not just coming, it’s here, and companies need to start thinking hard not only about what their customers want and need, but what they’re competing with. I’m not sure it’s possible to understand what you’re competing with if you don’t have an internal capability around AI. Just six months ago it might’ve still been true that the bar for creating “good enough” solutions internally was too high for a lot of customers. The list of products for which that’s still true is shrinking quite rapidly!

We do the same thing in the technology space. In the 80s, there were folks talking about computers taking all of the jobs. In the 90s, there were folks talking about the internet taking all of the jobs. In the 2010s people talked about AWS taking all of the jobs. Today we’re talking about AI taking all of the jobs. Still the world turns.

Whether kids brains were ever turned to mush is, perhaps, debatable. But regarding various technologies taking all of the jobs, we have data on that, and it’s provably false. Well, at least it’s false from a raw employment perspective. It’s absolutely true that certain roles ceased to exist, but the people in those roles migrated to other, possibly brand new roles.

That’s not to say there weren’t other impacts on peoples’ lives & careers, though: for those who saw computers and decided not to learn about them and how they could be adopted in their work, they eventually became seen as complacent, or in the way, or behind the times, etc., and those folks (and I’m speaking from my own observations of family members way back in the day) had trouble finding jobs, eventually, and when they did find work, it was with a company that was still doing everything manually, because (for example) the owner found it cheaper to hire people than migrate to computers because computers made filing clerks in very low demand. Sounds like a great place.

Something similar happened when AWS adoption was building steam back in 2008-2010. There were only a few services back then. The concept of a VPC was brand new, for example, as I recall. I thought it was great and was getting real work done with these new tools, where I could write code to deploy an EC2 instance, and store files in S3, etc. At that time I was a member of a lot of system administration groups and Linux user groups, and there was a meeting where someone took an informal poll that showed that less than half of the folks in the room wrote code on a regular basis.

That really shook me. I was friends with almost everyone in the room. I also could plainly see that AWS was getting bigger faster – not slowing down, or plateauing. Within the next meeting or two I put together a talk that more or less begged everyone in attendance to learn to code in whatever language fit their brain. To take their shell scripts and port them to Python, or Perl, or Ruby, or anything. Any language would do. “The reality is”, I said, “that Amazon is creating APIs to allow developers to do your job.” A couple of smirks, a couple of knowing nods of agreements – from coders. “It’s not that there won’t be any jobs” I continued, “It’s that the jobs that are left are going to be the ones we all hate doing now, like changing printer toner.”

It has been nearly 20 years since I gave that talk. LinkedIn is a thing. There were probably 30 people in attendance at that talk, all locals, and the people who smirked are verifiably either not working in technology at all, or are working in tiny 1-man shops that require a physical presence and require them to change printer toner.

So, I tell you all of that so I can tell you this: I don’t think AI is anything to panic about. There are realities, though, that history tells us are likely coming, though. What history also tells us is that the folks who wind up landing on their feet are those who learn to adopt the new technology. The ones who wind up having a difficult time staying afloat are those who are complacent in their roles, think that AI could never replace them, or who try to foster anti-AI cultures, or the like. And, like computers, and the internet, and AWS, it’s not that there won’t be any jobs. It’s just that you might not want the jobs left in the wake of AI.

Not only was this tool instrumental in getting the data migrated, but it also allowed me to very easily craft a way to quickly compare numbers between the source and destination to make sure everything I expected to be migrated was actually migrated.

The Mess I Made

It’s important to know that engineers with 25+ years of experience do dumb things sometimes. It’s also important to know that having decades of experience does not make you immune to fatigue, and really nobody should be working from 7AM until midnight. It’s not heroic. It’s unhealthy and problematic on a bunch of different levels. With that in mind, here’s the dopy stuff I did after working too late after too many hours:

Initially, after I migrated a subset of the data, I did a more manual, hacky check for consistency just using command line tools, manually querying each source in different terminal windows & eyeballing the output. I wanted a nicer way to view that data, so I created a one-liner using awk, paste, and column commands. That looked like this:

paste count_clicks.txt count_clicks_parquet.txt | column -t | awk '
NR==1{
  printf(
    "%12s %18s %18s %12s %12s \n", 
    "date", 
    "clickhouse_count", 
    "warehouse_count", 
    "diff", 
    "pct-diff"
  )
}
{
  printf(
    "%12s %18d %18d %12d %12.4f%%\n", 
    $1, 
    $2, 
    $4, 
    $4-$2, 
    (100-($2/$4)*100)
  )
}'

A quick overview of what’s happening there:

• NR==1 means the first record (NR==2 would mean the second record, etc). If the record number is one, awk will output what’s in the first set of curly braces. The printf function takes a format spec in the first argument. My format spec lays out 5 columns of either 12 or 18 characters. All of those columns will hold strings, hence the ‘s’ in %12s. Then I have a bunch of hard-coded strings, which become the column headers. If you forget to put NR==1 in there, they’ll print on every row. Ask me how I know!
• The second set of brackets specs out what will be printed in the rest of the rows. In this case, I have column widths that match up with those of the column headers, and then I have the columns in the output of the earlier parts of the command pipeline:
  • $1 is the date column
  • $2 is the count from the S3 data source
  • $4 is the count from the ClickHouse data source
  • $4-$2 shows the difference between the two data sources, and
  • The last column shows the percent difference between the two sources

The output looks something like this:

        date   clickhouse_count    warehouse_count         diff     pct-diff
  2023-12-01               1471               2445           74        0.017%
  2023-12-02               1665               1700           35        0.038%
  2023-12-03               4496               4537           41        0.045%
  2023-12-04               1650               1705           55        0.047%
  2023-12-05               1154               1237           83        0.069%
  2023-12-06               2777               2865           88        0.074%
  2023-12-07               9244               9293           49        0.041%

The data here is made up to give an idea of what the output looks like.

So, I had this issue where my data audit showed a mismatch. I did a little work, very late at night, and went to bed thinking I had straightened it all out. When I woke up the next day, my well-rested brain and eyes caught a problem: I copied output from my queries of the two sources into two separate files, and mis-labeled the data, and then compared data in a completely different window with that, and…. well, it was a mess, and I didn’t fix anything.

I was up too late working for sure, but I also had a messy process. I should’ve and could’ve done better.

Fresh Eyes, Fresh (and better) Ideas

Revisiting my work from the night before was painful. As soon as I looked at my process at a high level (by scrolling through my terminal window history) I almost immediately said “this is insane. ClickHouse Local should be able to query both sources. I shouldn’t need to copy/paste and introduce levels of indirection that leave room for errors like this.”

I was right. Using ClickHouse Local, you can query a ClickHouse Cloud instance using the remoteSecure function, and query the S3 data using the s3 function, which also lets me pass in the file format as an argument. So there’s support for my data sources and formats.

On top of that, ClickHouse supports Common Table Expressions (CTEs), so I can craft a query where I name the output from two separate sub select statements (one to each data source), and then write a third SELECT that references the two named result sets as if they were tables.

Below, s3_count and ch_count are named result sets. The last SELECT queries those two named result sets.

clickhouse local --query "
WITH s3_count AS (
  SELECT 
    toDate(time) AS day, 
    count() AS num_events 
  FROM s3('https://s3-endpoint/2023/12/**/*.parquet', 'Parquet')  
  GROUP BY day 
  ORDER BY day
), 
ch_count AS (
  SELECT 
    toDate(time) AS day, 
    count() AS num_events 
  FROM remoteSecure('clickhouse-instance-hostname:9440', 'db.tablename', 'clickhouse-user', 'clickhouse-password') 
  WHERE toYYYYMM(timestamp) = '202312' 
  GROUP BY day ORDER BY day
) 
SELECT 
  s3.day AS date, 
  s3.num_events AS s3_count,  
  ch.num_events AS ch_count, 
  ch_count - s3_count AS diff  
FROM s3_count AS s3 
LEFT JOIN ch_count AS ch 
ON s3.day = ch.day;"

The output has a couple of quirks: for some reason, I guess possibly related to querying multiple sources, or using CTEs maybe, the output columns are not ordered according to my query, and there is no header line to tell me which column is which. It looks like this:

2023-02-01	40	2778	2738
2023-02-02	43	4413	4370
2023-02-03	26	7024	6998
2023-02-04	54	3079	3025

Conclusion

So, this is not something I’d paste as-is into a slide deck and present to an executive team. However, clickhouse local in this case did give me a (relatively) quick way to verify the consistency (and quantify the inconsistency) between my data sources. Once the data was moved, I wasted probably an hour with awk and friends, but was able to recover the next morning and throw together the clickhouse local solution in maybe just another hour of reading docs, debugging the query, and running test queries. Hope this helps.