The old do-it-mostly-yourself approaches forced the idea of an “event loop” in to our code in a way that the newer async/await semantics hide. Deep under the covers there is a kqueue or an epoll which is doing a lot of work for you, and the abstractions are hiding a for loop that is iterating over file descriptors selecting whichever one is ready to continue execution.
The naming of the package asyncio in Python helps convey that it’s for I/O —
network calls, disk reads, waiting on things. That framing is mostly right, but
it invites a failure mode I keep seeing: someone discovers asyncio solves
their concurrency problems and but as time goes on they loose track of how it
works, and start CPU-heavy work inside coroutines. It does not end well. The
event loop stalls, tail latencies balloon, and suddenly your health check is
timing out in Kubernetes while your process is contentedly crunching data.
Once asyncio bites, the knee-jerk reaction I have seen is to rip out the
asyncio code and go back to the comfortable domain of multiprocessing or
equivalent. Hiding from the problem rather than addressing it.
We can have our cake while still using asyncio for CPU-heavy applications.
Andrew Lamb spoke about exactly this problem in the context of Tokio: the fix isn’t to avoid the async runtime for CPU work, it’s to stop mixing your latency-sensitive and compute-heavy work on the same executor.
The pattern that actually works
is a dedicated ProcessPoolExecutor for the heavy lifting, submitted via
loop.run_in_executor(), while your event loop stays unblocked and responsive.
Python still has a GIL so there may be some use-cases that still require multi-processing to achieve high levels of performance but a better segmentation of low-latency and compute heavy workload in the asyncio eventloop can go a long way.
Some tips to consider:
Don’t block the event loop — ever Async code should never spend a long
time without yielding. In Python, CPU-bound work in a coroutine starves the
event loop. Use await asyncio.sleep(0) periodically, or offload entirely.
Use a separate executor for CPU work I have seen errors where an applications’ health check got blocked by CPU heavy work, leading to seemingly interruptions of service as the container orchestrator killed unresponsive containers.
Run CPU tasks on a separate thread/process pool,
not the same one handling I/O. In Python: loop.run_in_executor(executor,
fn) with a dedicated ProcessPoolExecutor. Don’t share it with your I/O
event loop.
GIL means processes, not threads Unlike Tokio’s work-stealing thread
pool, Python threads don’t get true parallelism for CPU work due to the GIL.
Use concurrent.futures.ProcessPoolExecutor instead of ThreadPoolExecutor
for CPU-bound tasks.
Amortize overhead with chunked work For CPU-heavy workloads in Python consider making larger batches when possible,. A method call with a single row in a runtime like Python is going to have higher wasted overhead when invoking that method 100k times in a tight loop compared to invoking a method which is able to handle a batch of 100k rows outside of Python (e.g. in a Rust or C extension).
Cancellation and shutdown are hard Kind of niche advice, but
hard with custom schedulers are easy to get 99.9% right but corner cases
(shutdown, cancellation, draining) waste a bunch of time. Just use the
off-the-shelf schedulers. In Python, lean on asyncio.Task.cancel() and
executor.shutdown(wait=True) rather than rolling your own.
I love to advocate the use of
Rust for lots of projects, but modern Python with
judicious use of asyncio and a lot of the more modern APIs available since
Python 3.10-ish make it a much better option for high-performance applications
with a low barrier to entry.
Lakehouse//RT is powered by Reyden, a breakthrough new engine for real-time workloads that require immediate responsiveness at high concurrency.
One of the things I have been working on at Scribd has been building a low-latency data retrieval system for arbitrarily large datasets. While not as generally applicable as Reyden, the systems likely share a lot of the same ingredients. I submitted a talk which was not accepted to the event, which makes more sense in hindsight. From my abstract:
Using the same Delta Lake architecture we offer both direct data access for data scientists in Databricks Notebooks and online data retrieval in milliseconds for user-facing web services.
Keynote presentations for me are not typically something I pay attention to, but I knew that Reyden was going to be announced ahead of time and Reynold’s segment was the only one I really paid attention to.
Some points about Reyden which were mentioned are worth revisiting:
From their post on the subject:
Preview participants have seen up to 16x better performance vs. real-time serving layers, with response times as low as 10ms on smaller datasets and sub-100ms performance on larger ones. On standard analytical benchmarks, Lakehouse//RT delivers sub-100 millisecond latency at 12,000 queries per second.
Incredible.
I believe that I have an above-average understanding of how hard this actually was, which also makes me proud for Reynold and the team who built his Dream Engine; it is quite an accomplishment.
I wanted to share some wild speculation on what types of techniques could be at play to make Reyden so consistently fast.
There are a few different ingredients we know play a role in high-throughput low-latency systems like Reyden. Despite not knowing how Reyden was implemented, we can be sure they incorporate some of the same properties we see in other systems:
Intelligent query planning: Reynold mentioned as much in his keynote, but one property of every data system is coming up with increasingly intelligent ways to retrieve only the data that is needed. The stuff most people learn about creating indexes in traditional relational data systems gets tilted on its head when building intelligent query planning for large-scale data systems like Reyden. I have seen some systems describe “adaptive query planning” which is flavor of what I believe Reynold discussed: using machine-learning to build models for intelligently improving query plans over time and usage. There is great precedent for this type of work in interpreted languages which I would argue is related. Adaptive query planning reminds me of the work done by Oracle around Graal and Truffle which allowed interpreted code to get faster as runtime went on and the runtime could introspect and adaptively optimize based on real-world usage.
Caching: caching is not rocket surgery but the cost of cloud-based RAM was insane before “AI” and it has only gotten worse. Large data-systems must implement tiered caching solutions. Honeycomb has shared some of their work on Retriever which has some novel approaches for storing segments of query results in S3 for reduced retrieval time. Improving the speed of something like Reyden requires consideration of what data to cache but also where to cache it.
Object storage: The eighth wonder of the world, as Corey Quinn would like you to believe is S3. The magic sauce that makes some truly incredible data systems possible is commodity object storage with S3. Technologies like Delta Lake and Apache Parquet when wielded appropriately can lead to some pretty incredible low-latency reads.
InfluxDB, Lance, Databricks, Scribd, Turbopuffer, and a number of other companies have built and deployed compelling and fast data systems using some of the same underlying technologies that Reyden likely uses. But there are some real limitations that whet my appetite for more details on how exactly did Reyden overcome or hide these realities.
S3 is the foundational building block of almost anything data in AWS. You can build ridiculously fast data systems only using S3 if you know how to structure your writes. S3 is very fast but single object requests still live in the tens of milliseconds range. Personally I have not seen Standard storage requests below around 20ms.
Each S3 request has some IAM overhead and object routing overhead. This overhead you pay before you get the first byte of data back from the service. The roughly 20ms latency that I have seen is time-to-first-byte, if you need to transfer megabytes of data from S3 to a query/search service the time-to-last byte is the more important metric.
Once you get to the point of streaming bytes, it is possible to stream bytes extremely fast from S3. Multipart objects uploaded with an approximately 8MiB part size will be automatically fetched in parallel by AWS S3 client SDKs. This means that for a 64MiB file, your client can hit multiple backend services simultaneously and stream the parts of that object into memory at what is effectively line-speed of the network interface of the machine.
You can get blazingly fast reads, once you’re reading, but there’s always that 20-30ms latency to begin with.
This sets the floor of latency for a cold request to object storage.
To work around this you must build a multi-tier storage architecture where “interesting data” is being promoted to faster storage like S3 Express OneZone, Aurora/PostgreSQL, Valkey, or other in-memory stores. Meanwhile “uninteresting data” needs to be demoted downward to slower storage, otherwise cost blows up.
Databricks groups a couple different techniques under the umbrella of Predictive I/O which in its most simple incantation can be considered “predicate push-down to storage.” I have written about this approach before in Low latency Parquet reads and applied some of these techniques inspired by Andrew and Raphael’s work at Influxdata.
Applying a “predictive I/O” approach in an engine like Reyden is likely extending the “predictiveness” to address a multi-tier storage architecture. I would imagine that underneath the covers there is something along the lines of DuckLake which is denormalizes existing Delta Lake or Iceberg transaction logs into an online storage engine like PostgreSQL. Variants of the work that DuckDB has shared with DuckLake exist elsewhere in the ecosystem and improve performance for high-speed read scenarios with the trade-off of implementation complexity. Inside the walls of Databricks, something like DuckLake could probably be built on top of their Lakebase or Neon to offload that part of the storage/retrieval problem.
The largest most expensive workloads that I have seen on the Databricks platform end up being massive joins across datasets which must all be “considered” by the engine. If the user wants to compare nested JSON values joined between two datasets that are terabytes large, it is difficult to avoid loading or streaming terabytes of data and bringing compute online.
Optimize the I/O and query plan all you want, at some point somebody somewhere will have to do some hard work and actually compute something.
The unsung hero of Reyden is probably the Databricks Photon engine. If you are trying to build something super-fast, starting with a pretty fast engine that’s already in your toolbox is a good place to start.
In my work on a similar but less-generally-applicable system, I have been giving a lot of thought to Apache Datafusion and its ability to serialize logical plans over the wire via Protobuf. I’m not yet sure how I am going to abuse this capability, but I know that it’s a key ingredient in distributing query execution across compute resources in new and novel ways.
In a bespoke cloud environment Databricks has a myriad of ways to distribute compute workloads available to a system designed in 2025/2026. These could operate much more efficiently than Apache Spark operates today, with its driver/worker VM topology. Calling back again to the work that Honeycomb has done, I think they have really demonstrated that extremely bursty compute designs on AWS Lambda are both possible and cost-effective.
Even with a clever query plan, great predictive I/O, and bursty but powerful compute, there are still some laws of physics that have to be respected. Data locality and affinity to compute plays a big role in the speed of data processing engines.
In my own work I have been considering the totality of the stack, which I am sure Reyden does as well. The interaction between a client and server becomes important. What parts of the puzzle are computed on the client versus server? What data is sent back to the client and how?
Imagine you have a very fast engine for your users and they are primarily using it in your Notebook product.
You would be foolish not to *cheat a _little_.
The primary user interface for Reyden is a Databricks notebook. In a vertically integrated stack, it is possible to start preloading data or anticipating user behavior before they actually hit “Run.”
For a totally new notebook, I might start pre-filling caches or moving data into
warmer storage as soon as the user has typed a table name in, e.g. once the
editor has completed SELECT * FROM gold.schema.table .. I know with high
confidence that the user is going to query gold.schema.table and can start to
preload.
For an existing notebook that is being revisited, I can cheat slightly differently by preloading the necessary data for all the tables referenced by the cells. If I am feeling very fancy I might rely on a user-behavior-model to anticipate whether they’re going to “Run All”, “Run” one by one, or simply review prior computed results.
Databricks also has been heavily promoting their in-notebook AI Assistant to where I would also cheat a little bit by seeding that AI assistant with instructions on how to write queries that are more efficient on my very fast engine. The AI Assistant should produce code which is going to demonstrate the best possible query patterns for the given user intent.
Unlike general purpose database engines, Reyden has so much other options available to it to give the user the illusion of speed without needing to do as much hard or occasionally impossible work.
“Work smarter, not harder” as the saying goes.
Building a system like Reyden sounds like it was a tremendous amount of fun, with all kinds of unique and interesting constraints. Reyden would not be possible had Databricks not had a strong technology leadership to build from. In that respect both Reynold, Matei, and Ali deserve a tremendous amount of credit for their continued investment in Databricks’ technology platform.
Their investments compound in ways that are difficult to envision in detail “we need to do X, so that we can build Reyden.” The compounding benefits are more easy to imagine in the abstract “making this part of the stack faster/cheaper unlocks new capabilities to build from.”
Delta Lake was needed to move away from CSV, Parquet, Hive, and all manners of other junk holding Apache Spark back from beating Apache Hadoop. Photon was a no-brainer, a fast engine that beats Apache Spark for Databricks users is a foundational component that served an immediate customer need. Lakebase/Neon helps solve a data movement problem faced by every organization with a data lake.
The list of platform investments that are compounding in Reyden is probably much longer than I can imagine. I believe that Reyden could not have existed at Databricks in 2020, 2022, or 2024, but I do think the right pieces of were there in 2025 because of an astute platform-thinking that members of the technology organization have.
From my perspective, I recognize how challenging impressive Reyden is to build, and the long road that a lot of people have been on to make it a reality.
Well done y’all!
]]>My house is in the shockingly short boundary where city turns into wild. Troops of wild turkeys march through, the jack rabbits scurry away when you walk up the hill, and deer leap across the road heading from the creek on one side to the hills on the other.
I purchased this house from the woman who built it with her husband. Frank passed away about 20 years ago, but he still visits. His wife Marilyn moved to Sacramento and is an avid gardener. She cultivated a rich variety of flowers, bushes, and trees under the towering oaks of the home she built.
I let most of the non-native plants wither, encouraging and those which could survive the wet winters and dry summers. Continuing Marilyn’s garden would have taken too much time and water, neither of which are in abundance these days. Nonetheless I still feel a sense of responsibility to the land, the animals, and the oaks.
Returning from a week of travel, I was shocked to spot a fat doe lounging on the hillside. Deer are common but I have never seen a deer just lying on the ground. A couple days later walking through the backyard the doe, who was given the name “Butterflia” was just as surprised to see me as I was to see her. Later that evening I noticed her again further down the hill, a tiny fawn wobbling between her legs.
Last year a doe was struck and killed by a car on the road that runs alongside my house. The following day I discovered a fawn with a broken hind leg in the backyard. The nice woman from Fawn Rescue of Sonoma County and I cautiously crept through the woods with outstretched beach towels but could not catch the injured fawn and it escaped.
I called them back to see if they could help me relocate this doe and fawn. The same nice woman assured me that they would probably move on in a week or so, and to call back if their situation deteriorated.
Over the weekend I spotted a second fawn. Butterflia had twins! The backyard was off limits to ensure that neither Butterflia nor her fawns would be scared into the road. I am happy to share my space with such relatively benign neighbors.
By the end of the following week there was only one fawn that I regularly saw with Butterflia. During the day when I knew she was out I swept through the backyard to see if I could find a body, but never learned what happened with that fawn.
For whatever reason Butterflia had decided to call the backyard home.
A week later, on a Sunday while carefully walking through the backyard to get something from the shed I discovered the other fawn’s newly deceased body behind a rock near the house. The job of coroner would have to wait until Monday after work, Sunday’s plans did not include a dead fawn.
Butterflia discovered the body Monday morning around 4am. I was awaken by the mournful bellowing of a mother discovering the body of her child. It was absolutely gut-wrenching. I busied myself on the other side of the house, periodically checking back in the bedroom, only to hear Butterflia’s continued sorrow. She painfully wailed for almost three hours and spent the rest of the day near the rock. I would see her revisiting the body, poking her head under the leaves of the bushes as if in disbelief.
She was mourning and I found myself mourning as well.
I let the body rest for a couple days. I wanted to make sure Butterflia had left before I did anything, out of some sense of respect. Using an old spade I scooped up the fawn and gently placed it in the bin, covered it with dirt, and then some leaves and straw.
The week came and went without any additional sightings of Butterflia. Her sadness affected me more deeply than I anticipated. I don’t know if she mourned the first fawn, or even knew it was gone. I imagined her grief was compounded by losing both of her babies, the early morning anguish heavy with the knowledge that they both died.
She must have moved on, hopefully to greener pastures.
I swept through the yard this weekend to see if there were other remnants of deer habitation to be cleaned up. After completing my yard work, I startled Butterflia as she gingerly walked through the yard.
I said hello with a warm smile, pleased to see her, because I’m human.
She turned and looked at me dumbly, because she’s a deer.
]]>Chris’ talk below at North Bay Python was, as his always are, well-delivered and worth consideration.
The conclusion that he draws towards the end is similar to something I was noodling last year:
At some point somebody, somewhere, is going to have to actually understand how things work.
Chris makes the point, as he typically does, much more thoughtfully and with a stronger philosophical base.
Had some discussions with the delta-kernel-rs developers after they mistakenly added a ton of new files to tests/ blowing up test cycle times. Another community member shared this great overview about not using Cargo integration tests.
Catching up on Daniel’s thoughts on Data Quality and reconsidering the domain. The generation of slop has resulted in renewed discussions of “but how do we ensure correctness?” which is a great question to be trying to answer, but I am still rather disappointed with the state of the art for data quality tooling.
I recommend this blog post which has some good citations for negative AI behaviors affecting free and open source communities.
This is going to be a difficult problem to solve, more difficult than the email spam problem we have been unable to solve after 30 years of working on it.
This is also a very important problem, we are currently in an age where we have access to information that most people couldn’t even dream of 30 years ago. We also have disinformation that combines some of the worst aspects of authoritarian regimes throughout history combined with the worst aspects of cult brainwashing. If we lose access to the information but the disinformation remains (or get worse) then the result will be terrible.
I really enjoy Planet Debian as an aggregator of an international set of voices from the Debian community. I get exposed to so many different view points from around the free software ecosystem, which I really value. This past week I read this blog post by a debian maintainer which I was so flummoxed by I wrote out my thoughts on the topic here
Streaming tar over SSH is one of the more novel Unix tricks I don’t get to use
much anymore. Drew
Devault
shared some helpful tips for using it without needing to use incantations of
rsync(1).
The bad news I have for you is that today, in the year 2026 Unity Catalog does not work with S3 Access Points.
However it does show a different pathology than it once did, which leads me to believe that it could, if not for one silly little piece of technical debt.
The system I am building utilizes Amazon S3 Access Points for governance but must integrate into the Databricks platform. A platform which has its own ideas about governance: Unity Catalog. It should come as no surprise that a system which was named unity would go to great strides to make itself the center of the universe.
How troublesome!
Years ago a colleague and I tried to integrate Databricks Unity Catalog and S3 Access Points only for the approach to crash and burn. Integrating two different opaque tools like IAM permissions and Unity Catalog led to all sorts of attempted incantations, none of which actually succeeded.
The Databricks product team told us that the system did not support S3 Access Points “by design.” I found the reasoning very patronizing because it was presented as “we don’t support S3 Access Points by design to prevent users from circumventing Unity access controls.”
What I understand now is how that “by design” was more of an excuse “we just don’t want to test it” rather than something more substantive.
S3 Access Points can be referenced a number of ways like S3 Access Point Aliases to where even the most legacy system can integrate with them.
An access point alias name meets all the requirements of a valid Amazon S3 bucket name and consists of the following parts:
The first time we bounced off this problem S3 Access Point Aliases had been only recently released;
Despite all Unity Catalog’s protestations the errors we ended up seeing don’t convey a structural limitation when using S3 Access Point Aliases, instead they point to simply out-dated SDK support in the underlying Databricks Runtime.
My hunch is that the AWS SDK v1 being announced as deprecated over two years ago and being completely deprecated as of the end of 2025. Lots of Databricks and other Spark libraries still interact with S3 via the v1 SDK. That SDK was originally released in 2010 (lol) and so it’s likely that the issue we were authentication issue we were seeing was mixed up in the support for S3 Access Point Aliases with this old SDK.
Since we bounced off this problem a number of years ago one thing that has changed for the better in Unity Catalog is that it is now possible to grant Unity a completely read-only configuration in IAM-based S3 bucket policies. While we cannot use S3 Access Points as part of our governance strategy, we can at least still grant a fairly limited permission to Unity for read-only operations.
Now I can have my esoteric Delta Lake datastores present in Unity without any risk of misconfiguration or error in Unity leading to data corruption!
Governance to a lot of enterprise vendors is about centralization of control, but for me it’s about defense in depth. I never want a business-critical system to be a single misconfiguration away from granting read or write access to the wrong principal.
]]>I am a big believer in the four opens:
The Four Opens are a set of principles guidelines that were created by the OpenStack community as a way to guarantee that the users get all the benefits associated with open source software, including the ability to engage with the community and influence future evolution of the software.
- Open Source
- Open Design
- Open Development
- Open Community
There is an implied “to the public” in each of the four opens, at least how I have understood it over the past many (many) years. I have repeatedly advocated for open (to the public) discourse and transparency when working with companies like CloudBees and Databricks as they have engaged with open source projects.
The mounting negative pressures and in some cases outright hostility towards free and open source projects has me reconsidering the implied “to the public” and how these communities may need to evolve in the future.
While I have never been a fan of invite-only Discord or Slack servers, both of which are used by the Apache Datafusion project for some odd reason. There are good reasons to put the project’s shared spaces in slightly more private and slightly less AI-accessible systems. A little bit of privacy can lead to more candid conversations and potentially a stronger feeling of community and safety.
My first line of thinking led me to the idea of “vouching” which I recall mitchellh posting about in the fediverse, but I couldn’t find a good linkable reference.
Vouching is what we did as kids when a new friend was suggested to join the mischief, somebody would vouch for the new kid and say “hey, they’re my neighbor, they’re cool” and then we would go start new trouble together. In the context of an open source community vouching can:
I think vouching could also exacerbate the likelihood of a Jia Tan where the web of trust within the community is compromised by a malicious actor. Getting one member to vouch for you may lower the guard of all of the other members of the community making these style of attacks easier to pull off.
Since I started writing this post a whole week has passed by, without any new
ideas or patterns popping into mind. I’m curious how others are thinking about
it, so please let me know on Mastodon or via
email rtyler@~
systemd ended up so
contentious because it’s a decent piece of technology which suffers from
persistent scope creep that became a foundational component in a lot of
distributions. The drama du jour is that systemd is somehow implicated in “age
verification laws.”
systemd as an init system is pretty good. Once upon a time I worked on
porting launchd to
FreeBSD and so I have some familiarity with the
silliness of most init systems.
systemd as a katamari at the
root level of most Linux systems is not “pretty good.” There have been
numerous tendrils of what is understood to be “systemd” which are of lesser
quality and have resulted in security issues.
Anyways, I hope you get the point. systemd as an init system: good. systemd as a operating system: bad.
The drama du jour is about the latter.
One should not obey in advance. Especially in the domain free and open source software which is literally a political project.
I stumbled into this blog post through Planet Debian by a debian maintainer which is patently absurd.
Recently, the free software Nazi bar crowd styling themselves as “concerned citizens” has tried to start a moral panic by saying that systemd is implementing age verification checks or that somehow it will require providing personally identifiable information.
The author is correct insofar that systemd did not add age verification.
However most of the folks upset with the change are upset that their Linux
systems are obeying in advance.
systemd did make changes in order to obey. To take part in anti-free restrictions under the guise of “age verification” From the pull request
Stores the user’s birth date for age verification, as required by recent laws in California (AB-1043), Colorado (SB26-051), Brazil (Lei 15.211/2025), etc.
The whole motivation of the change was to obey in advance to these unjust laws.
The author then goes on to make some equally absurd claims about how this functionality is important for porents to implement controls on computers, for the children! Clearly this person must not know any actual children, or even parents for that matter. Children are excellent at finding ways to circumvent restrictions. The idea that a user-modifiable piece of data on a local machine should be trusted for “parental controls” is so detached from reality that I originally thought they were making a sarcastic joke.
I think this tongue-in-cheek systemd-censord post does better than anybody to exclaim how absolutely ludicrous this obeying in advance is:
Systemd units will be created for every desired censorship function, and will be started based on the user’s location. For example, a unit for Kazakhstan will implement the government-required backdoor, a unit for China will implement keyword scans and web access blocks (more on this later), a unit for Florida will ban all packages with “trans” in the name (201 packages in current stable distribution), a unit for Oklahoma will ensure all educational software is compliant with the Christian Holy Bible, a unit for the entire United States will prevent installation of any program capable of decoding DVD or BluRay media, and a unit for California will provide the user’s age to all applications and all web sites from which applications may be downloaded. As can be seen, multiple units may be started for a given location.
Do not obey in advance.
]]>At my height any seat
can feel like misery.
I wouldn’t be here today,
if not for last night’s delay.
35E, 35E
trapped in this humid
sneeze of
humanity.
Everything is expensive, and still it sucks.
The cheapest coffee was four lousy bucks.
The grumpiness was extreme at the TSA
acting in their theatre for deferred pay
35E, 35E
I’m not sure in which
timezone
I should be.
United customer support has been totally outsourced,
hour seventeen on the phone; just the worst.
For the shareholders the texts all lie,
about being powered by GenAI.
35E, 35E
all of the staff
on this flight and the last
have been really kind and patient which is a testament to their professionalism and hospitality despite the overtly customer-hostile environment of modern American commercial aviation.
Compressed Apache Arrow tables over HTTP
When discussing transport protocols for sending data between services at work recently, a colleague asked “why can’t we just yeet Arrow over HTTP?” It turns out, you absolutely can and Arrow IPC streams even have a registered MIME type:
Content-Type: application/vnd.apache.arrow.stream
Understanding Parquet format for beginners
A great introduction to the Apache Parquet format and why it makes so many things better with large data storage systems like Delta Lake. I have written on this topic before and encourage you to take another read through this blog post by some maintainers of the parquet crate.
Every layer of review makes you 10x slower
Every layer of approval makes a process 10x slower [..]
Just to be clear, we’re counting “wall clock time” here rather than effort. Almost all the extra time is spent sitting and waiting.
- Code a simple bug fix: 30 minutes
- Get it code reviewed by the peer next to you: 300 minutes → 5 hours → half a day
- Get a design doc approved by your architects team first: 50 hours → about a week
- Get it on some other team’s calendar to do all that (for example, if a customer requests a feature): 500 hours → 12 weeks → one fiscal quarter
This inspired these thoughts which I shared with the delta-rs community:
“what if we didn’t require code review for merging into main”
I’m exploring the thought more about what we might need to make that happen. “Why would you do such a thing, code review is so valuable!” I do find code reviews valuable but we do seem to lose a lot of flow time due to timezones, differing work schedules, and a number of other things. For something without a lot of changes, especially bug fixes that come with tests I would be much more comfortable with maintainers merging once CI goes green.
Some pieces of the puzzle that I think would be needed:
All of this is to say that reviews can still be requested, but I would love to see us land more improvements faster and I think we have a bunch of different schedules that can make pushing each change through a review queue a lot slower than necessary.
It’s possible in Rust to conditionally implement methods and traits based on the traits implemented by a type’s own type parameters. While this is used extensively in Rust’s standard library, it’s not necessarily obvious that this is possible.
I have been vaguely aware of this functionality but haven’t really taken the time to consider it, so I really appreciated this post walking through the conditional impl functionality in Rust.
]]>This video struck a chord for me. Contrasting the Bay Area hustle culture to the things that make life worth living:
San Francisco has always a destination for those seeking their fortunes. The frenetic enthusiasm radiates through seemingly everything there.
I also really enjoyed the energy of San Francisco when I first moved there. I had nothing else but work.
The trade-off for my relentless focus on my career was a tremendous level up in a short amount of time. I wouldn’t be where I am today without a few years of judicious networking and workaholism.
San Francisco was “Lord of the Flies” when I described it to friends from elsewhere. Awash in adult boys, untethered from the real world. I would hang out with men 10 years older then me who were doing the same dumb shit I was, except I was in my early twenties, a commonly accepted time to be foolish.
I did not want to end up like them and increasingly put both physical and mental distance between the them and myself.
There is more to life than panning for gold.
Today I was talking with an elder almost twice as old as me, who casually offered:
I still get up at 5am; at this age there are only so many sunrises left to see.
I’m going to try to not stay up too late dwelling on the comment, lest I miss tomorrow’s sunrise.
]]>