In short, many people fixate on the limitations of new technologies instead of their long-term potential. It's happened over and over again.

The Commodore 64 had just 64 kilobytes of memory. Programs could take several minutes to start up. A complete personal computer in the 80s could set you back $4,000 of today’s money. Today's $600 MacBook Neo has none of these problems.

The early Web ran on dial-up connections. Designers were stuck with a 256 color "Web-safe" palette and hacked together page layouts using table and font tags. Today's Web runs applications, TV shows, games, and more.

The first iPhone only connected to 2G networks, had no copy and paste, only had 16 built-in apps and no way to add more. I won't belabor the point but you'd think with all these examples behind us, we'd know early technology limitations don't last. And the fixation continues with AI:

• image models can't draw hands.
• language models hallucinate.
• coding models can't write good code.
• and so on...

These limitations have largely disappeared in short order. And the folks that focused on what AI models couldn't do are now grappling with the larger impact of what they can do. Designers, writers, and developers all of a sudden face capabilities they actively bet against, leaving them vulnerable to job disruptions and more.

If the last forty years are any guide... technology limits are temporary. Plan accordingly.

Before any of my articles, presentations, or files can be used to answer people's questions, they need to be cleaned up and turned into a format that makes semantic search possible. This happens through a content ingestion pipeline, which I had to kick off manually for every piece of new content.

Now the system automatically checks for anything new or changed each hour. Audio, video, PDFs, and images it finds get queued, processed in the background, and added to the corpus. How that happens depends on the types of content.

Audio and video files get transcribed into text spelling out what was said and who said it. The service doing this recognizes my voice so it can consistently label me in each transcript. An AI model then reads the transcript and generates a title, summary, list of topics, and date if it can find one in the text. Each transcript gets split into distinct passages so an answer can reference the exact moment in a talk where I said something, and you can jump straight to it in the embeddable video and audio players.

PDF files get read by a vision model that not only extracts all the text but also describes charts and diagrams in words. Since many of my presentation slides are visual, this process matters a lot. Each PDF page is individually saved so answers can show slides inline resulting in less walls of text.

Of course, images help with that as well, so an AI model analyzes each new image added, describes what's in it, and uses my captions as additional context if the image appears in a blog post.

That's all well and good, but I'm not writing that many articles and definitely not giving enough talks to keep content recent and relevant. Thankfully, there's Twitter (X now) and I've posted there nearly 20,000 times since 2008. Damn, I need hobbies.

Despite the volume, none of those tweets were available to Ask LukeW. Now when I post, X notifies our server which pulls in the full post or deletes it if I had a half-baked idea that I removed. Older posts were added through a one-time import though I skipped replies to other people, reposts, and anything linking to lukew.com (that content was already there).

For each post, shortened links are expanded so the text reads naturally, and quoted posts are included for context. Any photo, video, or GIF also gets described by an AI model.

So why does all this matter? Making content ingestion automatic, especially Twitter/X posts as they happen, allowed me to start the Ask LukeW experience with an answer vs. requiring a question. Which addresses one of the most challenging bits of AI products: knowing what they can do. Plus Ask LukeW answers are now better because they have more content to draw from. So that's cool.

How Jeff described our brains and how today’s AI models are built seem to have a lot of parallels. Specifically, a common set of building blocks for intelligence. How so?

Hawkins estimates the part of the brain that handles seeing, hearing, language, and thinking contains about 150,000 cortical columns. Each of these columns effectively runs the same program on nearly identical biology. But they differ in what they're connected to (eyes, ears, fingertips). Each column learns things by combining what it senses with where it senses it.

For example, you learn the shape of a coffee cup in the dark when you touch the rim here and the handle there. Hawkins claims the same process works for everything we learn, even math. In addition, cortical columns work together by sharing their guesses until they agree on what’s there. That’s why you recognize the cup faster by grabbing it with your whole hand.

The Transformers architecture behind many of today’s AI models is also built from a repeated block. It handles text, images, audio, and video once they’re broken into tokens. Each token carries a “what” (the word “cup”) and a “where” (third token in the sentence). An attention mechanism lets each token look at the others and work out how the pieces fit. It's kind of like the brain's voting between columns. Many tokens share information to settle on one interpretation.

During training, language models improve by predicting what comes next and adjusting after wrong guesses. This seems to mirror how Hawkins says the brain learns from surprise. But while Transformers learn most of what they know during training, cortical columns keep rewiring themselves as they experience the world. AI researchers are working on models that keep learning after deployment too.

So in both our brains and AI, intelligence may come from one building block, repeated many times and shaped by experience. Of course, that's at a high level and lots of details differ. But the similarities do make me think about intelligence of both types differently.

Do we eventually reach a final form? Does everything just trend toward average as mediocrity seeps in? Is it driven by what's possible or cost-effective in the supply chain? Or does everybody just copy everybody else? Here's the reasons I see people argue most often.

Once a product's purpose becomes clear, design can get out of the way. A TV is for watching a picture, so the ideal TV is mostly picture. Same for smartphones, cash registers, etc.

The more mature a technology, the more it competes on price and specs. Standing out is risky and expensive so less companies do it. The ones that do tend to be smaller so they have less resources and reach, making it hard to go beyond niche markets.

Nature does it too. Crustaceans keep independently evolving into crab-like shapes: a pattern we call carcinisation. Someone suggested the consumer product version should be called "crabitalism".

Aerodynamics, thermals, and materials push cars and planes toward the same shapes. Constraints do the same. Planes are designed around airports, and cars around safety standards. Change the job (an F1 car, a fighter jet) and the design changes with it.

Software is even more susceptible. Every app is already being chat-ified (video below) even though we're only a few years out from ChatGPT's release. Hardware convergence often took decades.

On that note, AI hardware is currently a bunch of experiments: pins, glasses, earbuds, and even pendants. We'll see which of them (if any) make it into the right-hand column, but for now it's a design playground.

When most people think about staying on top of agent activity, they jump to management views: an inbox like those found in Slack and email; a dashboard like those found in monitoring apps. Effectively these answer the question "what are my agents doing right now?"

I took a look at inboxes, Kanban boards, task lists, dashboards, and calendars in a post last year about agent management interface patterns. But while these kinds of agent views currently get all the attention, more specific and dynamic views may actually be more useful.

That's why in Intent, our large scale agent coordination tool for developers, we've built in multiple ways to see what's going on. Intent lets you create any number of documents (we call them notes) when working on a task, so you can have the specific views you need when you need them. A few examples:

Agents can draw architecture maps, flowcharts, sequence diagrams, and state machines inside notes and chats so you know what they are planning to do, and how, before spending the tokens.

Stepped walk-throughs move you through a diagram one stage at a time, each with a short explanation and the relevant parts highlighted. This not only makes plans easier to follow but also helps you quickly understand the current state of things. Boxes that reference files or notes open them when you click so can go deeper if needed.

Agents can take before and after screenshots of their UI changes and place them side by side in a note letting you judge results without reading any code. These notes can update with each change or stick around as a historical record. Agents in Intent can also pin a screenshot to show progress at a glance.

A app or website's design system can also be viewed as a note: colors, type styles, spacing, corner rounding, animation timings, and components with their variants, all in one place. Keeping a view like this open while you work makes "which padding?" one glance away. Agents read the same notes, so their UI work lines up with your references.

Of course, lots of views aren't much help if you can't set them up in a way that's useful. Every panel in Intent can hold a stack of agents, notes, files, terminals, and browser tabs. Drag things between panels, drop one on an edge to split off a new panel, and zoom any panel to full screen if you need focus.

Yes, chat still has a place for kicking off, moving along, and redirecting work. But as agents keep multiplying, a configurable workspace of useful views goes a long way toward steerings thing toward the outcomes you want.

The most common way to address this was to throw more resources at the problem: design system teams and/or centralized design reviews. But now, AI agents give design systems a different path forward. When agents write most of the code, the design system can become part of what I've called the AI steering layer: the context that keeps every website update aligned with brand, design, and development guidelines. Put simply, a design system for agents is a steering layer for a site's design and front-end code.

With AI agents, anyone on a team can update a website. Marketing adds landing pages, engineering adds docs, a PM tweaks the pricing page. Each of their agents will happily implement its own tone and voice, colors, spacing, etc. unless something steers them toward a unified whole.

That's the case for collaborative steering. A design lead and front-end lead define the grid, fonts, colors, spacing, and components once, and everyone's agents are "snapped to" it. The team keeps shipping and the site keeps its integrity. So what does that look like in practice? Here are a few examples from our recent projects.

On the Intent website, an AGENTS.md file tells AI agents to reuse the theme classes and CSS variables in the site's global stylesheet instead of duplicating color, typography, or grid values. That stylesheet defines colors, fonts, type sizes, the responsive grid, and light and dark themes as design tokens (named values the whole site shares). A design system page shows it all as live examples agents can inspect.

When someone asks an agent to add a new button, it reuses the existing Button component and its styling, which also adapts to dark mode automatically. And no human has to look anything up.

Intent's design system started from Figma specs that were automatically translated into code using a Figma MCP connection. After that, the code itself became the source of truth and was updated with a new body font, responsive layouts, dark mode, and more as the site evolved. We've even had agents write changes back to the original Figma files. But since the code is the source of truth, that hasn't been necessary in practice.

Sol and Aria's websites follow the same approach (links go to their design system pages), each with design tokens, agent instructions, and a design system page that stays current because it's built from the same code as the site.

Design teams have spent years trying to get people to actually read their design system documentation. Agents read it every single time.

Loads of AI use today is building a lot more of the tools we're used to, but much faster. With today's capabilities, though, we can actually skip the tools and jump straight to the outcomes they enable. A tool was always just a means to an end. If AI can get you to the end directly, why stop at the tool?

As usual, a concrete example helps. Two years ago, we built an AI-powered newsroom called Exposit. The system would find, aggregate, and report on global news. Behind the scenes AI agents would do the curating, writing, and editing that you'd find in a physical news room.

So it's not surprising that we presented the end result like a news site: headlines, articles, categories, etc. People could search, scroll, or navigate to find the news they were interested in. In other words, we built a news tool. Like all the other news sites out there, just run by AI.

As we continued to iterate on the product, we added a feature that allowed people to ask about a specific topic or news story and we'd compile a personalized report for them based on the latest news, related events, people, locations, and more. Really quickly this feature became the dominant way people used the site.

I've come to refer to this approach as just-in-time content: generated in real-time, for a specific person, with a specific need, at a specific moment. Instead of writing something once and hoping it fits everyone who comes along, you build a corpus that can be recombined endlessly and produce a timely, relevant answer on demand.

From the personalized report on Exposit, people could go deeper into articles, topics, entities, sources, and more. In other words, they started from the outcome: here's the news that you asked for. And if they wanted to, later engaged with the tool(s). Outcome first, tool second.

I do a similar thing on the Ask LukeW feature of this Website. Originally people had to ask a question before they got anything. Now each day I grab my most recent tweets, articles, and files and compile a "what's Luke thinking about now" answer automatically, so people get something to read without needing to ask anything. It's a small change, but it aligns with that larger theme: skip the tools and make the outcome. In this case, starting with an answer instead of requiring a question.

Now I ask myself (and pester others with): are you building another tool, or are you delivering the outcome the tool was supposed to produce? I've found just asking that question inspires new ways of solving problems.

Ask LukeW provides answers to digital product design and strategy questions using my corpus of thousands of articles, hundreds of presentations, and (more recently) thousands of images. To make the images I've created for my articles and talks searchable, an ingestion pipeline watches for new image uploads. When an image is added to my site, an AI model examines its contents and produces a title and description for the image. The title and description are both saved, and each is turned into an embedding so the image is also searchable semantically.

When someone asks a question, a retrieval system not only searches for any relevant images semantically (using the embeddings) but also using more traditional keyword search. This brings back a ranked list of images that the AI answering someone's question can choose to include in its response. The better the retrieval, the more likely an answer can include relevant images that break up what would otherwise be a wall of text.

For instance here's a comparison of the same question without a relevant image and with one. Most prefer the answer plus visuals version.

But with this system, images were rarely included in replies despite there being plenty of good candidates. Why? Looking at the titles and descriptions generated by AI during the aforementioned ingestion process provides some answers. Consider this image from an article about off-canvas responsive images.

When ingested this image was given a title of "Green block layout comparison". I mean, that's technically correct but who is going to ask a question about green block layouts? So despite a reasonable title, this image would pretty much never show up in results. Thankfully, we can learn how it should show up by looking at the article in which the image appeared.

For the past 30 years, when I've added images to my articles, I always included an ALT tag: a very common accessibility best practice that gives screen readers and more a useful description of images in Web pages. The ALT tag for this particular graphic was "Why Off Canvas Layouts?". Same image, totally different description. While neither is perfect, both descriptions are useful for retrieval.

As I often say "AI begets more AI" so the answer (of course) was to use a fast, yet smart, AI model to combine any existing ALT tags for images with their previously generated titles. For the image above that became: "Green block layout comparison showing why off canvas layouts are used". Wordy, but much better.

And since we can be wordy, the model writing the new title for each image can now also make use of the full visual description if it wants to. Here's another image to illustrate that.

The ingestion pipeline titled this image "Image Prompt Enhance Feature". The ALT tag was "Reve enhance feature". But the new title became "Reve enhance feature showing a prompt editor expanding a brief Spider-Man prompt into a detailed version" by pulling a bit from the full description. Much better.

So what's the impact of all this? More answers with images of course. Sticking with our example above, here's how images now show up in What are off canvas layouts?

Of course, this system needs to be dynamic. If I upload an image, it gets titled from the picture alone. If that same image shows up in a later article, it gets retitled. Updating ALT tags in old posts does the same thing. Lastly, If I ever rename an image manually, the pipeline won't overwrite it. AI begets more AI, but it should still defer to us humans for the last word (for now).

While I don't know how OpenAI coordinated their agents, I do know a lot about the large scale agent coordination techniques in Intent. Intent is primarily for software developers and therefore aligned with their workflows, but how it enables agent orchestration can underpin a wide range of domains. In fact, developers have used Intent's underlying system for reducing their electricity bill, making restaurant reservations, and more.

But first, what's agent coordination? I'd say: aligning lots of instances of back and forth messaging with AI models that have been trained to use tools in order to make progress on a unified task or goal. Coordination helps agents:

• Not to step on each other's work
• Communicate among themselves: say when they're done, delegate tasks, review each other's output., etc.
• Interface with humans when needed: asking clarifying questions, request permissions, etc.
• Get through workflows in parallel, sequence, and asynchronously

So how does Intent enable all this for developers?

Focused context

Every task runs on its own copy of your files in a dedicated workspace. That isolation keeps agents from overwriting each other's changes. A living spec lives in each workspace and keeps the agents coordinated, recording what got decided and why along the way. The spec allows each agent picking up work to know what came before and what's next.

Agent roles

Human have different jobs (ideally based on what they're good at) and so should agents. Intent comes with a set of default agent roles: a coordinator breaks work into pieces and delegates them; implementer agents write the code; verifier agents check that code against acceptance criteria.

You can also add your own specialists. for example, if your team has conventions worth enforcing (a particular testing approach, a security review step), you can encode that as a reusable role and it shows up in the mix like any other agent.

Intelligent handoffs

When writing the spec for a task, a coordinator agent will outline how to get the work done: in what order, by whom, and how. As each agent makes progress, they can determine if need to wait for something else to happen and wake up only when needed. For instance, an agent can monitor a pull request in the background, answer review comments as they arrive, and push when everything's ready.

Handoffs can happen between agents as well. When an agent determines its work is done it can do a back-and-forth with a new agent to then move things forward. This allows new agents to only carry important information forward.

Many agents, many devices

With isolated workspaces, focused context, agent roles, and handoffs, you can scale. Not just many agents per workspace, but many parallel workspaces, and many workspaces on multiple devices. Yes, that's a lot of work happening at once.

Not that long ago, a single agent finishing a coding task felt like magic. Now we're orchestrating thousands of them across devices. As with many things in AI, developer workflows and tooling are the most mature examples of large scale agent coordination. But the underlying approaches (focused context, agent roles, and intelligent handoffs) apply to a lot more than just writing code.

To illustrate, the Ask LukeW feature on my Website (which answers digital product design questions using my writings and files) could follow the examples above and start with a seemingly simple interface: just a greeting and an open text field.

While it looks simple, it can be anything but. People regularly struggle with knowing what to ask. To address this, Ask LukeW includes an always changing set of suggested questions to give people both a place to start and a sense of the kinds of questions they can ask. In usability testing, we consistently found that even just reading suggested questions helped people understand the site's purpose and how they could use it.

But even with suggested questions, people still have to ask a question to get started. After updating the retrieval system powering Ask LukeW, I realized things could be different. The new system supported time-based questions like: what did you write about recently? After seeing the results for this question a few times, I decided to just turn it into an answer.

Each day, I grab my most recent tweets, articles, and files and compile a "what's Luke thinking about now" answer automatically. That answer shows up above the suggested questions and gives people something to read and/or respond to without needing to ask anything.

It's a small change but one that addresses one of the most common AI product issues: capability awareness. In other words, how do I know what this product can do? It also aligns with a larger theme I've been poking at: skip the tools and make the outcome. In this case, starting with an answer vs. requiring a question.

While an increasing number of software companies have adopted agentic features in their applications, as usual it is developers that are pushing things forward. The most forward leaning developers I work with today are supervising many agents on many projects at once. Hundreds and in some cases more agents coordinating to build, test, and ship software. Which, as you might expect, is a lot to manage. And developer applications born in the chat era of AI products aren't keeping up.

To account for this, we recently redesigned, rebuilt, and open-sourced our developer tool, Intent. Intent scales agent work by turning tasks into separate workspaces, each with an isolated copy of your files so agents don't step on each other's work. Each workspace runs on a living spec and a coordinator that breaks work down, delegates to specialized sub-agents, and keeps things moving without requiring babysitting. There's also a higher level agent to check-in and manage work across both multiple workspaces and multiple devices.

That's a lot of feature talk. But the point is between isolated workspaces, specialist agents, living specs, and more, the product is designed to provide observability (see what’s happening) and introspection (figure out why) over large amounts of agentic work.

While an increasing number of developers are now comfortable no longer manually reviewing AI agent code changes and just interacting with coordinator agents that manage other agents, they still need enough control to dive in, see what's happening, and course correct if needed. But as these systems mature and trust builds, we'll shift to another stage of AI product evolution.

Just like most computer users don't know or care about the code their applications run on their computer much less the deeper code their operating system runs to interact with a computer's hardware, they soon won't know or care how their agents use the applications on their computers. A new layer of abstraction will allow people to get work done in ways that don't require them to know about (and much less know how to use) the software we spend all our time interfacing with today.

In my last post, I highlighted how Apple's Knowledge Navigator video informed by John Sculley's conversations with Alan Kay (who was an Apple Fellow at the time) pointed the way toward human-centric computing. Increasingly feels like the next stage of AI products will feel a lot more like that vision as we inevitably add another layer of abstraction.

For most of its history, software has been deterministic. An application's behavior was thought through by developers and hard-coded in: how to check for errors, where to save data, and so on. This rigidity usually made its way to the user interface. Think Web forms with strict input requirements or a limited set of filters for search.

The end result was that people had to adapt their behavior to computers, not the other way around. We learned what little icons meant, remembered which menu item features were in, and tried to resolve (often cryptic) error messages when they came up. Not how most of us would want to spend our day.

Increasingly, though, we have technology that can flip this paradigm on its head. Fast and inexpensive AI models allow software to bend to humans instead. As always, examples help. Search and forms are a huge part of today's software, and both used to force people into computer speak. Today, semantic search replaces a rigid set of filters by turning your question into whatever query syntax is needed behind the scenes.

Multi-modal parsing does the same for input, taking images, documents, or web pages and turning them into the structured entries a database requires.

The common theme in both of these examples is the computer adapting to the human. Instead of making people speak and use computer, computer figures out what people want and gets it done. Add enough bits of intelligence into software and the process of using software becomes dramatically different.

The vision I keep coming back to (minus the bowtie) that illustrates just how different is Apple's 1987 Knowledge Navigator video. Put simply, it makes computing... natural.

There's no voice commands to memorize, it uses natural language. There's no query syntax to learn, it uses fuzzy search. There's no cursors, it uses direct manipulation through touch. All this together lets people act like people, not like computers. Which has always been the goal.

Not all questions are the same. Some are time bound: "What did you write about Web forms in 2010?" Some require specific a format: "Any videos where you talk about onboarding?" Some want a concrete number: "How many talks did you give in Seattle?" Some are just off topic: "Ever dance with a capybara?" Despite this, the way Ask LukeW previously found answers was always the same.

When someone asked a question before, the system ran through four searches at once: semantic and keyword across both titles and contents. Why so many searches at once? A relevant detail can sit deep inside an article whose title looks unrelated, and a strong title match can surface a whole relevant file instead of just passages. The results that come back get merged, deduplicated so we don't repeat the same content multiple times, and re-ranked before being used to answer someone's question.

This approach worked well enough to answer the 50,000+ questions people asked to date. But since it was architected three years ago, things have progressed. Very fast and capable AI models can now layer bits of intelligence into places they previously would have been to slow and expensive to use. Context windows for models have massively expanded in size and new models and ways to embed content open up better ways to do retrieval. So... it was time to rebuild.

We added a lightweight classifier that analyzes each question being asked and decides how best to search for results: a date range, a recency ordering, a comparison, a specific media type, a count, or a casual conversation. Because a question can be both temporal and media-specific at once (for examples) each classification can stack together. This allows us to better answer...

Temporal Questions

The old system was mostly blind to when I said something. Now it isn't. You can ask things anchored to a moment or a span of time. And it handles the harder version, where the question is about change over time.

• What are you thinking about most recently?
• What did you say about mobile in 2008?
• How have your views changed on AI agents?
• What's the biggest difference between how you thought about AI two years ago and now?

Retrieval Questions

Often people don't want an answer, they want a particular artifact. They half-remember it and describe it. The system now figures out you're after one specific thing and gets it.

• Show me a video of your "AI Ate My Website" talk.
• You wrote an article about Mad Libs forms?
• The podcast where you talked about working at Google?

Quantitative Questions

These are closer to database queries than "find me relevant text" fuzzy text searches. We now recognize that and answer with actual counts and lists instead of a vague summary.

• What are your 10 most recent articles?
• How many talks did you give in 2012?

Visual Questions

Some questions are best answered visually, and the response should be an image, not a paragraph describing one. We re-indexed every image posted on my site and can serve up to three per answer. We also cite specific PDF pages and drop in a slide when appropriate.

• What's the difference between AR and VR?
• Why top-aligned labels in forms?
• How to design a pull to refresh feature?

Casual Questions

Sometimes people just want to have fun and ask about random topics. These used to get rejected when no result came back with a message saying I hadn't written about it. Now it recognizes a casual question and answers in kind, often with one of the thousands of LukeW character images people have generated thrown in for personality.

• Ever fight a panda?
• What kind of trails do you like?
• Ever clone yourself?

The new retreival system also breaks content into much smaller chunks (yes that's the technical term) to better match specific sentences instead of losing them in a big blob (not a technical term). But small chunks lose their surrounding text and thereby important context, so we pull also in neighboring chunks with each result.

There's still a re-ranking step to make sure we're using only the most relevant results to generate answers. But we tuned things to use 75% re-ranked results and 25% of results from our original embedding search, just to make sure the re-ranker is not over-powering the final result set.

Last but not least we diversify the results so there's not mulitple near-identical chunks that might overpower an answer. But because we now pull in neighbors, a small important set (say the 2008 results) can get washed out by a bigger pile from another result. So we diversify within each result separately.

But Does It Work?

Yes that's a lot of steps but it adds up to better answers. How do we know? Evals of course, this is an AI application after all. But as usual, automatic evals are not enough. Ours was confident the new system was better. But just to be sure, I compared 100 old vs new results by voting on them manually and guess what? The old system won.

Turns out we had a bug, which got fixed. But I mention it because, automatic evals tell you the things moved in the right direction. Human evals find the specific failures you'd otherwise be blind to.

Big thanks to Lukas Seklir, Jerome Paulos, and Sam Breed for this round of changes. I've written about many of the others updates to Ask LukeW over the years, if you want to go deeper into any of it, have at it:

• New Ways into Web Content: rethinking how to design software with AI
• Integrated Audio Experiences & Memory: enabling specific content experiences
• Expanding Conversational User Interfaces: extending chat user interfaces
• Integrated Video Experiences: adding video experiences to conversational UI
• Integrated PDF Experiences: unique considerations when adding PDF experiences
• Dynamic Preview Cards: improving how generated answers are shared
• Text Generation Differences: testing the impact of AI new models
• PDF Parsing with Vision Models: using AI vision models to extract PDF contents
• Streaming Citations: citing relevant articles, videos, PDFs, etc. in real-time
• Streaming Inline Images: indexing & displaying relevant images in answers
• Custom Re-ranker: improving content retrieval to answer more questions
• Usability Study: testing a conversational AI interface with designers
• Generation Model Testing: a personal benchmark for AI model evaluation

Digital image editing has always been canvas-centric. In Photoshop (used by something like 90% of creative professionals) you're adding, removing, or changing pixels. The application doesn't really know what it's looking at because there's no semantic understanding of the subject matter baked in. It's just pixels and, therefore, lots of pixel manipulation tools.

When every object in an image is identified and addressable, you can edit by changing the subject itself. Move the cat here. Make it a sunny day. Make the cat's eyes green. Popular tools like Photoshop were built before these kinds of capabilities existed so they put pixel manipulation front and center instead of behind the scenes. Object-centric editing flips this.

Large Layout Models

Most image models use text as their internal representation: a language model expands your prompt into a long description, and a diffusion model renders that into pixels. Text can say a lot, but it says it loosely, and loose input means loose control. To see that in action, just tweak a prompt and watch the whole image change.

Reve's new model is trained to think in layouts instead of prose. A layout is a structured, hierarchical description of an image where every element has a location, a size, a description, and attributes like color. The model takes layouts, instructions, and images as input, works out a layout in its thinking, and then renders the pixels. Layout is to an image what HTML is to a webpage.

When every image is built from a layout, all the elements in it are addressable. A designer can change the color of a car without regenerating the whole image. An interior decorator can move chairs around a room. Shadows, reflections, and perspective stay coherent because the model understands how the pieces relate, not just which colors land where on a canvas.

Because a layout is a readable, structured format, it can also be it a shared interface between people and AI agents. A agent can reason about a layout the way it reasons about code. That enables new forms of human machine collaboration that canvas-centric approaches can't. And we'll see a lot of new creative tools as a result.

When building, and even more-so maintaining, websites and software applications, cohesion matters. A consistent brand enforced by intentional use of visuals, colors, fonts, etc. Coding standards, libraries and frameworks, a common development process, etc. Today's AI models can handle all that and more but how do teams make sure they handle it all the way they'd like? And do so repeatedly.

The answer boils down to context: prompting models with clear, comprehensive and relevant instructions. Better context, better results. So how can we encode context into our applications so not only every update to a Website or application stays aligned but every person on a team does too? In our recent projects we've relied on a steering layer.

So what's a steering layer? Put simply, it's how context gets enforced across a project, a team, and even, a company. Every Website or application at its core has a codebase and a set of tools for people to modify that codebase. That's how features ship, performance improves, and ideally software companies make money. A steering layer (conceptually) sits in between the tools and the codebase.

Examples, please. Let's start with a really simple one. My LukeW Character Maker is a small application that allows anyone to make on-brand (for LukeW) image assets. It consists of a tool that allows people to create, review, save, and share assets. A steering layer composed of guidelines, reference images, and prompt rewriting. And the code that runs everything.

The steering layer is responsible for enforcing an AI image model's output aligns with the LukeW character's style, colors, appropriate use guidelines, and more. The LukeW Character Maker tool allows anyone to easily stay on brand when making a LukeW brand asset. And the codebase makes it all happen.

At this point, it's worth noting that the steering layer often lives in the codebase. When it's just text files, reference images, etc. It makes perfect sense to leverage the version control, review process, and team collaboration native to code. But not everyone on a team is comfortable, nor should have to be, working as a developer. For the cases where designers, copywriters, PMs, and others should contribute to and maintain the steering layer, a UI for doing so separate from the codebase makes sense.

As another example, we recently launched a Website for our Agentic HR company, Sol. The steering layer for the website consists of a set of design tokens, development instructions in an agents.md file, and some agent skills. These live in the codebase with everything else. But anyone using AI agents to update the site (with a tool like Intent) is "snapped to" the design and development guidelines in the steering layer. This allows everyone on the team to make Website fixes and add new content without diverging from the design and development guidelines.

It's also worth noting that a steering layer isn't limited to a few text or image files. In the case of the Ask LukeW feature on this Website, the steering layer is composed of not only instructions and examples, but multiple retrieval systems I've iterated on for the past few years as well. Their job is to find the right context for any given question and dynamically apply them to steer results. There's also a whole set of admin tools for managing what's in the context they pull from but that's a whole other topic.

With each new project, we're looking at better ways to make the steering layer visible (and for some, editable) to teams. We're also building steering layer powered tools that enable more people in an organization to contribute to projects without sacrificing cohesion. So... more to come.