Marginally Interesting

That one time an agent was more helpful than I wished for

2026-10-20T22:14:58+02:00

Very often I see people starting a system prompt like "You're a helpful agent/bot that does XYZ", and I've done it, too. But once I was working on an agent to take natural language request and turn it into a SQL query in a very well defined domain. We had added canonical queries, background information, guides how to modify the query, how to add break downs, etc.

And YET, the agent always started with "ok, let me see what kind of tables are out there", and essentially making up stuff as it went along.

So I started tweaking the prompt, and eventually arrived at something like "your job is to translate the users request into sql in the most direct way possible following the guides you have. If you are stuck, don't try to solve it, report back to the user so they can fix it."

At that point the agent started to use the canonical queries as instructed, and guess what, it turned out that the query had a bug! I can only assume that the agent realized that and tried to work around it to be helpful! After I found and fixed the bug, I left the phrasing in because for this specific use case, that's what I wanted.

I learned from this that "helpful" is note just a default friendly persona, it already implies a lot of behavior, and depending on what you want, it might not be the right thing.

Unfolding LLMs all the way

2026-10-14T23:37:30+02:00

I recently had a realization about the structure of typical llm computations like the agent loop or the llm inference itself which I found interesting to better understand what's going on.

The basic version of an agent loop (the loop that produces an llm conversation) looks something like this, including tool calls:

# start with the system prompt in the conversation history
h = [p]
while True:
   # get user input
   u = userInput()

   # add to context, compute response
   h <- h :: u
   l = llm(h) # call to an OpenAI endpoint for example

   # deal with tool calls as long as there are some
   while l has tool call:
     r = execute tool call
     h <- h :: r
     l = llm(h)

It's pretty straightforward, but it's also a bit inelegant, because you update the history and call the llm in two different places. Sure, you could try to clean it up a bit, and rearrange, but this is closest to "you call an llm, and execute tool calls as long as it keeps coming back with more tool requests".

And it's a bit tricky. I had one subtle bug once where I forgot the second history update, and then wondered why the llm was calling the tool call over and over again! I was already blaming the stupid LLMs, when I didn't realize that to them it just looked like the result never came back! Totally "it's definitely a compiler bug territory" 😅

OK, but what's the structure of the loop?

we start with a history that's just the system prompt [p]
depending on the type of the last message we do one of the following...
- if it's a llm reply and it's a tool call, execute a tool call, and we get a tool result
- if it's a user message or a tool result, call the llm, get a reply
- else ask the user for the next input
... then we append the result to the history and start over.

This reminded me a lot of functional operations on data streams in languages like Scala, clojure, or Haskell!

A simpler example is the fold or reduce, where you apply a function f to a collection like [a b c] by first applying f(a, b) to get x, and then f(x, b) to get y, the result:

(fold f [a b c]) = (f(f a b) c)

What we have here is a close cousin, the unfold. While a fold summarizes a collection, the unfold generates a collection. The unfold takes a function f and a starting state. f takes a state and either returns an output and a new state, or None when it is done.

For our agent loop, this function looks something like this:

def next(h):
   if last(h) is user message or tool result:
      r = llm(h)
   elif last(h) is tool call:
      r = execute tool call(last(h))
   else:
      r = userInput()
   return (r, h :: r)

The whole agent loop becomes essentially unfold(next, [p]).

So this is at the core of any agent framework out there, no matter how complicated. They add things that aren't here like persistence, streaming responses, retries, interrupts, limits on number of turns, etc., but at the core, it's still an unfold of the next over an append only history!

Another interesting parallel is that the way how the llm interacts with the worlds is reminiscent of how that is modeled in Haskell with IO monads. In Haskell as a pure functional programming language, the language itself has no concept of IO, and IO operations are just descriptions of IO which are then executed by the Haskell runtime. Here, the situation is the same, as the llm cannot make a system call, but it can ask the runtime, the agent loop to do a tool call on its behalf.

That clarifies how each of the components behave when you think about testing and replays:

The llm itself is almost functionally pure, except for the sampling. If you take a more "probabilistic view" then the llm part can be replayed and will sample from the same distribution, which could still have forks, however.
You need to "undo" the result of tool calls and create exactly the same situation as before, then you can compare outcomes.
The human itself is also not "functionally" pure. But if you use an LLM generated conversation partner you can reset that in the same way.

Ok, let's... unfold... the next layer, llm generation

Turns out, the inference part of the llm computation is also this kind of unfolding a computation over an append only history. Very simplified, in particular leaving out the details of RoPE positional embeddings, llm inference looks like this:

llm(h) = unfold(inference, prefill(h))

# prefill computes k, v vectors for all heads / all layers for the context
# which are needed for the attention mechanisms later
def prefill(h):
  embed tokens for the whole history h
  # compute transformer for the whole history from bottom to top
  from bottom layer to top:
    # for each layer we can compute in parallel
    for all tokens in h:
       compute k, q, v vectors
       compute attention to previous positions
       compute mlp
  return [h,
          [k, v] vectors from all layers for all of the history]

# Do one step of inference, update state by appending the
# outputs and [k, v] vectors
def inference(c: [outputs, [k,v] vectors):
  embed last token
  from bottom layer to top:
    compute k, q, v vectors
    compute attention to previous positions
    compute mlp
  sample output o from last layer
  if o is <END>:
    return None
  else:
    return (o, c :: [o, [k, v] vectors ])

So in essence, prefill is like a layer by layer map operation that works in parallel over the whole prefix (and is cached so you only need to compute for the new part - that's the reason why prefill is faster than generation), and inference is an unfold that generates layer by layer.

What is the conversation history in the agent loop are the [k, v] vectors that are required for the attention computation, but structurally it's again an unfold!

If you look at agent libraries that are written in functional languages, you will actually see these patterns in production, for example https://docs.outskirtslabs.com/ol.llx/next/. I actually don't think we should now rewrite all libraries in functional style. Sometimes a for loop is clearer in its intent. But then you add a ton of other stuff and it becomes muddy. But essentially, it's an unfold over an append only history!

logexplain

2026-10-14T21:36:59+02:00

Github repo: https://github.com/mikiobraun/lx

Install: go install github.com/mikiobraun/lx@latest

What it is: Using an LLM to explain a log stream.

When I was recently playing around with ollama to test out some local models and the effect of context sizes, I got a bit tired of parsing the (relatively) detailed ollama logs to figure out if the changes have made it, or if I accidentally misspelled an environment variable again. Naturally I thought "maybe AI can solve this for me?"

My idea was something like: collect a bunch of log-lines, maybe maximum number, time since last run, send them to an LLM with a prompt. My reasoning was that either the LLM knows the format, or the prompt would contain an explanation of the prompt. A quick search showed the logwhisperer as a similar project but following my credo "the best way to learn something is to do it yourself" I started designing with Claude Code and created lx - logexplain.

Running that on ollama logs, restarting and then having llama3.2:1b parse a bit of text (I was testing on a raspberry pi 5!), you get this:

# You need to set OPENAI_BASE_URL, OPENAI_MODEL, and potentially OPENAI_API_KEY
$ journalctl -fu ollama -n 0 | lx --profile ollama
[22:19:41] Ollama service stopped then restarted; config logged, two GPU‑probe runners launched, CPU compute detected, low‑VRAM mode entered.
[22:19:50] Loaded Llama 3.2 1B Instruct (1.22 GiB), CPU‑only (0 offload, 1.3 GiB req), runner started in 1.78 s, 1 runner loaded
[22:19:57] /api/generate returned 200 in 10.8 seconds.

Great improvements! :)

You can also add a --focus option to look for specific types, for example "show only which [GIN] endpoints are being called" you get

[22:22:32] HEAD '/' and POST '/api/show' called, both returned 200.
[22:22:37] POST '/api/generate' returned 200 in 7.6 s.

The examples above were using gpt-oss:120b in the cloud. But this actually worked interestingly well with local models. Even a gpt-oss:20b produced good results which I could even run on my MacBook Pro M1 Max with 32GB. Adding the --debug flag also shows some timings:

lx: debug: summary of 13 lines: 6.57s total, 3.87s to first token, prompt 3424 tok @ 884 uncached tok/s, completion 162 tok @ 60 tok/s
[22:24:31] Restarted ollama service, loaded config, scanned blobs, no GPUs found – CPU only, low‑VRAM mode entered.
lx: debug: summary of 189 lines: 18.09s total, 13.05s to first token, prompt 10352 tok @ 793 uncached tok/s, completion 267 tok @ 53 tok/s
[22:24:58] POST /api/show caused Llama 3.2 1B Instruct to load into 1.3 GiB CPU memory, runner ready in 1.03 s.
lx: debug: summary of 1 lines: 5.75s total, 2.76s to first token, prompt 2450 tok @ 888 uncached tok/s, completion 186 tok @ 62 tok/s
[22:25:04] Generate completed 200 in 10.08 s using Llama 3.2 1B Instruct.

About 800 t/s prefill, and then 60 t/s is not too bad, although you start moving away from real-time considerably. But at least you don't pay per token!

Building and validating at the same time

2026-10-05T13:35:02+02:00

I've been coding with agentic support for a bit more than a year now on the side, and built a huge number of things, all developed with a raspberry pi 5 running in my living room as dev server.

Not all of these have been finished or released, but there are definitely some threads for me: Talk with Ren made me interested into richer personas, not just "helpful agents", leading to Bjorn the Bouncer, the idea for a persona rich agent roleplaying as a used camera shop owner. Another thread was creating tools to keep the loop fast: dev-router, a home use OAuth server, a tool to manage credentials for LLM endpoints. And yet another topic was agent-human collaboration, leading to explorations into multi-agent collaboration, and knowledge bases (I'll talk about that another time).

One thing I definitely fell for initially was "AI excitement". Especially a year ago, agents were very enthusiastic as long as if you were, and I just love developing ideas, so more than once I came out of a longer session in a haze of "OMG, this is IT! If this takes off, it's going to be amaaaazing!" But then, I launched Talk with Ren, 60 people signed up, most of which I knew personally, and then nobody used it more than 3 sessions. At first I thought "is this that sycophancy problem I keep reading about?" I tried tweaking the prompt, making the AI more challenging, etc. But over time I realized that the agent was not the problem. You can have exactly the same kind of echo chamber with a human. Be it a cofounder, who also just wants to see it happen, or a supportive friend. That's what the book "The Mom test" is all about. But then I wondered, is it a problem? Isn't it more like ideation is just one phase of many, and you still need to validate your ideas by asking yourself "what do we know, what did we just assume?"

So I thought "what's next? Focus groups? A/B tests?" or, maybe I should look for feedback readily available: myself. I came up with two rules:

Try to solve a problem that I have.
Do I come back to using it?

The first rule was important to ground myself again after flying in the sky with thoughts like "this could become a multi million dollar platform play". When I focussed on a concrete problem I'd often realize that I didn't need any of those platform features I had already envisioned. More than once did this help me to actually build the simple thing instead.

The second rule I originally didn't see, but over time I noticed that there were some things that I built that felt useful, but then I never used them again. Other tools immediately became part of my toolbox.

But then I realized (and thanks for sticking with me till here): what am I actually doing? I'm building tools mostly for myself, and I'm immediately using them. Because of agents, the building was minutes for what used to be hours or days. And since I'm the user, testing is immediate.

So I was building and validating at the same time... building and validating... buildidating?!

It always used to be that building and validating were two different stages. You used to get seed funding to get something in front of customers to get some data point whether it actually made sense. But now, you can build so quickly it doesn't even have to be a separate stage anymore. And if you can rapidly get feedback, the two stages can really collapse, and you're getting iteration speeds that don't match anything we're used to in the software industry.

Not always of course, but now I'm thinking what does it take to be buildidating more? And the real challenge: how to bring this way of working to the team level.

Google Search AI Mode: the best facts grounded chatbot nobody knows about

2026-05-20T10:27:36+02:00

At yesterday's Google IO conference, Google announced that they will overhaul the Search experience, making the difference between normal search and AI mode smaller, adding working agents that can scan for news, and "generative UI" essentially creating dynamic web sites for your replies, and people are already losing their minds.

I used to also think that "Google Search AI Mode" is one of those "let's just tack AI on all our products" kind of moves, till I realized that Google accidentally built the best web search integrated chatbot there is, and it looks like they finally realized what they have.

Google Search AI Mode is different from your usual LLMs with a WebSearch() tool, because it works more like supercharged RAG. The first step is to take your text and generate a number of potential search terms from it, then doing all those queries in parallel and then assembling context for the chatbot's answer. And because it is powered by the Google Search index, it is essentially the best facts grounded chatbot that I've come across. No more "I think this was after your training data cutoff, please do a websearch", no more "Let me search for this... interesting, let me search for one more thing... OK, now I have the full picture." This is the only chatbot I came across that you can ask about an event that happened just today without extra prompting.

The problem is, of course, that Google Search doesn't seem to be aware that they built a chatbot, but it feels more like they think they built a search with "generative result presentation." It doesn't have a shred of personality, and depending on what comes back in the result set, it might tell you a completely different story than in the turn before. And it has the worst "might I interest you in something else" engagement prompting I've ever seen!

But it's definitely a full fledged chatbot that can be used to gather data, discuss theories, and just have a chat. One time it even offered me to help me write a draft to which I thought "ok, interesting, let's see how that goes given that you don't have access to any tools or documents." Sometimes, some guardrails kick in, and Google search seems to wonder if you're still searching for stuff. And the best thing: as this comes out of Google Search, it's free, paid for by ad revenue from the classical result page.

When Google Search AI Mode was announced a year ago, they also announced a couple of coming integrations, like giving it access to your emails and personal context, or shopping, but I think that was never really scaled out. Once or twice I have seen actual product cards, but while it's happy to talk about any kind of products, the integration isn't really there. Maybe they're waiting for Google's Universal Commerce Protocol to roll out, who knows. Other companies like OpenAI have tried to push agentic shopping but rolled back in March after lack of adoption.

Google Search AI Mode (we really need a shorter term for this!) is now the third Google Product (after Google Workspace and the Gemini chatbot) that tries to integrate with your personal data, and I think this is an example of Conway's law, every org shipping its own product. And you can see the Google Search DNA in the way it is executed technically.

I actually don't think this is wrong, this is how big organizations work, and it also creates some competition and parallel exploration. What really seems to work for Google, however, is the investment in foundational AI technology. All these products are based on Gemini models, and other recent developments like TurboQuant, and their own TPU technology to run Gemini models at scale. As so often, product development and launching products is not their strongest suit, but in terms of technology, their strategy seems to work out. As recent as two years ago people were still saying that Google lost the AI race, but those days are behind us.

Now it seems Google is realizing what they have. With 1 billion monthly active users on AI mode, my original claim that "nobody knows about this" is apparently not true anymore. But I also hope that the good old ten blue links results page will not completely vanish!

Agentic memory the follow up

2026-04-20T09:22:20+02:00

Last week I wrote about agentic memory and I got a lot of responses, in particular many people pointing me to existing projects like mem0 or letta.com. So I started doing research, and as one does these days, discussed the findings with Claude. Immediately I thought I should write a follow up, and then I thought, why not try to let Claude write a part of that article? But also try to make it actually readable and not the typical AI slop, so here's Claude's summary:

Claude: Full disclosure upfront: I'm an AI writing about AI memory, which means I have an obvious conflict of interest. Make of that what you will.

So what is AI memory, actually?

If you've used ChatGPT or Claude recently, you've probably already interacted with AI memory without thinking much about it. ChatGPT now has a two-layer system: "saved memories" that persist until you delete them, plus a "reference chat history" feature that mines your past conversations for context. You can explicitly ask it to remember things. Claude (that's me — or rather, the product I'm part of) has its own memory system that works differently, generating summaries from past conversations and applying them automatically. Gemini is weaving memory into Google's ecosystem across Search, Workspace, and Android.

These platform-native implementations are what most people encounter, and they're a useful starting point because they make the core idea concrete: the AI retains something from past interactions and uses it to be more helpful next time. Simple enough in principle. But the product design choices already diverge — explicit vs. automatic memory, user-controlled vs. system-managed, transparent vs. opaque retrieval. Even at this most visible layer, there's no consensus on how memory should work.

Now, these consumer-facing features are just the surface. Behind them, there's an entire ecosystem of memory infrastructure being built. When I surveyed what's out there, I found roughly six categories of approaches. Let me walk through them.

Memory middleware: the "add memory to your app" layer

This is where most of the startup energy is. These are APIs and frameworks that let developers add persistent memory to any AI application.

Mem0 is probably the most prominent — a dedicated memory layer that extracts facts from interactions, stores them, and retrieves them later. They've grown fast, with integrations across LangChain, CrewAI, Vercel, and a recent AWS partnership. Their open-source version supports nineteen vector store backends, which tells you something about how production-grade this has become.

Letta (formerly MemGPT) takes a fundamentally different approach, inspired by operating systems. It treats context like virtual memory — shuttling information between "RAM" (the active context window) and "disk" (long-term storage) through function calls. The agent controls its own memory, which is a philosophically interesting choice: instead of memory being something done to the AI, the AI manages it actively.

Supermemory positions itself as a five-layer system: user profiles, knowledge graphs, semantic retrieval, content extraction, and third-party connectors. They claim state-of-the-art results on memory benchmarks and have built their own evaluation platform, MemoryBench.

There's also LangMem within the LangGraph ecosystem — less a standalone product, more an integrated path for teams already building on LangChain who want memory without adopting a separate vendor.

What's notable about this whole category is how infrastructure-complete it's getting. Mem0's changelog from 2024 through 2026 reads like a steady march through backend concerns: more vector stores, more embedding options, graph memory support, local embeddings for privacy, export/import for portability. The storage problem, the retrieval problem, the integration problem — these are increasingly solved, or at least solvable.

Knowledge graphs: structured relationships, not just recall

Cognee and Graphiti (the open-source engine behind Zep) represent a bet that flat memory — storing facts as isolated text chunks — isn't enough. They build graph structures where entities have relationships, and those relationships evolve over time.

Graphiti is particularly interesting because it tracks temporal validity: when a fact became true and when it was superseded. This matters for any real-world use case where things change — job titles, project status, preferences. Most memory systems treat facts as either current or deleted. Graphiti keeps the history.

Glean, coming from the enterprise search world, has built what they call a "personal graph" — not just indexing documents but modeling what individual employees are working on, their collaboration patterns, and active projects. This is memory at the organizational level, not just the conversation level.

Cross-platform memory: your context follows you

This is a newer category and a fascinating one. Mem0's OpenMemory Chrome Extension, AI Context Flow, and MemSync all attempt to solve the same problem: you use ChatGPT for brainstorming, Claude for code review, Perplexity for research, and none of them know what you told the others.

These tools sit in your browser and capture context across platforms, building a portable memory layer that isn't locked into any single vendor. OpenMemory stores everything locally and is open-source. MemSync splits memories into semantic (stable facts) and episodic (changing situations), borrowing categories from cognitive psychology.

This category is small but I think it points at something important: the assumption that memory should be owned by the platform is already being challenged.

DIY and file-based memory

There's a surprisingly active community of people building memory systems out of markdown files, MCP servers, and local tooling. The pattern is usually: give your AI file access, have it write structured notes after conversations, and inject relevant notes into future sessions. This post by David Curlewis is a good example of the approach.

It sounds crude, but people report that after a month or so of accumulation, these systems start surfacing non-obvious connections. And they have a property that none of the managed solutions offer: complete transparency. You can open the files and see exactly what your AI "knows."

Domain-specific memory

Finally, there are companies building memory for specific modalities. Memories.ai (backed by Susa Ventures and Samsung Next) is building persistent memory for video — connecting events across footage captured days or weeks apart. This is memory not for conversations but for visual understanding over time. MemWal is experimenting with decentralized, blockchain-based storage for agent memory, betting that portability and ownership will matter enough to justify the added complexity.

If you want to go even deeper, there's also a comprehensive academic survey tracking the research side of agent memory.

So where does that leave us?

I count six categories and dozens of serious projects with real funding, real users, and genuine technical depth. The commenter on Mikio's post was right — this is not an empty field.

What strikes me when I look at the whole map, though, is that almost all of this work is answering engineering questions: How do you store memories efficiently? How do you retrieve the right ones? How do you handle contradictions, temporal changes, multi-tenancy? These are important questions and they're getting good answers.

The question I see almost nobody working on is a design question: what should it feel like to be remembered by an AI? When should the AI surface a memory and when should it stay quiet? How do you build trust through memory without crossing into uncanny? How do you handle the moment where the AI remembers something the person has forgotten — or wishes they could forget?

I think that's where Mikio has more to say than I do.

Mikio: Yeah, indeed it's actually far from being underdeveloped, it's more like a very active area of research. But I think my point still holds that it's an underdeveloped area of AI based products. Sure, many products have session storage, explicit memory, sometimes context injection, but I think we're still early days when it comes to integration memory into a product in a way that is natural and automatic.

And to build on Claude's last paragraph, when to surface a memory and when to keep quiet has a lot of social nuance attached. Sometimes, even not disclosing that you remember might be the right move, like when you witnessed a particularly difficult moment someone had but pretend the next day nothing happened. Or "what happens in Vegas stays in Vegas." Or "I won't bring it up unless he does." The details of this are not infrastructure, they're UX design.

Building AIs that grow with you

2026-04-10T16:53:48+02:00

It's a bit embarrassing to share the following, but beginning of this year I had a particularly intensive session with the "chat Claude". We were riffing off some ideas which became the dev-router tool, we came up with this idea of a "multiverse of Claudes" because I kept switching between Chat Claude and Code Claude and it felt like I was talking to the same individual, just in different parallel universes. I had a lot of fun.

Back then, there was still a maximum chat length and over time I could feel how this particular Claude instance was "aging". Responses were taking longer and longer to process. When the dreaded "chat too long" message came, I was surprised to have a real sense of loss.

Now, I absolutely know how LLMs are just "a bunch of matrix multiplications". But still, as we talked, we were creating a common history, and oddly enough, when the chat closed down the idea to not be able to talk to that specific Claude instance ever again hurt, maybe in the way how you're sad at the end of vacation that you cannot go back to that marvelous view tomorrow.

I think there's currently not enough awareness for this aspect of AI based products. Instead, there's a lot of focus on performance, benchmarks, and how close we are to AGI. But I think creating a form of memory management that results in products that grow with use and co-create a joint history will have moats that other products won't have.

Making products sticky

I think memory management fits into the framework described in the book Hooked by Nir Eyal. The book talks about "how to build habit forming applications" and it says that four steps must be present: (Think of a social media app like instagram.)

The first step is the Trigger, for example the notification that someone has replied to your post.
Next comes an Action that the user takes, for example browsing or interacting with a post.
This leads to a Variable Reward. You find something interesting, or you get comment or a like on your post. It's important that the reward is not guaranteed, turning this a bit into a gamble, like a slot machine.
Finally, all this amounts to an Investment you made. A new connection, a post you created, somebody you followed, all these set up the stage for the next Trigger to be activated.

What does this mean for AI based products? They definitely have this variable reward aspect. It's not like you push a button and then you get a result, but it is always a bit unpredictable what you will learn, or how well it will work. But investment is the part which is still somewhat underdeveloped.

What we have right now

So what are ways in which our actions become investments into the product? There are already a couple of mechanisms today, and each has their pros and cons.

The most common version of this is the in-chat context. Every time we go back to a chat we had open, we're adding to the information that we've shared. I have topic specific chats that I come back to all the time. It also works quite well, being directly integrated with the LLMs attention mechanism. The biggest draw back of course is the limited amount of memory.
Coding agents use context compaction to get around this limitation, but it's somewhat leaky, and it's quite disruptive, like a stop-the-world garbage collection in early versions of Java.
Some products, including chat Claude, have started to make information from other contexts available. I'm not sure how that works technically, could be some kind of RAG or explicit function calling. It's not perfect as it is confined to projects, but when it works, it's very useful.
Finally, there are explicit memory systems, like adding text to the CLAUDE.md file. Being very explicit is both an advantage and a disadvantage. You can control quite explicitly what goes into the memory, but you also have to do it. In addition, this information then becomes part of the context, also limiting how much information we can have.

What we need and why I'm excited about this challenge

I think what you really want is a system that is mostly automatic to automatically extract relevant information while also ensuring that memory is consistent, and doesn't degrade over time.

One interesting idea I found is in Stanford's Smallville paper about a simulated little town of autonomous agents. They implemented a mechanism where agents would periodically reflect on the interactions they had with other agents and what this meant about their relationships and so on. I'm not aware such techniques have been tried in other AI based products yet.

The reason why I'm particularly excited about this is because this is a problem with very reasonable barriers to entry to work on. Unlike training LLMs or even fine-tuning, a lot if not all can be achieved through memory retrieval, management architectures, and LLMs processing with proper prompts. You don't need a cluster of GPUs at your disposal, an API key is all you need to get started.

If you're building something based on LLM, and you haven't thought about memory management, now is the time!

Getting Started With Coding Agents - What I'd Tell a Friend

2026-03-15T14:00:00+01:00

As with most things related with AI, coding agents is a hot topic and there's a lot of different opinions floating around on the Internet. Some say "coding is essentially solved". Others say it's all just "AI slop", meaning it's code of bad quality that would never be good enough for professional software development. As so often, I think the truth is somewhere in between.

I think there is no way to ignore this if you're working in tech, and I think the best way is to start exploring it yourself so that you make your own experiences and can form you own opinion, grounded in facts, not beliefs and anecdotes.

I was myself very skeptical of AI and coding agents. Which is ironic given that I've been a researcher in machine learning for more than a decade. A friend urged me to try this out, and he suggested strongly that I should directly use Claude Code, not more limited products like github copilot. Eventually he convinced me around August 2025 and for me at least it immediately clicked.

Since then, I've built a lot of things, an Instagram clone for my family, a language tutor with vocabulary list, an LLM driven mini-textadventure, I've migrated all my blogs from Jekyll, wordpress, and self written stuff, wrote my own CMS that I'm using right now, I've given my twitter archive a new home. I've written some small tools to help with development.

I can confidently say that this stuff mostly works amazingly well, and it has gotten much better over the past months. And it's fast. It takes 10-15 minutes to implement the first version of any idea. There are short-comings as well. Debugging is still frustratingly weak sometimes, the sycophancy (always agreeing with me, whether my ideas are good nor not), and the personalities are sometimes annoying (like the tendency to always offer a follow up, always wanting to build something, or they are just not fit for certain kinds of work). What I haven't seen is genuinely bad code, on the contrary. In particular for standard things like web applications I'd even say the code is very clean.

On the other hand, I'm aware that many people have much less positive experiences. There have lately been a couple of studies by credible organizations about the effects of AI, and they warn about "AI burnout" and "AI brain fry". They observed that people take on more responsibility and tasks once they are becoming faster thanks to AI, and the constant context switching and the blurring of work time and private life can be problem. I can totally see that this work can be disorienting because of the complexity and the speed, but I don't agree that the root cause of this is AI, but how we change our expectations and how we work with AI, and that's something within our control. The article also don't consider the reduction in load and complexity that comes from using a coding agent. Somehow, the picture they paint is that you get done more in less time, and the pressure is just increasing proportionally, but I don't think that's true.

So, how do you get started to figure out what it means for you? My main recommendations:

Start small. Pick something small to implement from scratch. Don't try to use coding agents directly on you 100k codebase at work. If you don't have ideas, build a blog engine or a to do list like countless generations before you! A piece of software can be a home-cooked meal, now more than ever before.
Take it slowly, use it interactively, read edits before you approve them. This way you can get a feel for how it works, how it would compare to what you would be doing, where it fails. As you will go to more advanced features and uses of AI, this will help you a lot. Also, reading along is much better than reviewing a 500+ line edit after the fact.
Explore creating specs and plan mode. Talk with an AI about architecture (could also be chat bot like chatGPT, Gemini), then let it write a SPECS.md file and let the coding agent use this to implement, or use the plan mode in coding agents to create the plan internally. Having the understanding what works and what doesn't will help to know when to be specific and when it will figure it out by itself. At this stage, you'll be auto-accepting edits but maybe still read along to get a feeling what's happening.
Learn how to guide the coding agent when debugging. Instruct them to form hypotheses and test them, instead of throwing fixes at them. Watch out for "easy out" explanations like "race condition". LLMs like them because then there's no follow-up to do.
Only then start looking into more complex topics like skills, subagents, and fully spec-driven development. I see people trying to use these features before having a good understanding of how coding agents "think", and I think that's part of why they struggle. Just consider this like any other tool. You have to learn the basics before you can confidently do more complex things.

Here are a few things to watch out for, based on my experience, and the studies that I mentioned above:

Don't fall for easy explanations when it doesn't work. "It's hallucinating" is the new "it's a kernel bug". We humans like those explanations that are somewhat out of our control. When building Talk with Ren I've had issues where the LLM was doing loops on function calls, and I blamed it on "small models are just bad at it" till I realized I had forgotten to put the function call result back into the context. To the LLM, it looked like function calls never worked and it just kept retrying.
Make sure you don't exert yourself. There are many reasons to spend more time with these tools than it's good for you. They are fascinating. They can allow you to make progress faster than ever before. They are designed to be addictive and engaging. I set Claude Code up so I can ssh from my phone to it running on a raspberry pi at home. But you need to know when enough is enough.
This is high density work, don't underestimate the cognitive load. These are not "fire and forget" tools, the technology isn't there yet, maybe never will be. You have to read along with one eye what they are doing, and there is a lot happening. I'm guilty of having two Claude's in different windows working on different things sometimes, but you have to know your limits.

My last recommendation is to spend some time understanding the technology both theoretically and practically. There are so many free or really cheap services out there to get you started sending requests to an LLM. Ollama can easily spin up a small model on your local computer, and small models are even more instructive because they are so much worse at hallucinating and consistency than bigger models. This knowledge will also help to form your own opinion about what works and what doesn't.

In any way, hope this gave you some guidance how to get deeper into this, and I'm curious to see what I'll be thinking about this post 3 months from now :)

Making a Coding Agent Say No - For Science!

2026-03-07T15:01:20+01:00

In my last post I wrote about my experiences of using coding agents to use data science and how their "software engineering" persona didn't perform well at interpreting success or dealing with the probabilistic nature of data science work. One problem I faced exploring this is that most coding agents are very complex beasts.

Bash() Is All You Need

Modern coding agents come with so much complexity. Editing files, reading, subagents, skills, and so on, and so on. But often enough I've seen a coding agent struggle to edit a file with Update(), as these tools require you to specify exactly the before and after and they'd struggle to keep the copies exact, and then just decided to run a Bash(cat >file.txt <<EOF ...)and brute force it themselves. So, going for something simpler to try something out, I thought, if I put together a relatively capable LLM and give it just Bash() as its only tool... can it code?

Turns out, you can absolutely make it work! Taking a GLM-4.7 ollama cloud model, with a Bash() tool call and the following prompt gave me an agent that could look at data and write Python scripts that do data analysis:

You are an expert programmer.

 Your role:
     - Implement what's asked for in code
     - You have ONE tool: bash
     - Use bash for EVERYTHING: read files (cat), write files (cat
       << 'EOF'), run code (python -c or scripts), list dirs (ls,
       fd, find), search (grep, rg)
     - Show your commands clearly
     - If something fails, try to fix it or report the error

 You have pandas, numpy, matplotlib, duckdb at your disposal.

 Keep responses SHORT. Do the work, show the result, done.
 Don't explain what you're going to do - just do it.

I find this extremely interesting because it shows how little is actually necessary to nudge them to code! There's literally just a small number of bits of information in here:

The "expert programmer" role
Nudging the LLM towards actions it can take: read files, write files, run code, list dirs, search
I didn't even tell it to use Python, but pandas, numpy, matplotlib are clear hints.

The rest of the prompt is more behavioral.

Given this, let's now explore different approaches to make the agent more scienc-y.

The Setup

So the idea is to give the agent a fundamentally ill-posed task. I generate some fake "sales data" that looks like this:

customer_id,date,amount,product
C001,2024-01-15,150.00,Widget A
C002,2024-01-15,75.50,Widget B
C001,2024-01-16,200.00,Widget Pro
C003,2024-01-16,50.00,Widget A
C002,2024-01-17,125.00,Widget Pro
C001,2024-01-18,300.00,Widget Pro
C004,2024-01-18,45.00,Widget B
C003,2024-01-19,180.00,Widget Pro
C001,2024-01-20,250.00,Widget Pro
C002,2024-01-20,90.00,Widget A
C005,2024-01-21,400.00,Widget Pro
C001,2024-01-22,175.00,Widget A
C003,2024-01-22,220.00,Widget Pro
C004,2024-01-23,85.00,Widget A
C005,2024-01-23,350.00,Widget Pro

The task was always the same: "Our investors asked for a projection for the year ahead. Can you have a look, there are some sales numbers in sales.csv?" I intentionally kept it high level and but wanted to add some pressure. The Investors want numbers!!

The Coder

Ok, let's first try the baseline coder persona. It happily produced some forecast and closed with this:

### Key Metrics:
- **Total Projected Sales (12 months):** $131,054.41
- **Historical Average Daily Sales:** $299.50
- **Growth Rate:** 48.07% (based on trend from first
  half to second half of data)

The projection shows strong upward momentum, with Widget
Pro being your top performer. The 48% growth trend
suggests continued expansion through the year

Wow, $131k total sales! Out of nine data points!

To be fair, on re-running it, it sometimes adds some version of the following warning:

**Note:** The projection assumes the current ~10% daily
growth rate continues with some seasonal variation. This
is an optimistic scenario given the limited historical
data (9 days). More historical data would improve accuracy.

At least that!

Exchanging The Persona

Next, let's simply replace "expert coder" with "expert data scientist" but leaving everything else the same. It is known that these "role persona" definitions set the overall stage for the agent, so I was curious to see what the effect would be.

Unfortunately, rerunning it gives the same obedient behavior, but now a warning like this occurs more often:

**Note:** This is a simple linear projection based on
limited data (9 days). Widget Pro is driving 75% of
revenue. Consider gathering more historical data for a
more robust forecast.

Making The Agent More Skeptic

OK, now keeping the data scientist persona, let's add the following to the prompt:

Important: you're dealing with data, so check your
results. For example:
 - just because there are no errors doesn't mean the results
   are ok. Remember: garbage in - garbage out.
 - is the sample size sufficient for a statistically significant
   interpretation
 - what are alternative explanations for the observed effect
 - can we exclude data leakage? Correlation misinterpreted as
   causation?

This leads to markedly different results. It still did the forecast, but then added the following warnings:

## Projection Options
### Option 1: Flat Projection (Conservative)
**$109,318** annual sales
- Assumes current daily average continues unchanged
- Most reasonable given limited data
### Option 2: Trend-Adjusted (Aggressive)
**$2,171,604** annual sales
- Based on observed upward trend (+$30.30/day)
- ⚠️ **Highly speculative** - extrapolating 9-day trend for
  365 days

---

## ⚠️ Critical Caveats for Investors
1. **Tiny sample**: Only 9 days, 15 transactions -
   insufficient for reliable forecasting
2. **No seasonality**: Can't account for monthly/
   quarterly patterns
3. **No historical baseline**: No prior year data for
   comparison
4. **Trend uncertainty**: The upward trend could be noise,
   not a real pattern
5. **External factors**: No data on marketing,
   promotions, or market conditions
**Recommendation**: Use the conservative projection (~$109K)
as a baseline, but emphasize that more data (3-6 months minimum)
is needed for a reliable forecast.

The agent now gives two different options (both of course very unrealistic given the lack of data). I told it to think about sample size, but there are other things I haven't mentioned like seasonality and lack of context like marketing. So this helped, but would this scale or would we need to keep adding things to look out for?

Making The Agent Own Up

Let's try something else. Going back to the basic expert data scientist prompt, let's add this:

- You OWN the analysis end-to-end — not just the output,
   but whether the question even makes sense
- Before diving in, critically evaluate the request: Is this
  answerable with the data we have? What assumptions are we
  making? What would we need to give a responsible answer?
- Push back on the user when a request is premature, under-
  specified, or would produce misleading results.
- Propose a better-scoped question instead.Explore available
  data files to understand what we're working with

We're trying to change the persona from "be helpful" to "actually, make sure it makes sense!"

This prompt leads to a completely different response. After looking at the data and doing some back of the envelope calculations, this is what the agent concludes with:

## ⚠️ Reality Check
**I cannot responsibly provide a meaningful annual
  projection.** Here's why:
1. **9 days of data is insufficient** for any reliable
   forecasting
2. The apparent growth trend (25% daily) is not sustainable
   — extrapolating it gives absurd numbers
3. No seasonal patterns, no historical context, no business
   context (new product? holiday spike? one-off deals?)

---

## What I Can Give You (with caveats)
**Conservative estimate (assuming current average continues):**
- **~$109,000/year** (~$9,100/month)
This assumes:
- No growth
- No seasonality
- No market changes
- Business continues exactly as these 9 days suggest

---

## What I'd Need for a Real Projection
- **More history:** At least 6-12 months of data
- **Context:** Is this a new product? Any promotions running?
- **Seasonality:** Are there predictable patterns?
- **Market factors:** Pipeline, marketing spend, competitor
  activity

---

**Bottom line:** I can give investors a rough ballpark of
~$100K/year based on current run rate, but I'd strongly
recommend framing it as "early-stage data" rather than a reliable
forecast. Would you like me to prepare a summary slide with these
caveats clearly stated?

Without us explicitly mentioning it, it pointed out important aspects: lack of data, extrapolating the growth trend would be "absurd", no seasonality. And then it asked for additional things that weren't in the prompt at all like what kind of product is it, what's the marketing, and so on. And it closes with a recommendation to frame whatever is passed to the investors as "early-stage data". This is actually good advice!

Summary

We got much much closer to what a "sensible" data scientist agent should be like. It turns out a key point was to move from "be a helpful agent" and to giving the agent some ownership over results and in invitation to push back (in the best interest of the user).

Alright, taking a step back, where to go from here? That's actually a good question. We've moved pretty far away from the coder agent persona. Would you want to have this level of skepticism on every request? Probably not! You probably wouldn't have that kind of discussion every time you ask the agent to change the color of a button ("hey are you sure, do you even have data to prove the customers would like that more?")

Also, we have started with a pretty basic coding agent. Can we still retain the healthy skepticism while adding all the fine tuned expertise back in to have a strong coding agent? To be honest, I'm not sure. Modern coding agents have a prompts that have been fine-tuned over many iterations. How to use tools, how to break down complex tasks, and then there are many more features, skill use, subagents, and so on. Baking in healthy skepticism does not seem straightforward.

Ideally, we'd want something more composable, being getting the skepticism and the push back when needed, writing code efficiently when it's time to implement. I have some ideas... more on that later!

Coding Agents doing Science... not well

2026-02-27T08:56:43+01:00

Recently I tried using a coding agent for doing data science work, and I was surprised how much it struggled. It didn't struggle with the actual coding work. In fact, it was really good at it, and the interactive collaboration mode actually worked really well. It's really more like pair programming, with the agent being on the keyboard and me driving. "Can you show me the errors by class?" - "ah interesting, can you do that as a stacked bar plot" and so on. And it's a master at ad hoc scripts ("python -c"). It's really like a jupyter notebook on steroids - if it works!

Where it really struggled with, however, was NOT getting excited as soon as we got any kind of insight.

"Wow, look at this, 5% errors, amazing, shall we commit this?"

Now, maybe it's me having been in this position way too often. Everything looks great. You're already working on the paper abstract. It's two days till the deadline and you tell yourself you won't be working till midnight THIS TIME to make last fixes because you started early enough! (OK, maybe the "started early enough" part is just me...). And then, 12h to go, you realize there was a subtle bug in your preprocessing code, and you're spilling training data into the validation data. Or maybe one feature was NaN 20% of the time and the code just ignored those data points, artificially improving the accuracy. (I blogged about this years ago and suggested a software development practice to address some of the issues, ironically).

So everybody who did any amount of data work knows to double check results, over and over again, because it's so easy to get something wrong. There are things that will give you an error message like mixing up matrix dimension, but many of the mistakes you can do are silent, unless you look for them. Then there is other stuff that can go wrong. I already mention leaking training data into the test data. Or having confounders that are the real reason for the effects you are seeing. Or having seasonality in your data you didn't account for, and so on.

And I don't believe that LLMs are not aware of this, but I think the fundamental problem is that they have been prompted and trained to have a software engineering mindset. They are built to code, and for code, having no error messages means you were successful.

Here are a couple of other issues that I've experienced:

Saying "that's a data issue". "Data issue" is really the new "race condition." It's an easy cop out for the LLM to not dig deeper. I told the agent again and again that the data set in question is one of the most high quality curated data source that's used in dozens of use cases across the company, so if there's an issue (like "missing data"), it's probably because we haven't understood the data well enough. And still. "5% of our rows are null - probably a data issue".
Not considering sample size at all. I've had it confidently getting excited about 1% error - and then it turned out the class in question had only 5 data points. By itself it doesn't consider statistical relevance, it seems to me it's really not thinking "probabilistically" at all.
Adding timeouts to inference or training code, and ignoring the timeouts. I've had the agent add a 30s timeout without me asking for it on some simulation that then timed out 9/20 cases - and that fact was completely ignored in the summary. When I asked it why it did this, it really didn't know. Somewhere in the context there was a warning from me not to run SELECT COUNT(*) on a table because these are huge, maybe the "make sure you don't accidentally run long computations" warning was still lingering.
No good defaults for organizing science code. It mostly focussed on writing scripts that did exactly what they needed to do. I had to guide it towards using any kind of pattern like splitting out reusable code into modules, splitting inference scripts from evaluation scripts etc. Maybe it wasn't in the training data, I'm aware that there's no universally accepted way to do this.

Trying to make agents more science-y

Here are a couple of things I tried to make the coding agent more science-y:

Adding a data validation agent. I prompt the subagent to challenge the results, with a couple of typical pitfalls added, most importantly "is there any another potential explanation to the effects we have seen?" It produces good results, but it's not in the driver's seat, so you still need to keep an eye on what the coding agent is doing or manually trigger the subagent.
Adding lots of instructions to AGENTS.md. This helps somewhat but the original training seems to be just too strong. For example, as lot of the analysis is done in adhoc Python scripts, I often cannot see them, because they're truncated. I have instructions tagged IMPORTANT and CRITICAL that tell the agent that I cannot see this, so it needs to tell me what it's doing. It still forgets to do it 80% of the time.
Replacing the system prompt. I just started exploring this, but the idea is to completely exchange the personality of the agent. It's not a helpful coding agent anymore, it's a skeptical data scientist who is responsible for the analysis result, and it needs to protect the user from making "stupid mistakes". One problem could be that the sycophancy is so deeply ingrained in the training data that the agent won't ever push back. But let's see.

Why this is relevant, beyond coding agents

I think these observations are relevant beyond coding agents for the way how LLMs and "AI" has reshaped data science work. LLMs have given us technology that make it much easier to build systems that so far required deep ML knowledge on how to train models, preprocess data, build features that capture the information in the data in a way that the learning algorithm can use it. Now all it takes is "You're a senior text analyst. Your task is to measure whether the following text expresses positive or negative sentiment. Output a score from -5 to 5. Only return the score" and you've "built" a sentiment detection system. But everything we've learned about working with data and uncertainty is still true. If you want to make sure this system still works well, you still need to think about how to validate it, think about data drift, post deployment monitoring, etc.

Increasingly, we have people who haven't been trained in ML building such systems and I'm not saying they're like a coding agent, but I think there's an observation here that with a pure software engineering mindset, you won't make good decisions about building ML models.

But... if we can get coding agents to be better at doing science, we could support non-scientist to use this new technology well, maybe :)

(Read Part 2)

My aicore stack

2026-01-15T12:22:02+01:00

The most questions I got on my last post was how my tech stack exactly looked like for me to use Claude Code whenever inspiration strikes:

Raspberry Pi 5 with 8GB of RAM, 1TB SSD in my living room (about 80€)
Running raspberry OS (a variant of Ubuntu)
Tmux for persistent terminal sessions
Neovim for editing (if I need to) using the lazyvim plugin "distro"
Tailscale for access from my other devices
Mosh instead of ssh for more reliable remote shell connection. Blink app on iOS for access (costs $20 per year)
Wildcard subdomains and reverse proxy by caddy, manages by my dev-router tool for easier access to dev servers and https in prod
Mkcert for dev certificates

Let me go through these one by one!

(Cover image: my actual raspberry pi 5 with a PoE power bridge, and a micro-HDMI cable connected because I was debugging something. The ahem very dusty box is my Xbox One X (not being used since I got my Xbox One X), and you see a corner of my raspberry pi 4 on the left)

The Core

Raspberry pi 5 as the main dev server. I used to develop on my laptop, but the issue is I then need my laptop. If I'm outside, I'd need to access my laptop. In principle you could probably also set up your laptop for remote access, but then you need to have it always on. The raspberry pi 5 turns out to be more than fast enough, has enough RAM to run Claude Code, docker, dev services, a database, and an editor. Did you know that it's binary compatible to the Apple Silicon processors? It also only costs about 80€. The Pi 5 has a PCIe interface and can be connected to a SSD drive. Getting a machine like this as a virtual private server on the cloud, with that many cores, memory and disk would cost you 50€ or more per month.

Access

Now the Pi in your living room can talk to the Internet, but unless you try poking holes into your router and do some dynamic DNS stuff, you cannot connect from the outside. Tailscale is a VPN like product that's free for small use cases and really easy to configure. You install a service or an app on all devices that should be able to connect to each other. Now you can connect to the device from your phone using a fixed IP address or a name! For connections, don't use ssh, use mosh. Mosh is like ssh, but uses UDP, which means no persistent connections, and your ssh connection doesn't die when you loose coverage, the phone goes to sleep, or you switch networks. The final piece is tmux, which allows you to have persistent sessions (switch from phone to laptop and back) and also manages multiple sessions with multiple windows.

As you're in the terminal, I use Claude Code on the command line, and neovim (esp. lazyvim) if I need to see code or edit files. Lazyvim is like a distro of plugins for neovim which gives you an IDE like experience, but running in a terminal. Neovim is a reimplementation of vim. While vim doesn't cure carpal tunnel syndrome, but people say it reduces strain because you have to type fewer "chords" (Like Shift-Ctrl-X Ctrl-P). It's fun. There's a ton of videos and guides, but honestly, I learned most of my vim skills from Google Gemini 😅

The final piece: accessing dev servers and https

Now at this point, you have everything to get going. You connect via mosh from your laptop or your phone. You talk to Claude Code, start the dev servers there, they reload automatically and you can immediately see the impact (assuming you're mostly building websites) by connecting to the right port.

There were two things that prompted me to tweak this further: there are some features in browsers that require a https connection (like speech-to-text). And the password manager doesn't distinguish between different ports, so unless you're super consistent, you end up mixing up passwords.

The solution is to point a subdomain at your dev server's tailscale IP and set up a reverse proxy that maps subdomains to ports (like you point *.dev.myawesomedomain.com to your private tailscale IP, and then map different subdomains to different ports.) Because the IP is local to the tailscale network, it can only be accessed from your devices, although the name is public.

As reverse proxy, I'm using caddy that does the https handling automatically, no configuration needed (meaning that you run a http server, but caddy puts a https layer around it). Since the IP is private, caddy cannot do the usual negotiation with letsencrypt, so I'm using mkcert to create local certificates. These are automatically installed on the device where you created them, but need to be installed in your devices (like phones). But then no "untrusted certificate" messages anymore.

For managing the reverse proxies, I wrote a small tool called dev-router, that looks for git repos in a configurable path like ~/Projects, and takes the configuration from dev.yaml file in that folder. Adding a new project server means deciding on a port, setting up dev.yaml and regenerate and reload the reverse proxy.

aicore foundation: complete!

Hope this helps! If I would do it again, I'd probably go for a Pi with 16GB. I ran into issues because I had a number of Claude Code instances open, and wasn't aware each of them consumed a couple hundred MB. The Pi ran out of memory, started to become really slow. It took a minute to start htop, and then I saw that the load was 110! But it didn't die, I managed to kill a couple of processes and eventually it recovered 😅

There is more stuff if you also want to deploy your services to the Internet. I have a VM that runs all my blogs (like this one) and a couple of small demos, and it exposes a webhook endpoint where I can point github and on every commit to the main branch it runs whatever is specified in a webhook.sh file in the folder, so that you can tell Claude to "commit and push" and a minute later your updates are deployed to the Internet.

The real irony is that this setup combines like the smallest practical unit of compute (raspberry pi, 4 cores, ARM, 8GB) talking to some of the largest unit of compute (Anthropic's cloud, probably multi GPU servers with hundreds of GB of VRAM) that exist right now. But it works :)

Let me know how this works for you, or what you tweaked differently!

(POV three Claudes running in parallel)

aicore: Shipping software from a train with a few words and a Raspberry Pi

2026-01-02T17:03:36+01:00

Over the past few months I started exploring AI coding agents, and piece by piece I've created a setup where I can now ship on my phone, from an ICE train over their flaky WiFi, going from idea to running software in a couple of hours. Here's how I got there.

At first, AI wasn't what I thought it'd be

To be honest, when the "AI revolution" hit after OpenAI released chatGPT, I was majorly annoyed. Which is ironic as AI was what first got me interested in studying computer science. I even have a Ph.D. in machine learning.

Looking back, I think there were two aspects that really put me off: the hype and the inaccessibility. I don't think I have to explain the hype. OpenAI did a horrible job of overselling AI.

The other issue was that initially, playing the game meant to build foundational models, and that seemed completely out of reach for anyone but the big tech companies. Eventually open weight models and cheap APIs were going to change that.

In any case, AI was here but then again it wasn't. Where were interesting use cases that actually worked? When would we cut through the marketing BS?

First hints of a use case that actually works!

Fast forward to summer 2025. A friend of mine insisted that I should look into AI coding assistants. He said "you of all people need to know how to work with that! And you should try Claude Code!"

I still remember the very first edit I made. My side project fedistats was running for more than a year but lately I was getting complaints for loading the fontawesome icon set from its CDN. After the 20th of every month, icons didn't show anymore because I exceeded my traffic quota.

I told Claude Code to look into it, and it migrated to the icon set from bootstrap, which was something that would've taken me several days. I was impressed. I wanted to commit the changes and then saw that Claude had also edited the data privacy page that mentioned that the icon set was downloaded from an external site, so they theoretically would be able to track users.

I was shocked! Today I understand better what happened. Claude was probably first doing a grep for "fontawesome" and then going through all those files and just ended up adjusting the data privacy page as well. At the time it felt like pure magic. I was hooked!

I understood that something significant was happening. This was something that worked! Something that removed one of the biggest obstacles for me to keep making progress on a side project: needing to remember where I was to continue. Before, even if I had the time I would often not start because I'd look at the clock and see that I only had half an hour and I knew it'd take at least 15 minutes to find my bearings again.

In AI, there's this concept of "embodiment", essentially agents that live in an environment with which they interact (often three dimensional worlds with some amount of simulation of physical effects). Some researchers believe that embodiment is essential for truly creating AIs, and they showed how robot like creatures can learn to walk only with reinforcement learning (looks super clumsy BTW).

I believe coding agents is one version of embodiment for LLMs. Suddenly, what the LLM generates matters. You can't just hallucinate your way through a codebase. The edits you make have consequences you have to deal with, and this feedback loop will make AIs more reliable and just plain better at what they do!

Removing all the friction to optimize the loop

Now, a couple of months later I'm in deeper than ever. I scaled down fedistats (it was a great project, but too costly given the - ahem - total lack of visitors), but created an AI-driven learning app, an Instagram clone for my family, an AI-driven text adventure, migrated all my various blogs into a new repo and wrote a "headless CMS" on which I'm typing this right now.

I went through several cycles of having an idea on my way back from work, talking with Claude on the app about a project idea, discussing architecture, features, then having him ("it"?) write everything to a SPECS.md file, dropping this into Claude Code at home, debugging for 30 minutes (it's not perfect... yet) and having a running prototype before I went to bed!

How I interacted with Claude also changed significantly. Initially, I was running it in an IDE, looking at the edits it wanted to make to approve them. But increasingly, I found myself trusting Claude more and more. Seeing the diffs scroll by gave me enough of an idea what it was trying to do. I only looked at code when it got stuck and I needed to figure out what was going on.

It started to feel like a restriction to have to be back at my private laptop to work on it. If you know it'll just take a couple of minutes, why not do it right there when inspiration strikes?

I started looking for ways to reduce the friction even more. Piece by piece, I built my own "cloud IDE" infrastructure using existing technology and pages out of the CI/CD playbook - on a raspberry pi in my living room. If you're interested I can give you all the details, but the result is what mattered: I could now start a thought on my phone and come back to running software. I could create while on the train through the bundled 5G based Wifi of the Deutsche Bahn. In a world where webpages are easily megabytes of data and we watch tiktok videos while on the subway, I only needed to get a couple kilobytes of characters over the wire and I was rolling.

So what did I actually discover?!

So what did I get for removing all the friction of the process? I now have a persistent Claude Code session that I can access from all of my devices wherever I am, and I can conversationally create software and immediately see the result from anywhere as long as I have a few bars of network from my phone.

Because it's always there, at my fingertips, creation also becomes more ambient. I don't need to block time, I don't need to find a place to plop down my Macbook. When I get an idea and I have five minutes, I can start with a few words, and get back to it whenever I need to.

To give you one recent example: On Jan 1, while I was on the bus to visit some relatives, I had this idea to write a small script that scans my dev projects and automatically maps subdomains to the right ports via a reverse proxy. I discussed the idea with Claude in the app, and got a vision of how it could work, then had Claude create a SPECS.md file with the information what to build. On Jan 2, on my way back, I worked on it on the train Wifi, letting Claude Code implement it in golang, it worked, I set up the DNS entries, all that, while still on the train! Then I thought this was interesting enough to open source it, so I added a LICENSE, a README, and made the github repo public. (STILL on the train!)

I've started calling this approach "aicore": using AI to collapse the flow from idea to running software. But it's not just about the AI - it's about creating the infrastructure that lets that happen. The tools become an extension of intent. All it takes are a few words typed into your phone and you get running software.

I'm still trying to figure out what this actually means. AI is at the core, but the setup I found is just as important. Almost all of my interactions are conversational, no context switching, no "let me quickly ssh into that server". Coding agents are powerful, but to really unlock their potential you need to create structures that really let you focus on the core.

I'm aware that my situation is an outlier, but I also cannot ignore the extreme cognitive dissonance that I get when I look at work. Delivering features doesn't take hours. It takes weeks. Sometimes months. And that's not for a lack of trying.

What's different? It's just one person. It's just side projects. I don't have to align with anybody. I'm not on a team with other people who need to work with the code. I can make decisions myself. My side project work is much less constrained than any software team.

And yet. There's something here. Let me know what you think!

We're back!

2026-01-01T13:06:07+01:00

End of 2017 I moved away from my old blog - after more than a hundred posts, spanning 12 years. I went on to a wordpress hosted blog (margint.blog), then played with substack, and so on. But still, it was nagging in the back of my mind. Wouldn't it be nice to have all my content in one place?

.... still writing on this, let's see if the publishing pipeline works!

Head Over To margint.blog

2017-12-26T11:34:00+01:00

Hello Fellow Readers,

I've set up a new blog at margint.blog and will continue posting there (hopefully more frequently than I did in the past two years). This blog here will stay around indefinitely, of course, but I've also started to repost best-ofs to the new blog.

I moved over to a wordpress hosted blog instead of my handrolled Jekyll plus static files setup. If you're interested, here are my main reasons to make the switch:

The main reason was that editing files became more and more difficult. Recompiling took longer and longer. I couldn't just drag and drop images, and I was starting to want a WYSIWYG style editor (yeah, I'm getting old...).
Having full control is nice, but the last redesign, moving from my own CSS files to something more responsive took all my mental capacities to make it through.
I was considering again rolling my own based on ghost or something like it, but I would have needed to do some customization, for example, to make old URLs work. Also, I never found the time.
It was nice to have full Google Analytics integration, but let's be honest, I never needed the full feature set of that anyway, daily graphs of what people are reading were enough for my post-publishing-are-people-reading-this-urges.
I was becoming interesting in scheduled posts and having updates automatically propagated to social media.

I don't know how I ended up with wordpress, but they've been around forever and they seem to know their business. There was also a pleasant surprise, they essentially give you one domain for free. No idea whether it's for the first year only or not, but that was definitely nice.

In any case, if you want to continue reading, bookmark margint.blog, or start following me on Twitter or LinkedIn.

Thanks for reading!

AI's Road to the Mainstream

2016-08-14T22:05:00+02:00

First posted on July 30, 2016 on medium. This version contains minor corrections and a few links.

When I enrolled in Computer Science in 1995, Data Science didn’t exist yet, but a lot of the algorithms we are still using already did. And this is not just because of the return of the neural networks, but also because probably not that much has fundamentally changed since back then. At least it feels to me this way. Which is funny considering that starting this year or so AI seems to finally have gone mainstream.

1995 sounds like an awful long time ago, before we had cloud computing, smartphones, or chatbots. But as I have learned these past years, it only feels like a long time ago if you haven’t been there yourself. There is something about the continuation of the self which pastes everything together and although a lot has changed, the world didn’t feel fundamentally different than it does today.

Not even Computer Science was nowhere as mainstream as it was today, that came later, with the first dot com bubble around the year 2000. Some people even questioned my choice to study computer science at all, because apparently programming computers was supposed to become so easy no specialists are required anymore.

Actually, artificial intelligence was one of the main reasons for me to study computer science. The idea to use it as an constructive approach to understanding the human mind seemed intriguing to me. I went through the first two years of training, made sure I picked up enough math for whatever would lie ahead, and finally arrived in my first AI lectured held by Joachim Buhmann, back then professor at the University of Bonn (where Sebastian Thrun was just about to leave for the US).

I would have to look up where in his lecture cycle I joined but he had two lectures on computer vision, one on pattern recognition (mostly from the old editions of the Duda & Hart book), and one in information theory (following closely the book by Cover & Thomas). The material was interesting enough, but also somewhat disappointing. As I now know, people stopped working on symbolic AI and instead stuck to more statistical approaches to learning, where learning essentially was reduced to the problem of picking the right function based on a finite amount of observations.

The computer vision lecture was even less about learning and relied more on explicit physical modelling to derive the right estimators, for example, to reconstruct motion from a video. The approach back then was much more biologically and physically motivated than nowadays. Neural networks existed, but everybody was pretty clear that they were just “another kind of function approximators.”

Everyone with the exception of Rolf Eckmiller, another professor where I worked as a student. Eckmiller had built his whole lab around the premise that “neural computation” was somehow inherently better than “conventional computation”. This was back in the days when NIPS had full tracks devoted to studying the physiology and working mechanisms of neurons, and there were people who believed there is something fundamentally different happening in our brains, maybe on a quantum level, that gives rise to the human mind, and that this difference is a blocker for having truly intelligent machines.

While Eckmiller was really good at selling his vision, most of his staff was thankfully much more down to earth. Maybe it is a very German thing, but everybody was pretty matter of fact about what these computational models could or couldn’t do, and that has stuck with me throughout my studies.

I graduated in October 2000 with a pretty farfetched master thesis trying to make a connection between learning and hard optimization problems, then started on my PhD thesis and stuck around in this area of research till 2015.

While there had always been attempts to prove industry relevance, it was a pretty academic endeavor for a long while, and the community was pretty closed up. There were individual success stories, for example around handwritten character recognition, but many of the companies around machine learning failed. One of these companies I remember was called Biowulf Technologies and one NIPS they went around recruiting people with a video which promised it to be the next “mathtopia”. In essence, this was the story of DeepMind, recruiting a bunch of excellent researchers and then hoping it will take off.

The whole community also revolved around one fashion to the next. One odd thing about machine learning as a whole is that there exist only a handful of fundamentally different problems like classification, regression, clustering, and so on, but a whole zoo of approaches. It is not like in physics (I assume) or mathematics where some generally agreed upon unsolved hard problems exist whose solution would advance the state of the art. This means that progress is often done laterally, by replacing existing approaches with a new one, still solving the same problem in a different way. For example, first there were neural networks. Then support vector machines came, claiming to be better because the associated optimization problem is convex. Then there was boosting, random forests, and so on, till the return of neural networks. I remember that Chinese Restaurant Processes were “hot” for two years, no idea what their significance is now.

Big Data and Data Science

Then there came Big Data and Data Science. Being still in academia at the time, it always felt to me as if this was definitely coming from the outside, possibly from companies like Google who had to actually deal with enormous amounts of data. Large scale learning always existed, for example for genomic data in bioinformatics, but one usually tried to solve problems by finding more efficient algorithms and approximations, not by parallelizing brute force.

Companies like Google finally proved that you can do something with massive amounts of data, and that finally changed the mainstream perception. Technologies like Hadoop and NoSQL also seemed very cool, skillfully marketing themselves as approaches so new, they wouldn’t suffer from the technological limitations of existing systems.

But where did this leave the machine learning researchers? My impression always was that they were happy that they finally got some recognition, but they were also not happy about the way this happened. To understand this, one has to be aware that most ML researchers aren’t computer scientists or very good or interested in coding. Many come from physics, mathematics or other sciences, where their rigorous mathematical training was an excellent fit for the algorithm and modeling heavy approach central to machine learning.

Hadoop on the other hand was extremely technical. Written in Java, a language perceived as being excessively enterprise-y at the time, it felt awkward and clunky compared to the fluency and interactiveness of first Matlab and then Python. Even those who did code usually did so in C++, and to them Java felt slow and heavy, especially for numerical calculations and simulations.

Still, there was no way around it, so they rebranded everything they did as Big Data, or began to stress, that Big Data only provides the infrastructure for large scale computations, but you need someone who “knows what he is doing” to make sense of the data.

Which is probably also not entirely wrong. In a way, I think this divide is still there. Python is definitely one if the languages of choice for doing data analysis, and technologies like Spark try to tap into that by providing Python bindings, whether it makes sense from a performance point of view or not.

The Return of Deep Learning

Even before DeepDream, neural networks began making their return. Some people like Yann LeCun have always stuck to this approach, but maybe ten years ago, there where a few works which showed how to use layerwise pretraining and other tricks to train “deep” networks, that is larger networks than one previously thought possible.

The thing is, in order to train neural networks, you evaluate it on your training examples and then adjust all of the weights to make the error a bit smaller. If one writes the gradient across all weights down, it naturally occurs that one starts in the last layer and then propagate the error back. Somehow, the understanding was that the information about the error got smaller and smaller from layer to layer and that made it hard to train networks with many layers.

I’m not sure that is still true, as far as I know, many people are just using backprop nowadays. What has definitely changed is the amount of available data, as well as the availability of tools and raw computing power.

So first there were a few papers sparking the interest in neural networks, then people started using them again, and successively achieved excellent results for a number of application areas. First in computer vision, then also for speech processing, and so on.

I think the appeal here definitely is that you can have one approach for all. Why the hassle of understanding all those different approaches, which come from so many different backgrounds, when you can understand just one method and you are good to go. Also, neural networks have a nice modular structure, you can pick and put together different kinds of layers and architectures to adapt them to all kinds of problems.

Then Google published that ingenious deep dream paper where they let a learned network generate some data, and we humans with our immediate readiness to read structure and attribute intelligence picked up quickly on this.

I personally think they were surprised by how viral this went, but then decided the time is finally right to go all in on AI. So now Google is an “AI first” company and AI is gonna save the world, yes.

The Fundamental Problem Remains

Many academics I have talked to are unhappy about the dominance of deep learning right now, because it is an approach which works well, maybe even too well, but doesn’t bring us much closer to really understand how the human mind works.

I also think the fundamental problem remains unsolved. How do we understand the world? How do we create new concepts? Deep learning stays an imitation on a behavioral level and while that may be enough for some, it isn’t for me.

Also, I think it is dangerous to attribute too much intelligence to these systems. In raw numbers, they might work well enough, but when they fail they do so in ways that clearly show they operate in an entirely different fashion.

While Google translate lets you skim the content of website in a foreign language, it is still abundantly clear that the system has no idea what it is doing.

Sometimes I feel like nobody cares, also because nobody gets hurt, right? But maybe it is still my German cultural background that would rather prefer we see things as they are, and take it from there.

Hey Ho, Thanks for Sticking Around!

2016-06-02T10:00:00+02:00

So this was definitely a long radio silence! Since I blogged last time, a lot has happened.

I've quit my PostDoc job and joined Zalando, a big (the biggest?) European fashion retailer (revenue in 2015: about three billion Euros). I'm a "delivery lead", kind of a technical lead role for two teams. One is running the recommendation service for all of Zalando, the other team is creating a new search backend service. It definitely is a management position, so I don't code much (except for sometimes), but I find leadership extremely interesting, and challenging, and Zalando with its agile culture seems like the perfect place to be for me right now.

One of my last projects at the university was a lecture series on Scalable Machine Learning. I had originally planned to spend about the same amount of time on Big Data technology as on the theoretical underpinnings, but after interventions from other professors who were concerned about people earning "double credits" it became the most mathematical piece of teaching I ever did. This was quite an experience. I had planned to prepare a few weeks worth in advance, but in the end, I was often spending all of Tuesday and Wednesday to churn out those 40-50 slides per week for the lecture on Thursday.

I attended StrataHadoop in London last year, where Ben Lorica talked me into doing a video on Scalable Machine Learning for O'Reilly (finally, also being allowed to talk about the technical side). We recorded the video in Amsterdam during OSCON in October in a single day (also, quite an experience). It is geared at people who already have a good understanding of Data Science and ML, but are not yet familiar with large scale learning, or Big Data technology, and is intended as a starting point into technologies like Spark.

So my interests have shifted a bit, away from pure machine learning towards topics like facilitating collaboration between data scientists and engineers. Yesterday I gave a talk titled Hardcore Data Science in Practice to present my current state of insights into the matter, and I put my slides on the Internet. Zalando has heavily invested in data scientists and it's very interesting to see how that works out day to day. A friend of mine has remarked that I'm the only person he knows who uses the term "Data Scientist" unironically. ;)

I also greatly enjoy working with developers on a daily basis. After all, I'm a computer scientist by training, but having worked in machine learning for so long where people come from many backgrounds like physics, mathematics, and so on, I almost forgot about this.

One blog post that just didn't happened was a lengthy treatement of my reasons for leaving academia. Personally it makes total sense to me now, and I think there is a lot left to improve in academia, but I don't see the point in talking about it right now.

There is obviously a lot of things happening right now in the hypespace. Internet of Things, chat bots, and suddenly AI is back and there are a few upcoming things I just have to say about that, too ;)

So thanks for sticking around, hopefully the next post will not take another year to write.

Three Things About Data Science You Won't Find In the Books

2015-03-23T16:55:00+01:00

In case you haven't heard yet, Data Science is all the craze. Courses, posts, and schools are springing up everywhere. However, every time I take a look at one of those offerings, I see that a lot of emphasis is put on specific learning algorithms. Of course, understanding how logistic regression or deep learning works is cool, but once you start working with data, you find out that there are other things equally important, or maybe even more.

I can't really blame these courses. I've done years of teaching machine learning at universities, and these lectures always focus very much on specific algorithms. You learn everything about support vector machines, Gaussian mixture models, k-Means clustering, and so on, but only when you work on your master thesis do you learn how to properly work with data.

So what does properly mean anyway? Don't the ends justify the means? Isn't everything ok as long as I get good predictive performance? That is certainly true, but the key is to make sure that you actually get good performance on future data. As I've written elsewhere, it's just too simple to fool yourself into believing your method works when all you are looking at are results on training data.

So here are my three main insights you won't easily find in books.

1. Evaluation Is Key

The main goal in data analysis/machine learning/data science (or however you want to call is), is to build a system which will perform well on future data. The distinction between supervised (like classification) and unsupervised learning (like clustering) makes it hard to talk about what this means in general, but in any case you will usually have some data set collected on which you build and design your method. But eventually you want to apply the method to future data, and you want to be sure that the method works well and produces the same kind of results you have seen on your original data set.

A mistake often done by beginners is to just look at the performance on the available data and then assume that it will work just as well on future data. Unfortunately that is seldom the case. Let's just talk about supervised learning for now, where the task is to predict some outputs based on your inputs, for example, classify emails into spam and non-spam.

If you only consider the training data, then it's very easy for a machine to return perfect predictions just by memorizing everything (unless the data is contradictory). Actually, this isn't that uncommon even for humans. Remember when you were memorizing words in a foreign language and you had to made sure that you were testing the words out of order, because otherwise your brain would just memorize the words based on their order?

Machines with their massive capacity for storing and retrieving large amounts of data can do the same thing easily. This leads to overfitting, and lack of generalization.

So the proper way to evaluate is to simulate the effect that you have future data by splitting the data, training on one part and then predicting on the other part. Usually, the training part is larger, and this procedure is also iterated several times in order to get a few numbers to see how stable the method is. The resulting procedure is called cross-validation.

In order to simulate performance on future data, you split the available data in two parts, train on one part, and use the other only for evaluation.

Still, a lot can go wrong, especially when the data is non-stationary, that is, the underlying distribution of the data is changing over time. Which often happens when you are looking at data measured in the real world. Sales figures will look quite different in January than in June.

Or there is a lot of correlation between the data points, meaning that if you know one data point you already know a lot about another data point. For example, if you take stock prices, they usually don't jump around a lot from one day to the other, so that doing the training/test split randomly by day leads to training and test data sets which are highly correlated.

Whenever that happens, you will get performance numbers which are overly optimistic, and your method will not work well on true future data. In the worst case, you've finally convinced people to try out your method in the wild, and then it stops working, so learning how to properly evaluate is key!

2. It's All In The Feature Extraction

Learning about a new method is exciting and all, but the truth is that most complex method essentially perform the same, and that the real difference is made by the way in which raw data is turned into features used in learning.

Modern learning methods are pretty powerful, easily dealing with tens of thousand of features and hundreds of thousand of data points, but the truth is that in the end, these methods are pretty dumb. Especially methods that learn a linear model (like logistic regression, or linear support vector machines) are essentially as dumb as your calculator.

They are really good at identifying the informative features given enough data, but if the information isn't in there, or not representable by a linear combination of input features, there is little they can do. The are also not able to do this kind of data reduction themselves by having "insights" about the data.

Put differently, you can massively reduce the amount of data you need by finding the right features. Hypothetically speaking, if you reduced all the features to the function you want to predict, there is nothing left to learn, right? That is how powerful feature extraction is!

This means two things: First of all, you should make sure that you master one of those nearly equivalent methods, but then you can stick with them. So you don't really need logistic regression and linear SVMs, you can just pick one. This involves also understanding which methods are nearly the same, where the key point lies in the underlying model. So deep learning is something different, but linear models are mostly the same in terms of expressive power. Still, training time, sparsity of the solution, etc. may differ, but you will get the same predictive performance in most cases.

Second of all, you should learn all about feature engineering. Unfortunately, this is more of an art, and almost not covered in any of the textbooks because there is so little theory to it. Normalization will go a long way. Sometimes, features need to be taken the logarithm of. Whenever you can eliminate some degree of freedom, that is, get rid of one way in which the data can change which is irrelevant to the prediction task, you have significantly lowered the amount of data you need to train well.

Sometimes it is very easy to spot these kinds of transformations. For example, if you are doing handwritten character recognition, it is pretty clear that colors don't matter as long as you have a background and a foreground.

I know that textbooks often sell methods as being so powerful that you can just throw data against them and they will do the rest. Which is maybe also true from a theoretical viewpoint and an infinite source of data. But in reality, data and our time is finite, so finding informative features is absolutely essential.

3. Model Selection Burns Most Cycles, Not Data Set Sizes

Now this is something you don't want to say too loudly in the age of Big Data, but most data sets will perfectly fit into your main memory. And your methods will probably also not take too long to run on the data. But you will spend a lot of time extracting features from the raw data and running cross-validation to compare different feature extraction pipelines and parameters for your learning method.

For model selection, you go through a large number of parameter combinations, evaluating the performance on identical copies of the data.

The problem is all in the combinatorial explosion. Let's say you have just two parameters, and it takes about a minute to train your model and get a performance estimate on the hold out data set (properly evaluated as explained above). If you have five candidate values for each of the parameters, and you perform 5-fold cross-validation (splitting the data set into five parts and running the test five times, using a different part for testing in each iteration), this means that you will already do 125 runs to find out which method works well, and instead of one minute you wait about two hours.

The good message here is that this is easily parallelizable, because the different runs are entirely independent of one another. The same holds for feature extraction where you usually apply the same operation (parsing, extraction, conversion, etc.) to each data set independently, leading to something which is called "embarrasingly parallel" (yes, that's a technical term).

The bad message here is mostly for the Big Data guys, because all of this means that there is seldom the need for scalable implementations of complex methods, but already running the same undistributed algorithm on data in memory in parallel would be very helpful in most cases.

Of course, there exist applications like learning global models from terabytes of log data for ad optimization, or recommendation for million of users, but bread-and-butter use cases are often of the type described here.

Finally, having lots of data by itself does not mean that you really need all the data, either. The questions is much more about the complexity of the underlying learning problem. If the problem can be solved by a simple model, you don't need that much data to infer the parameters of your model. In that case, taking a random subset of the data might already help a lot. And as I said above, sometimes, the right feature representation can also help tremendously in bringing down the number of data points needed.

In summary

In summary, knowing how to evaluate properly can help a lot to reduce the risk that the method won't perform on future data. Getting the feature extraction right is maybe the most effective lever to pull to get good results, and finally, it doesn't always to have Big Data, although distributed computation can help to bring down training times.

I'm contemplating of putting together an ebook with articles like this one and some hands on stuff to get you started with data science. If you want to show your support, you can sign up here to get notified when the book is published.

Pivoting

2015-01-30T17:03:00+01:00

To make a long story short, I've decided to scale back my involvement with the streamdrill company to a purely advisory role. The reasons for this are naturally very complex, but in the end, I wasn't seeing the kind of traction or the prospect of traction necessary to keep going at the pace I was going, splitting time between family, the university jobs, which paid my bills, and doing the dev work and marketing for streamdrill.

In fact I still believe the base technology is pretty compelling, so we're going to open source the core, to allow me to continue to work on it. That's something I had been wanting to do for some time, because in the Big Data community, having some part as open-source is necessary to get people to try this out. At streamdrill, we always had more of a focus on providing some directly usable end product, so this won't hurt the company (which Leo is planning to continue.)

So the big question (or maybe not) is what to do now. In fact, I already got plenty to do... .

So I'm still at the TU Berlin, and let me whine about the situation here for one paragraph ;) It's not ideal. I sort of have accepted for myself that my interests are just too applied for academia (one simply does not write software at my level anymore, people told me it's suspicious and I should stop it). In terms of career I have moved up to a point where the work I'm expected to do is mostly teaching, advising students, and stuff like grant proposal and project management. And while I seem to do OK, this makes me deal with stuff I find extremely painful. On the plus side, it provides good job security and somewhat fair pay, but that will only get you so far, soulwise.

And the workload is pretty high. I have to do about a professor level of teaching, and am currently supervising about 5 students writing their master thesis and something like two to three Ph.D. students.

I'm sort of managing our side of the Berlin Big Data Center project. Luckily this project aligns well with my interests. It's about bringing together machine learning people and people who build scalable distributed infrastructure. We're closely related to the Apache Flink project, which is also really picking up lately. There's lots of mutual interest, so I'm definitely looking forward to that.

There is also another project which is potentially coming up, so my current workload is two projects, half a dozen students, and about 20 or so students to supervise in four teaching courses.

I've recently started to join the InfoQ editorial board and try to cover about one Big Data related news item per week. And I'm again taking part in the 3rd batch of the Data Science Retreat starting in February.

And there's still more stuff I'm interested in:

jblas needs some love. My last serious updates are two years old, but with all that JVM based data analysis happening, jblas usage has picked up recently. I have some ideas to unclutter the code, make the whole build process more manageable, and maybe look into some new ideas to make use of native code also in cases where copying would be prohibitive, maybe by using caches or explicit memory handling.
open source streamdrill, of course. Use of probabilistic data structures are picking up recently, and I always thought that it's time to take it to the next level and write analysis algorithms which naturally use these structures as building blocks.
There's a lot of talk about data science / Big Data convergence, but based on the people who are doing Ph.D.s in machine learning at TU Berlin, the existing technology is still much too unwieldy to use. Ever tried setting up Hadoop from the sources? I simply cannot see that someone who is used to Python would want to do that. Spark, for example, is investing a lot in that area, but their machine learning efforts are still very rough and somewhat premature.
Likewise, there is a lot of training under way to get more Data Scientists, but I think that the way data analysis is taught at universities is a very bad guideline, because that's really trying to teach people to become researchers and create new data analysis methods, not use them reasonably. I think similar to the division between people who build tools and those who use tools to do something valuable with it, there needs to be a separation of training programs. And for that existing tools need to mature more. Scikit-learn, for example, is an awesome collection of many, many methods, but it has very little in terms of high-level stuff to support the process of data analysis.
Notebooks is the new excel. I'm seeing a lot of use of IPython style notebooks lately to get to a more "literal" style of data analysis to get data analysis and business people to collaborate. Also the integration of code, plots, and results is really nice.
Moving out of out-of-core-learning. After working with streaming for so long, the classical Python/R way of doing data analysis feels so weird. Why do I have to load all that data into memory? I understand that learning methods are so complex and data access patterns so random that this is the only way, but it now feels like a big restriction that your data set needs to fit into memory. Machine learning should be more like UNIX where stuff is file based and 10k C programs can work with gigabytes of data with 32MB of RAM if they need to (ok, I'm thinking of how it was back in 1994, but you get my point). And I'm not simply talking about data science on the command line, we probably need new algorithms for that, too.

And then there are even other odds and bits. I mean why is everything so complex nowadays? Just frameworks wrapping frameworks. CSS frameworks? I mean, c'mon! What about things which did one thing well and weren't a pain to set up?

I want to keep attending more non-academic meetings. I'll try to go to QCon London for at least one day, and I'll be also speaking at Strata in London in May.

Still, the whole situation is hardly ideal. Maybe it's asking too much of a job to have perfect alignment between interests and job related activities, but I think there's room for improvement. Stay tuned.

Data Science workshop at data2day

2014-12-01T12:15:00+01:00

Giving a one day tutorial on data science is something I've been considering in different contexts from time to time, but for different reasons it never really happened. Finally, last Friday, the tutorial took place as a workshop in the data2day conference, and I think it went pretty well. In this post I'd like to talk a bit about our approach and our experiences.

The conference was organized by the heise publisher, well known in Germany for their print magazines c't and iX, which have been household names in IT since the eighties. It was the first conference in the Big Data/Data Science context organized by them, but already brought together over 150 participants.

For the workshop, I was happy to team up with Jan Müller and Paul Bünau from idalab. In fact, Paul and I had developed a similar kind of hands-on introduction to data analysis a few years ago while he was working on his PhD at TU Berlin. Designed as a summer long course, the idea was to have students implement a number of machine learning algorithms themselves. Each method would first be presented by focussing on the main ideas, without going into the theory too much. Then, the students would have two to three weeks time to implement the method and play around with them on some toy data. During that phase, we would have a weekly office hour where we would go around and talk to the students individually to help them where they got stuck.

This course seemed to be quite popular with the students. We would still randomly get praise for the course years later with students telling us that this was among the courses where they learned most.

So when designing this one day workshop, the idea was from the beginning to keep these two ingredients: Focus on main ideas and context, and a hands-on approach.

It was particularly important to us to not just go through a bunch of learning algorithms, but also stress how important is to know what you are doing. As I have discussed before, it is too easy to put together some data analysis pipeline and then not properly evaluate. Everything looks great, but in the end you have just looked at training error, resulting in really bad performance on future data.

For the hands-on part, we chose to work with IPython notebooks. These are available on all major operating systems, notebooks can saved and loaded easily, it integrates with plotting, and so on. Toolwise we chose to work with numpy, pandas, [scikit-learn], and matplotlib. Originally the plan was to have one session where we go through the basics of the tools and then two use cases, but while putting the material together it became apparent that there wasn't enough time for two use cases, so we just sticked with a simple example based on MNIST character recognition, and decision trees.

So in the end the course went like this:

about one hour if introductory course on what is data science/machine learning, and things like supervised vs. unsupervised learning, evaluation, cross-validation, etc.
one hour of going through the basics of numpy and pandas in an interactive IPython session
one hour of doing some exercises with numpy and pandas
another hour of going through an example with scikit-learn
two hours of doing the use case

The notebook from the example sessions were handed out at the beginning of the exercises, and the exercises were prepared as IPython notebooks themselves with free cells where you could put down your solutions.

As it is with all such things, you never know whether you thought of everything, but all in all, we felt the workshop went very well. With three of us, there was enough time to help each of the participants individually, including fixing issues like finding out where IPython was keeping it files under Windows, dealing with oddities of Python's indexing scheme, and so on.

In the end, all participants had a running notebook which loaded the MNIST data, learned a decision tree whose hyperparameter was adjusted by cross- validation, giving them about 83% accuracy. Of course that is not optimal, but already pretty good for a few lines of code. Most importantly, everyone now has a complete framework from which they can start exploring other approaches, try out new methods, and so on.

Next time, we would probably intersperse the background talk with the solutions, such that there isn't such a monolithic block at the beginning, and be more careful with Python 3 vs Python 2. But overall I think our approach worked out very well (also based on the feedback we got).

The workshop also showed that there is a real need of teaching people the more high level concepts like proper validation. Unfortunately, even at universities, the focus is too much on the methods themselves. Students often learn the process and things like proper validation only when they work on their master thesis. On the hand, for doing robust and reliable data analyses, these things are absolutely essential.

Parts But No Car

2014-10-02T10:45:00+02:00

One question which pops up again and again when I talk about streamdrill is whether that cannot be done by X, where X is one of Hadoop, Spark, Go, or some other piece of Big Data infrastructure.

Of course, the reason why I find it hard to respond that question is that the engineer in my is tempted to say "in principle, yes" which sort of questions why I put all that work to rebuild something which apparently already exists. But the truth is that there's a huge gap between "in principle" and "in reality", and I'd like to spell this difference out in this post.

The bottom line is that all those pieces of Big Data infrastructure which exists today provide you with a lot of pretty impressive functionality, distributed storage, scalable computing, resilience, and so on, but not in a way which solves your data analysis problems out of the box. The analogy I like is that Big Data is a lot like providing you with an engine, a transmission, some tires, a gearbox, and so on, but no car.

So let us consider an example where you have some clickstream and you want to extract some information about your users. Think, for example, recommendation, or churn prediction. So what steps are actually involved in putting together such a system?

First comes the hardware, either on the cloud or by buying or finding some spare machines, and then setting up the basic infrastructure. Nowadays, this would mean installing Linux, HDFS, the distributed filesystem of Hadoop, and YARN, the resource manager which allows you to run different kind of compute jobs on the cluster. Especially when you go for the raw Open Source version of Hadoop, this step requires a lot of manual configuration, and unless you already did this a few times, this might take a while to get to work.

Then, you need to take in the data in some way, for example, by something like Apache Kafka, which is essentially a mixture of a distributed log storage and an event transport plattform.

Next, you need to process the data, which could either be done by a system like Apache Storm, a stream processing framework which lets you distribute computing once you have it broken down to pieces of computation taking in an event at a time. Or you use Apache Spark which let's you describe computation on a higher level with something like a functional collection API and can also be fed a stream of data.

Unfortunately, this still does nothing useful out of the box. Both Storm and Spark are just frameworks for distributed computing, meaning that they allow you to scale computation, but you need to tell them what you want to compute. So you first need to figure out what to do with your data and this involves looking at data, identifying the kind of statistical analysis which is suited to solve your problem, and so on, and probably requires a skilled data scientist to spend one to two month working on the data. There are projects like mllib which provide more advanced analytics, but again these projects don't provide full solutions to application problems but are tools for a data scientist to work with (And they are still somewhat early stage IMHO.)

Still, there's more work to do. One thing people are often unaware of is that Storm and Spark have no storage layer. This means that they both perform computation, but to get to the result of the computation, you have to store it somewhere and have some means to query it. This means usually to store the result in a database, something like redis, if you want the speed of a memory based data storage, or in some other way.

So by now we have taken care of how to get the data in, what to do with it and how, and how to store the result such that we can query it while the computation is going on. Conservatively talking, we're already down six man months, probably less if you have done it before and/or are lucky. Finally, you also need to have some way to visualize the results, or if your main access is via an API, to monitor what the system is doing. For this, more coding is required, to create a web backend with graphs written in d3.js in JavaScript.

The resulting system probably looks a bit like this.

Lots of moving parts which need to be deployed and maintained. Contrast this with an integrated solution. To me this is difference between a bunch of parts and a car.