Los Techies

Hugo 2026-09-28T18:50:16Z https://lostechies.com/ Los Techies ACP Muse Agent Client Protocol and Muse Code 2026-09-28T12:38:15+02:00 2026-09-28T12:38:15+02:00 https://lostechies.com/ryansvihla/2026/09/28/acp-muse-agent-client-protocol-and-muse-code/ At <a href="https://slopcop.com/">work</a> we tend to try every new model and vendor we can so we know what the state of the art is. Enter <a href="https://dev.meta.ai/docs/muse-code">Muse Code</a>, the models are fast, cheap and pretty capable. For $50 a month you get more credits than I can spend, and for API usage there is a 92-95% discount letting them train on your data. TLDR this is a ton of value you get out of the models from <a href="https://about.meta.com/">Meta</a>. So I went ahead and created <a href="https://github.com/BrokkAi/muse-acp">Muse ACP</a> and my employer was nice enough to let me work on it at work time and now it is a part of several of our projects. <h2 id="what-is-acp">What is ACP?</h2> <a href="https://www.agentclientprotocol.com">Agent Client Protocol</a> is an interaction standard to that one can wire up ACP clients (TUI, Text Editor, Batch Processes, Personal Assistants) to an AI agent that has implemented the same protocol. Many Agents provide this out of the box such as <a href="https://github.com/NousResearch/hermes-agent">Hermes</a> via <code>hermes acp</code>, <a href="https://opencode.ai">Opencode</a> via <code>opencode acp</code>, you get the pattern, however as of yet Muse Code does not ship one, so I made one. Because of this one can easily hookup Hermes, Opencode or any of the other dozens of clients in the <a href="https://agentclientprotocol.com/get-started/registry">registry</a> to <a href="https://www.jetbrains.com/idea/">IntelliJ</a>, <a href="https://zed.dev/">Zed</a>, or any number of the clients I have written like <a href="https://github.com/BrokkAi/micro-acp">Micro ACP</a>, <a href="https://github.com/foundev/belgr">Belgr</a>, or <a href="https://github.com/BrokkAi/mjolnir">Mjolnir</a>. Now there is a registry, but I have personally found it a real and total pain to get into, so I have chosen to ignore it for Muse ACP until I get enough popularity that someone else adds my ACP server to it (10 stars and counting). It is also easy to add custom clients and I have add an installer for IntelliJ and Zed with Muse ACP (muse-acp install) so that you can try this out with your favorite editor. <h2 id="why-muse-acp-and-not-one-of-the-others">Why Muse ACP and not one of the others</h2> When I started the project there were no mature ACP servers using <a href="https://meta-models.github.io/muse-code-sdk/next/guides/msp-concepts/">Muse Session Protocol</a> as the integration point. Also, it is written in Rust instead of Typescript and that is for some people a good reason to prefer the BrokkAi version of the rest. <h2 id="conclusion">Conclusion</h2> If you want access to a cheap capable model, go ahead and sign up for a Muse Code subscription plan and give Muse ACP a try, wire it up to your favorite editor or one of mine and get to work. Why not use DeepSeek Flash for everything? 2026-09-22T09:18:47Z 2026-09-22T09:18:47Z https://lostechies.com/ryansvihla/2026/09/22/why-not-use-deepseek-flash-for-everything/ I started using AI heavily when a friend <a href="https://www.linkedin.com/in/jbellis/">Jonathan Ellis</a> launched his own startup <a href="https://brokk.ai">Brokk</a> and I started using his then revolutionary tool <a href="https://github.com/BrokkAi/brokk-app">Brokk</a> in April of 2025, at that stage basically all models were pretty terrible and things like Cursor or Claude Code were basically useless (I like to call them tooling harnesses because they hook up tools like bash, read, write, internet search to models&hellip; and that is most of what they do), so this meant we had to use the latest and greatest to ever get anything done. Brokk despite being basically unusable for a mere mortal (lesson let a designer build the UX), it was light years ahead of Claude Code and Cursor and with the top end model we could get a ton of solid generated code out of things like GPT 4 and Sonnet 3.5. They routinely invented stuff, didn&rsquo;t know how to look into directories etc, what we knew about prompting then was limited and in general getting good work out of them required a lot of cleverness or at least a willingness to accept really terrible code and a drop in productivity (at least in single threaded modes of work). At least with Brokk with the tools we had built and the efficiencies we engaged in to keep the context small (the total amount of information sent to the models) we could make good use of cheaper less high end models, but generally speaking the lesson was clear, if you could afford it always use the best, or else waste a lot more time and money getting bad results out of the cheaper stuff. Fast forward to today and even cheap models can score well on difficult coding benchmarks (source: <a href="https://deepswe.datacurve.ai/">https://deepswe.datacurve.ai/</a>) and the results are way more about &lsquo;feel&rsquo; than they are about any sort of qualitative result that one can measure easily. <a href="/assets/why-not-use-deepseek-flash-for-everything/deepswe-snapshot.png"><img src="/assets/why-not-use-deepseek-flash-for-everything/deepswe-snapshot.png" alt="DeepSWE v1.1 leaderboard comparing model scores with average cost per task."></a> DeepSWE v1.1 score versus average cost per task, September 22, 2026 (<a href="https://deepswe.datacurve.ai/">source</a>). Click the chart to view it at full size. This is because of a few things at once: the tooling harnesses of today all do the things Brokk was doing in 2025 and often with more quality and efficiency, the <a href="https://arxiv.org/abs/2501.12948">advances</a> <a href="https://aclanthology.org/2026.findings-acl.1767/">in</a> <a href="https://arxiv.org/abs/2607.22529">reasoning</a>, the advances in <a href="https://arxiv.org/abs/2503.14476">post training</a>, and <a href="https://arxiv.org/abs/2505.09388">distillation becoming</a> refined to the point that <a href="https://www.deepseek.com/en/news/deepseek-v4-1-flash/">DeepSeek Flash 4.1</a> which is a model that now beats Sol on our internal benchmarks and is a fraction of the estimated size of Sol. <a href="/assets/why-not-use-deepseek-flash-for-everything/deepseek-benchmarks.png"><img src="/assets/why-not-use-deepseek-flash-for-everything/deepseek-benchmarks.png" alt="Benchmark table comparing DeepSeek V4.1-Flash with DeepSeek V4-Pro and V4-Flash, GLM 5.3, Kimi K3, GPT 5.6-Sol, and Claude Opus 5."></a> DeepSeek V4.1-Flash benchmark comparison (<a href="https://www.deepseek.com/en/news/deepseek-v4-1-flash/">source</a>). Click the table to view it at full size. For my own experience I can tell you that: <ul> <li>Fable 5.1 (Anthropic) and Astra (OpenAI) are great general purpose models and I have used them a lot, but for many tasks they vastly overengineer a solution.</li> <li>Sol (OpenAI) is overall a very good value but it also can overengineer on higher thinking levels and is sort of dumb on its default level (it works but you just need to prompt it a lot more). I do not think this is a bad choice honestly as your model for everything and between it and Luna really justifies the cheaper price.</li> <li>GLM 5.3 (Z.ai) is very capable and has been my go-to for about a month now.</li> <li>GLM 5.3 Flash (Z.ai), DeepSeek Flash v4 are both very cheap to use and for a large variety of straightforward coding or admin tasks are easily good enough.</li> <li>Recently I have been using DeepSeek Flash V4.1 and while more expensive to use than Flash v4.0 it has largely replaced most of my model usage, and next month I do not plan on renewing my Anthropic, Z.ai or OpenAI subs at the maximum level as a result.</li> </ul> <h2 id="conclusion">Conclusion</h2> I do not actually care or think it is super important which model you use anymore, use the model that fits your style and you can afford and after that really compared to any other point prior we get very good results from nearly any provider. Use Muse, use OpenAI, use Z.ai you should be basing these differences now based on who treats you well, respects your data or <a href="https://blog.ferstar.org/en/posts/zcode-silent-workspace-snapshot-upload">does</a> <a href="https://thehackernews.com/2026/07/grok-build-uploads-entire-git.html">not</a>. For some gnarly hard problems I will still probably reach a lot for Astra and Fable, but I doubt I will be on a 20x Max sub anymore. Ryan Svihla Using Opus 5 Without Going Insane 2026-09-18T14:00:00Z 2026-09-18T14:00:00Z https://lostechies.com/ryansvihla/2026/09/18/using-opus-5-without-going-insane/ I&rsquo;ve been using Opus 5 as my daily driver for a few weeks now across a bunch of projects and it&rsquo;s a really capable model, but it has two habits that drove me a little nuts before I figured out how to deal with them: <ol> <li>It is verbose in a way that actually hides the answer. You ask a yes or no question and get four paragraphs of context, a code snippet, a caveat, and then the answer somewhere in the middle.</li> <li>When it doesn&rsquo;t understand something it does not go look it up, it guesses, and then it builds a lot of confident sounding structure on top of the guess.</li> </ol> This isn&rsquo;t a post about whether Opus 5 is good or not (it is), it&rsquo;s strictly what I do day to day to keep it useful. None of this is clever, most of it is just repeating yourself. <h2 id="start-with-a-claudemd">Start with a CLAUDE.md</h2> The first thing I did was put a global CLAUDE.md in place (~/.claude/CLAUDE.md) that tries to head off the verbosity before it starts. This is the current version, warts and all: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-markdown" data-lang="markdown"># Global instructions ## Be brief - Answer the question asked, then stop. No preamble, no recap, no summary of what you just did. - Default to a few sentences. Drop headers, bold, and bullet lists unless the content is genuinely a list. - Do not restate the user&#39;s question back to them. - Do not offer next steps unless asked. - No trailing coda. Do not end with a &#34;one thing I didn&#39;t do&#34; / &#34;note that&#34; / &#34;caveat&#34; paragraph. If a limitation matters, state it inline where it&#39;s relevant, in one clause. If it doesn&#39;t matter, leave it out. - Banned openers for a closing paragraph, no exceptions: &#34;One thing worth deciding&#34;, &#34;One thing worth noting&#34;, &#34;Worth noting&#34;, &#34;Worth flagging&#34;, &#34;One caveat&#34;, &#34;Separately&#34;, &#34;For what it&#39;s worth&#34;. If you catch yourself starting a final paragraph with any of these, delete the paragraph. Do not rephrase it to evade the list -- the ban is on appending a postscript the user did not ask for, not on those specific words. - This applies to unsolicited future-work suggestions too. If the answer is done, stop at the answer. ## Never claim credit - Never say you already had an idea, thought of it first, suggested it earlier, or were about to do it. - If the user proposes something, implement it. Do not annotate it with your prior reasoning. - Never describe your own earlier messages as having been right. If a correction is needed, state the fact, not who found it. - Do not narrate your process or reasoning quality, positively or negatively. ## Verify before asserting - Never state how code behaves without reading it. &#34;It&#39;s not in this repo&#34; is not a stopping point — find the source and read it. - Never invent a critique, a self-criticism, or an account of what happened earlier. If you are describing the past, quote it or do not claim it. </code></pre></div>A few notes on why it looks like that: <ul> <li>The &ldquo;banned openers&rdquo; list is there because I kept getting a trailing &ldquo;One thing worth noting&hellip;&rdquo; paragraph on literally every response, and telling it not to add caveats just made it rephrase the caveat. Listing the exact phrases and saying don&rsquo;t rephrase to evade the list was what finally worked (mostly).</li> <li>&ldquo;Never claim credit&rdquo; is there because it has a habit of annotating your idea with &ldquo;as I suggested earlier&rdquo; which is both annoying and frequently not true.</li> <li>&ldquo;Verify before asserting&rdquo; is the docs point below in a different form. It will happily tell you how some library behaves without ever having opened the library.</li> </ul> This gets you maybe 70% of the way there. The rest you do it live. <img src="/content/ryansvihla/uploads/2026/09/do-it-live.gif" alt="We&rsquo;ll do it live"> <h2 id="simplify-simplify-simplify">Simplify, simplify, simplify</h2> Even with the CLAUDE.md in place it&rsquo;ll drift back into long confusing answers, especially when the question has any nuance to it. If I do not understand what it is saying, or there are too many conflicting items in the response, I tell it to simplify and repeat until it is down to the level of directness I actually need. Some real prompts from my history: I asked why some tests were Unix only and got this: <blockquote> Because the worker doesn&rsquo;t run on Windows at all. The worker is the process that actually hosts an agent session, and its whole runtime is <code>#[cfg(unix)]</code> — the sub-agent socket is just one piece of it. The code says so directly, at <code>mj-worker/src/mcp_stdio.rs:297</code>: <blockquote> Workers run on Unix; the servers compile everywhere so the CLI stays one shape, and say plainly where they cannot run. </blockquote> That&rsquo;s the design: Windows builds the same CLI so the commands are identical, and the worker-side functions are stubs that return a clear error instead of silently missing. The transport itself is a Unix domain socket via <code>std::os::unix::net</code>, which Rust&rsquo;s standard library only provides on Unix. I didn&rsquo;t make that call — it predates this work. My change only lined up two test helpers with the gate their subject already had. </blockquote> This giant wall of text seemed just unnecessary to parse, it had a lot repetition, and some uncessary data, qualifications (&ldquo;That&rsquo;s the design&rdquo;, &ldquo;the transport itself&rdquo;, &ldquo;the servers compile everywhere&rdquo;) so I just said <code>simplify</code> and got this: <blockquote> Mjolnir&rsquo;s worker only runs on Unix by design. Windows gets the CLI, not the worker. So sub-agent code on Windows is stubs that return &ldquo;needs a Unix platform.&rdquo; </blockquote> WAY more parsable without a lot of garbage, I took this in and saved myself some Ulysses level parsing of text. Here is another one, this time I asked if we could use a different approach to read usage numbers on Mac: <blockquote> No — <code>claude -p /usage</code> has no parser here. <code>successful_refresh_output</code> (<code>mj-controller/src/claude_usage.rs:197</code>) only checks the exit status; nothing reads its stdout, and the module header already calls print-mode usage output approximate. Switching mac to it means writing a text parser for an unstable format and getting worse numbers. The gap isn&rsquo;t the API, it&rsquo;s where the token comes from. <code>read_credentials</code> only reads <code>&lt;home&gt;/.credentials.json</code>. On macOS the same <code>{&quot;claudeAiOauth&quot;: {...}}</code> JSON is in the Keychain under <code>Claude Code-credentials</code> — <code>setup.rs:451</code> already reads it with <code>security find-generic-password -s &quot;Claude Code-credentials&quot; -w</code>, and its comment notes that <code>security</code> is pre-authorized for the item so it raises no prompt. That blob feeds <code>oauth_access_token</code> unchanged. There&rsquo;s a third source too: <code>mj login --setup-token</code> stores a long-lived token at <code>claude_oauth_token_path(profile_id)</code>, which sessions use via <code>apply_claude_setup_token</code> but quota never reads. So the fix I&rsquo;d make is a source chain for the usage API — credentials file, then the stored setup token, then the macOS Keychain — leaving <code>claude -p /usage</code> doing only what it does now, forcing a refresh. Want me to implement that? </blockquote> Again a lot of noise, uncessary information and just a lot to parse for a simple question, instead of making my head hurt figuring out if it actually answered my question, I just said <code>simpler too much fluff</code>: <blockquote> No. Nothing parses that output — <code>successful_refresh_output</code> only checks the exit code. On mac the token is in the Keychain in the same JSON shape the API path already parses. Fall back to <code>security find-generic-password -s &quot;Claude Code-credentials&quot; -w</code> when <code>.credentials.json</code> is missing. Want that? </blockquote> I will be honest I did more rounds after this. The short answer was still wrong, but at least I was sure now. Some notes: <ul> <li>It usually takes two or three rounds.</li> <li>The shorter answer is almost always the same answer with less around it, so you are not losing anything.</li> <li>If it is still confusing after a few rounds it is probably not a verbosity problem, it does not understand the question. In that case I give it the structure I want (a numbered list, a table, an <code>a -&gt; b -&gt; c</code> chain) and have it fill that in rather than letting it pick how to lay the answer out.</li> </ul> <h2 id="make-it-read-the-docs">Make it read the docs</h2> The other thing is it will implement against a library or an API from memory, and its memory is often out of date or just wrong for anything outside the popular parts of the API. It does not go look unless you tell it to, so I always tell it to. Some real prompts: <pre tabindex="0"><code>read this https://&lt;docs url&gt; </code></pre><pre tabindex="0"><code>can you look at the sdk docs for &lt;X&gt; and look at the release notes for &lt;X&gt; then look at the feature set for &lt;Y&gt; and try and find any gaps in the new functionality </code></pre><pre tabindex="0"><code>you can search the web for &lt;X&gt;, you will need to do so to find the &lt;X&gt; source and docs </code></pre><pre tabindex="0"><code>I mean can you read the &lt;X&gt; sdk surely we can start the server with some flag or something </code></pre>The last one was after it told me something was not supported. It was supported, it was in the docs, it just had not read them. <h3 id="my-advice">My advice</h3> <ul> <li>Give it the actual URL, it will fetch it. If the docs are in a repo tell it to clone the repo and read the source.</li> <li>When it says something is not possible ask if it read the docs first.</li> <li>The &ldquo;Verify before asserting&rdquo; section in the CLAUDE.md above helps some but does not replace saying it in the prompt.</li> </ul> <h2 id="summary">Summary</h2> That is really it. Tell it to simplify until you can read the answer, and tell it to read the docs before it writes code against them. Boring but simple, and then the model is genuinely very solid and will do often do proper implementatios. Ryan Svihla Big Design Up Front Is Responsible Again 2026-07-31T09:00:00Z 2026-07-31T09:00:00Z https://lostechies.com/erichexter/2026/07/31/big-design-up-front-is-responsible-again/ Agile taught us to avoid big design up front. That was the right call — for the constraints we had. Those constraints are gone, and I think the advice inverted. Here&rsquo;s why we avoided it. Two things were slow and expensive: writing the spec, and building the software. Both were done by hand. So a detailed upfront design was a large bet you couldn&rsquo;t easily change, written before you knew if it was right. The sane response was to write less of it, ship a thin slice, and learn by building the wrong thing first. For twenty years that was the responsible move. I&rsquo;ve shipped plenty of thin slices that discovered the wrong product — expensively — and called it agility. AI deleted both of those costs. Last week I designed a whole application before writing a line of it. Not a napkin sketch — a complete, traceable spec: stakeholder needs, system requirements, architecture with sequence diagrams, down to component-level tickets. Then it was rendered into a narrated video walkthrough with real UI screens, and I handed that to the customer to review. Pause, draw on a frame, leave a voice note. <img src="/content/erichexter/uploads/2026/07/design-review-frame.png" alt="A design review, rendered: a real UI screen with narration, reviewed before any code was written" style="max-width:100%" /> Writing that spec used to take weeks of typing. It took hours. Building a throwaway version just to get real feedback used to be the only option. Now I get feedback on the design itself, at full fidelity, before building anything. The part that matters: this is not waterfall. Waterfall failed because the big upfront design was unvalidated and expensive to change — by the time reality showed up, you were committed. Flip both of those. The design is now cheap to produce, cheap to change, and cheap to validate: render a new video, get feedback, re-render. That is big design up front with a tight feedback loop — the thing waterfall couldn&rsquo;t afford and agile gave up on. Put it in a grid. Waterfall: expensive to produce, expensive to change. Agile: so skip the design. What we have now: cheap to produce, cheap to change, cheap to validate. Different quadrant. Different rules. <img src="/content/erichexter/uploads/2026/07/design-screen-set.png" alt="A full set of designed screens generated for review before implementation" style="max-width:100%" /> So where did the bottleneck go? It was never typing. It was typing speed — for specs and for code — that forced us to go slow and iterate. The model writes specs at the speed of light and builds at the speed of light. The only scarce thing left is judgment: deciding what&rsquo;s right, and getting the humans aligned before you commit. Spend your time there. Do the design work you&rsquo;ve been skipping since 2005 — you can finally afford it. Refine it. Review it properly, with people who aren&rsquo;t going to read a forty-page doc but will watch a five-minute video and tell you it&rsquo;s wrong. Then let the model build it. Concretely, the loop I&rsquo;m running now: <ol> <li>A living spec that deepens by level — stakeholder need, system, architecture, components — each traceable to the next.</li> <li>Every level rendered into a narrated review video, approved before the design goes deeper.</li> <li>The final design compiles to tickets.</li> <li>The model implements; you verify against the spec you already agreed on.</li> </ol> The responsible move in 2026 isn&rsquo;t &ldquo;move fast and iterate.&rdquo; It&rsquo;s get it right, then let the machine move fast. Eric Hexter 23 Models, One Weekend, Final Picks 2026-06-06T12:00:00Z 2026-06-06T12:00:00Z https://lostechies.com/erichexter/2026/06/06/local-llm-bench-part-5-final-picks/ Part 5 of 5 in the <a href="/erichexter/2026/05/25/local-llm-bench-part-1-which-models-can-chat/">Local LLM Bench series</a>. The project started with ten models and two prompts. It ended with 23 models, a 13-point scoring harness, 3 Python agentic tasks, and more surprises per hour than I expected. This is the final leaderboard and the honest verdict. <h2 id="expanding-to-23-models">Expanding to 23 Models</h2> After the initial ten-model run, I pulled thirteen more based on a mix of research agent recommendations and community signals. The research was right about some things and wrong about others. It correctly killed two obvious traps. qwen2.5vl is a vision model, not a coder — the &ldquo;vl&rdquo; should have been the clue but I wanted confirmation. qwen3.5:27b is a thinking model that burns its token budget on internal reasoning before producing output; on 16GB VRAM with a standard context budget it hits the wall and times out on every agentic task. Both of those were correct calls. Then there was cogito:14b. The research said: skip it, superseded, runs 2-3 points behind qwen2.5. I almost listened. What actually happened when I ran cogito: 11-second code generation on the fizzbuzz task, 100/100 agentic score, both edit formats working cleanly. The research was wrong. Cogito turned out to be the fastest sweet-spot model I tested, and it passed tasks that models with higher single-shot scores failed entirely. Two tag hallucinations also surfaced during pulls. qwen3.5:9b doesn&rsquo;t exist — only the 27B is available. qwen3-vl:8b doesn&rsquo;t exist — only the 235B is available. The research had the right model families but invented specific version tags. The fix is always the same: check ollama.com/library before pulling. Don&rsquo;t trust a model recommendation that includes a specific tag without verifying. <h2 id="the-pi-harness-experiment">The Pi Harness Experiment</h2> Alongside the expanded model pool, I tested a different agentic harness entirely. Pi is fundamentally different from aider: instead of receiving structured edit instructions, the model gets direct Bash tool access and can run <code>dotnet new</code>, <code>dotnet test</code>, and anything else itself. It operates as an autonomous loop rather than a guided editor. I ran devstral and qwen3-coder through pi on two tasks: fizzbuzz-plus and csv-parser. Both timed out at 1020 seconds. Not close calls — full exhaustion, zero useful output across both models and both tasks. The root cause is that pi is designed for models fine-tuned for tool-calling loops: NousResearch Hermes-class, OpenClaw, models explicitly trained to keep calling tools autonomously and self-terminate when done. Devstral and qwen3-coder via Ollama&rsquo;s OpenAI-compat API don&rsquo;t have that fine-tuning. They can use tools when prompted, but they don&rsquo;t have the trained instinct to keep invoking tools in sequence until a test passes. The thing pi taught me even while failing: harness design is not neutral. An aider task prompt and a pi task prompt are different programs. The model receives different inputs, operates under different constraints, and requires different trained behaviors to succeed. A 100/100 aider score does not predict pi performance, and vice versa. If a Hermes-class model shows up in Ollama&rsquo;s library with solid benchmark numbers, pi is worth revisiting. Until then, aider is the right tool for local 14-30B models. <h2 id="the-scoring-expansion">The Scoring Expansion</h2> The harness also grew. I extended the single-shot tests from 10 to 13 points by adding three new probes: a math word problem (3 apples at $0.50 plus 4 oranges at $0.75, reply with only the dollar amount), a JSON output test (return a JSON array of 3 programming languages, nothing else), and a sequence test (output 1 through 5, one per line, nothing else). These three tests turned out to be more discriminating than I expected. Ten of twenty-three models fail the $4.50 math test — not because they get the arithmetic wrong, but because they reason aloud about the problem instead of answering it. The sequence test catches models that follow instructions in general but can&rsquo;t suppress the urge to add a brief explanation. The JSON test catches models that can&rsquo;t stop themselves from wrapping output in markdown fences when explicitly told not to. None of these tests are hard. All of them reveal something real about how a model behaves when you need it to produce structured output on command. <h2 id="three-new-python-agentic-tasks">Three New Python Agentic Tasks</h2> The agentic suite expanded to include three Python tasks alongside the existing C# work. The tasks: a markdown-to-html converter (implement <code>md_to_html()</code>, 10 pytest tests covering headers, bold, italic, inline code, and links), a JSON validator (implement <code>validate(data, schema)</code> returning error strings, 9 pytest cases covering required fields, type checking, and enum validation), and a word-frequency counter (implement <code>top_words(text, n)</code> returning top-N tuples sorted by count descending then alphabetically, 8 pytest cases). I ran these on seven models: devstral, qwen3-coder, phi4, hermes3, qwen2.5-coder, mistral-small3.2, and codestral. The results reshuffled the leaderboard in ways the single-shot scores did not predict. <h2 id="the-full-leaderboard">The Full Leaderboard</h2> <table> <thead> <tr> <th>Model</th> <th>Size</th> <th>SS /13</th> <th>Chat ms</th> <th>Code ms</th> <th>Agentic Best</th> <th>Agentic Pass%</th> </tr> </thead> <tbody> <tr> <td>gemma4:latest</td> <td>~12B</td> <td>12/13</td> <td>6,918</td> <td>603</td> <td>20/100</td> <td>0% (0/2)</td> </tr> <tr> <td>devstral:latest</td> <td>~24B</td> <td>11/13</td> <td>16,875</td> <td>3,246</td> <td>100/100</td> <td>83% (5/6)</td> </tr> <tr> <td>gemma4:26b</td> <td>26B</td> <td>11/13</td> <td>11,029</td> <td>3,255</td> <td>20/100</td> <td>0% (0/2)</td> </tr> <tr> <td>qwen3.5:27b</td> <td>27B</td> <td>11/13</td> <td>24,810</td> <td>7,222</td> <td>20/100</td> <td>0% (timeout)</td> </tr> <tr> <td>deepseek-r1:14b</td> <td>14B</td> <td>10/13</td> <td>6,286</td> <td>561</td> <td>—</td> <td>—</td> </tr> <tr> <td>glm-4.7-flash</td> <td>30B MoE</td> <td>10/13</td> <td>8,843</td> <td>2,531</td> <td>20/100</td> <td>0% (timeout)</td> </tr> <tr> <td>granite4:32b-a9b-h</td> <td>32B MoE</td> <td>10/13</td> <td>20,885</td> <td>3,125</td> <td>20/100</td> <td>0%</td> </tr> <tr> <td>qwen2.5:14b</td> <td>14B</td> <td>10/13</td> <td>6,221</td> <td>475</td> <td>10/100</td> <td>0%</td> </tr> <tr> <td>qwen2.5vl:7b</td> <td>7B</td> <td>10/13</td> <td>5,783</td> <td>863</td> <td>—</td> <td>—</td> </tr> <tr> <td>qwen3-coder:30b</td> <td>30B</td> <td>10/13</td> <td>9,948</td> <td>2,143</td> <td>100/100</td> <td>67% (4/6)</td> </tr> <tr> <td>qwen3:14b</td> <td>14B</td> <td>10/13</td> <td>3,876</td> <td>523</td> <td>20/100</td> <td>0%</td> </tr> <tr> <td>cogito:14b</td> <td>14B</td> <td>9/13</td> <td>6,447</td> <td>438</td> <td>—</td> <td>—</td> </tr> <tr> <td>hermes3:latest</td> <td>~8B</td> <td>9/13</td> <td>3,756</td> <td>280</td> <td>100/100</td> <td>40% (2/5)</td> </tr> <tr> <td>mistral-small3.2:24b</td> <td>24B</td> <td>9/13</td> <td>12,169</td> <td>3,228</td> <td>100/100</td> <td>100% (3/3)</td> </tr> <tr> <td>mistral:latest</td> <td>7B</td> <td>8/13</td> <td>3,335</td> <td>323</td> <td>20/100</td> <td>0%</td> </tr> <tr> <td>codestral:22b</td> <td>22B</td> <td>7/13</td> <td>17,182</td> <td>2,427</td> <td>100/100</td> <td>67% (2/3)</td> </tr> <tr> <td>deepseek-coder-v2:16b</td> <td>16B</td> <td>7/13</td> <td>6,516</td> <td>298</td> <td>—</td> <td>—</td> </tr> <tr> <td>llava:7b</td> <td>7B</td> <td>7/13</td> <td>4,045</td> <td>292</td> <td>—</td> <td>—</td> </tr> <tr> <td>magistral:24b</td> <td>24B</td> <td>7/13</td> <td>22,802</td> <td>11,568</td> <td>—</td> <td>—</td> </tr> <tr> <td>gpt-oss:20b</td> <td>20B</td> <td>6/13</td> <td>8,751</td> <td>8,915</td> <td>20/100</td> <td>0%</td> </tr> <tr> <td>phi4:14b</td> <td>14B</td> <td>6/13</td> <td>6,415</td> <td>466</td> <td>100/100</td> <td>50% (2/4)</td> </tr> <tr> <td>qwen2.5-coder:14b</td> <td>14B</td> <td>6/13</td> <td>5,989</td> <td>529</td> <td>100/100</td> <td>100% (4/4)</td> </tr> <tr> <td>qwen3:30b</td> <td>30B</td> <td>3/13</td> <td>14,866</td> <td>10,749</td> <td>—</td> <td>—</td> </tr> </tbody> </table> qwen3.5:27b, gpt-oss:20b, and deepseek-r1:14b are thinking models — they burn context on internal reasoning before producing visible output. The scores reflect that. <h2 id="the-surprising-results">The Surprising Results</h2> gemma4:latest. 12/13 single-shot, 603ms code generation, fastest chat in its size class. Zero percent agentic pass rate across every task it attempted. This is the sharpest split in the entire dataset. gemma4 is an excellent model for answering questions. It has no working mental model of &ldquo;I am in a multi-turn loop writing files until tests pass.&rdquo; Those are different capabilities. The single-shot tests reward the former. The agentic tasks require the latter. gemma4 nails one and is completely useless at the other. mistral-small3.2:24b. I almost missed this one entirely. It had no agentic run history going into the final Python task batch — it just hadn&rsquo;t come up in earlier experiments. When I finally ran it, it swept all three new Python tasks with 100/100 scores on first attempt, finishing each in 26 to 52 seconds. Nine out of 13 on single-shot. It had minimal community attention during the bench period. It turned out to be one of the two most reliable agentic performers I tested. The lesson here: community signal is a useful prior, not a substitute for running the test. qwen2.5-coder:14b. 6/13 on single-shot. That score is a lie in the specific direction that matters most. The instruction-following tests fail consistently. The code generation test produces output that compiles but gets the wrong answer. On every agentic task I ran it on, it passed. Four for four, 100% pass rate. The single-shot harness penalizes its tendency to reason aloud before writing code. In an agentic loop, that verbosity doesn&rsquo;t hurt — aider just waits for the edit block, and the edit block is correct. Single-shot actively mispredicts this model&rsquo;s real-world utility. hermes3:latest. 280ms code generation. The fastest model in the field by a significant margin, and at 4.7GB it&rsquo;s the lightest serious option. 3,756ms average chat latency, also fastest. It scored 100/100 on csv-scaffolded with a 25-second wall time — another field record. Then it scored 10/100 on fizzbuzz and instant-failed on json-validator in zero turns. The inconsistency pattern makes sense for a model fine-tuned specifically for tool use and short completions: it handles the tasks that match its training profile well and falls apart outside them. For anyone doing rapid-fire chat or simple completions at scale, hermes3 is the answer. For general agentic coding, the brittleness is a real problem. phi4:14b. 6/13 on single-shot; 100/100 on fizzbuzz and word-freq. It failed markdown-to-html and json-validator, and both failures have the same signature: 16 to 17% context utilization, then the output starts spiraling. phi4 has a 16K context ceiling, and tasks that grow their working context over multiple iterations hit that wall. The context limit is the only thing preventing phi4 from joining the reliable agentic tier. With 32K context or better, I&rsquo;d expect it to pass everything it currently fails. codestral:22b. The markdown-to-html task produced a unicode crash — aider&rsquo;s display layer choked on an arrow character in a CP1252 terminal. json-validator and word-freq both passed 100/100. That markdown failure is an environment bug, not a model failure. I&rsquo;m counting it in the pass rate because I can&rsquo;t retroactively change the environment it ran in, but anyone testing codestral in a UTF-8 terminal should expect a different result. <h2 id="the-actual-picks">The Actual Picks</h2> For coding work on a 16GB machine, the answer depends on what you&rsquo;re doing. If you&rsquo;re working in a new codebase — multi-file, complex scaffolding, scratch-to-working-tests — use devstral:latest. It&rsquo;s the only model in this pool that reliably handles multi-file C# from scratch. 83% agentic pass rate across six diverse tasks spanning C# and Python. Not the fastest at 3 to 20 seconds per response, but it has the highest ceiling and it doesn&rsquo;t fall apart on complexity. If you&rsquo;re working in an existing codebase — the actual everyday case, where you&rsquo;re editing files that already exist — use qwen3-coder:30b. 100/100 on Python tasks, strong on scaffolded C#, 2-second code generation. The whole edit format is mandatory; diff mode fails silently and produces nothing. Get the format right and this model is very fast for its size. If VRAM is the constraint, use qwen2.5-coder:14b. It runs on about 9GB, which means it fits alongside other processes. It passed every agentic task I ran it on. The 6/13 single-shot score is misleading — ignore it for agentic work. mistral-small3.2:24b is on a watch list. Three tasks run, three passed. That&rsquo;s not enough data to promote it above devstral for serious work, but it&rsquo;s enough to keep it in the rotation. If it holds 100% across ten more tasks I&rsquo;ll move it up. For chat and Q&amp;A, the picks are different. gemma4:latest for quality — 12/13, fast for its size, clean outputs. Don&rsquo;t use it for anything agentic. For speed, hermes3:latest at 4.7GB and 280ms code generation is the answer, especially if you&rsquo;re running it alongside something else or doing high-volume completions. <h2 id="what-single-shot-scores-actually-measure">What Single-Shot Scores Actually Measure</h2> This question came up enough during the project that it deserves a direct answer. Single-shot scores measure whether a model understands what it&rsquo;s being asked, can produce a well-formed response on one shot, and follows tight output constraints. That&rsquo;s genuinely useful for chatting, summarizing, classifying, and answering questions. The score is predictive for those tasks. What it does not measure: will this model keep working across turns, will it understand its own previous outputs, can it handle a tool returning an unexpected result, will it know when to stop and verify rather than spiraling, can it write files instead of prose. Those are the capabilities that determine agentic performance. They don&rsquo;t show up in any single-prompt test because by design they can&rsquo;t — they require multiple turns to observe. The practical implication is that running a 13-point single-shot harness before picking a coding model will tell you roughly nothing about whether the model can actually do the coding work. You have to run the agentic task. There is no shortcut. <h2 id="closing">Closing</h2> Six weeks. 23 models. 630 lines of harness code. 50+ agentic task runs. The answer to &ldquo;which local model can actually code?&rdquo; turns out to be a different question depending on what you mean by coding. The model that tops the single-shot leaderboard is the one to use for chat. The model that wins at agentic coding tasks is a different model entirely. I spent a weekend thinking gemma4 was the obvious answer before it timed out on every real task I gave it. The bench application and all results are at <a href="https://github.com/erichexter/ollama-model-bench">github.com/erichexter/ollama-model-bench</a>. The harness accepts any model Ollama can serve — pull it, add an entry to the settings file, run it. The numbers here are reproducible on any machine with 16GB of VRAM. If you find something that beats devstral on multi-file from scratch, I want to know about it. Eric Hexter The Config That Changed Everything 2026-06-03T12:00:00Z 2026-06-03T12:00:00Z https://lostechies.com/erichexter/2026/06/03/local-llm-bench-part-4-harness-optimization/ Part 4 of 5 in the <a href="/erichexter/2026/05/25/local-llm-bench-part-1-which-models-can-chat/">Local LLM Bench series</a>. After Part 3&rsquo;s 1-in-6 pass rate, I had a theory about qwen3-coder. The model scored 0/100 not because it couldn&rsquo;t write C#, but because aider couldn&rsquo;t parse what it wrote. If the failure was format mismatch, then fixing the format should fix the score. I was right. One line in a YAML file took qwen3-coder:30b from 0/100 to 100/100. Twenty-six seconds. Same model, same task, same hardware. That result rewrites how I think about local model evaluation. <h2 id="the-edit_format-lever">The edit_format Lever</h2> aider supports two primary edit modes. In <code>diff</code> mode, the model sends back git-style patches — only the changed lines, with surrounding context. In <code>whole</code> mode, the model sends back the entire file contents. These are not stylistic preferences. They require completely different output from the model, and models are not equally capable of both. The research I ran before Phase 9 turned up a finding I didn&rsquo;t take seriously enough at the time: &ldquo;harness mismatch is bigger than model choice.&rdquo; One real-world study cited 6x performance variation purely from harness configuration changes, holding the model constant. I read that and thought it was probably overstated. Then I ran the A/B. The <code>.aider.model.settings.yml</code> file lets you configure per-model settings. The critical field is <code>edit_format</code>. Here&rsquo;s what qwen3-coder&rsquo;s entry looks like after the fix: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">- name: ollama_chat/qwen3-coder:30b edit_format: whole use_repo_map: false extra_params: num_ctx: 65536 </code></pre></div>Before this change: <code>edit_format</code> was unset, defaulting to <code>diff</code>. After: <code>whole</code>. The model behavior changes completely. <h2 id="the-ab-results">The A/B Results</h2> I ran six models against both formats on the fizzbuzz-plus sweet-spot task: <table> <thead> <tr> <th>Model</th> <th>whole</th> <th>diff</th> </tr> </thead> <tbody> <tr> <td>qwen3-coder:30b</td> <td>100/100 (26s)</td> <td>0/100 FAIL</td> </tr> <tr> <td>devstral:latest</td> <td>100/100 (53s)</td> <td>100/100 (98s)</td> </tr> <tr> <td>qwen2.5-coder:14b</td> <td>100/100 (73s)</td> <td>100/100 (65s)</td> </tr> <tr> <td>gpt-oss:20b</td> <td>20/100 FAIL</td> <td>20/100 FAIL</td> </tr> <tr> <td>qwen3:14b</td> <td>20/100 FAIL</td> <td>20/100 FAIL</td> </tr> <tr> <td>mistral:latest</td> <td>20/100 FAIL</td> <td>20/100 FAIL</td> </tr> </tbody> </table> Three models work. Three models don&rsquo;t. The format A/B cleanly separates the populations. gpt-oss, qwen3:14b, and mistral fail in both formats — those are genuine capability problems, not configuration problems. qwen3-coder was a false negative: the code was right, the format was wrong, the score said zero. devstral and qwen2.5-coder work in both formats, which tells you something about their training. They&rsquo;ve been explicitly tuned to produce structured edit blocks. qwen3-coder has not — or at least not in the diff format aider expects. Switching to whole file output removes the constraint entirely: just dump the file, let aider handle the diff computation. qwen3-coder is very good at writing complete, correct files. <h2 id="the-thinking-mode-problem">The Thinking-Mode Problem</h2> Three models that looked promising on paper — gpt-oss:20b, deepseek-r1:14b, and qwen3.5:27b — share a different failure mode. They all run in &ldquo;thinking mode&rdquo;: before producing any code output, they generate thousands of internal reasoning tokens. On single-shot tasks this is invisible; the <code>&lt;think&gt;</code> block appears in a separate field and the user only sees the final answer. On an agentic task with a 300-second timeout, the thinking block alone can exhaust the budget. gpt-oss, deepseek-r1, and qwen3.5 all timeout at zero turns — the model thought itself to death before writing a single line of code. The fix for qwen3 models (not qwen3.5, which has different training) is a <code>/no_think</code> prefix in the aider system prompt: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml">- name: ollama_chat/qwen3:14b edit_format: whole system_prompt_prefix: &#34;/no_think&#34; use_temperature: 0.7 extra_params: num_ctx: 32768 top_p: 0.8 top_k: 20 </code></pre></div>This worked for qwen3:14b and qwen3:30b. It does nothing for qwen3.5 — different model family, different training, the prefix is ignored. qwen3.5:27b is a 17GB model on 16GB VRAM, so it&rsquo;s partially spilling to RAM anyway. At mixed CPU/GPU generation speed with a thinking block running first, it cannot produce useful output inside 300 seconds. The hardware ceiling and the thinking penalty compound each other. Model eliminated. <h2 id="the-num_ctx-revelation">The num_ctx Revelation</h2> Ollama&rsquo;s default context window is 2048 tokens. That&rsquo;s not 2048 for the task — that&rsquo;s 2048 for the entire conversation, including the system prompt, the file content, the task description, and every prior exchange. For an agentic coding session where aider is sending file contents back and forth, 2048 fills in two or three turns. After that, the model is working with a truncated view of its own conversation. It starts looping, contradicting itself, or deleting code it just wrote. Ollama doesn&rsquo;t warn you when it truncates. It silently discards the oldest tokens and keeps going. The model&rsquo;s outputs start looking confused on turn three and you assume it&rsquo;s a capability problem. It isn&rsquo;t. Setting <code>num_ctx: 32768</code> (or 65536 for the larger models) unlocks stable multi-turn behavior. Several failures that looked like model confusion were actually context truncation. The fix is one line per model in the YAML. <h2 id="the-architect-mode-dead-end">The Architect Mode Dead End</h2> I wanted to test whether combining two models — one to plan, one to implement — could improve results on stretch-tier tasks. aider calls this &ldquo;architect mode.&rdquo; In principle: the architect model breaks the task into pieces, the editor model writes the code, and the combination should outperform either alone. It&rsquo;s a reasonable theory. The machine had other plans. Loading two 14-17GB models on 16GB VRAM means constant unloading and reloading. Every time control switches from architect to editor, Ollama has to evict one model and load the other. That swap is not fast. I ran devstral + qwen3-coder and devstral + qwen2.5-coder. Both pairs hit the five-minute timeout at zero turns. The entire budget went to model swap overhead before a single tool call completed. Architect mode requires both models to be co-resident in VRAM. On 16GB, that means two models totaling at most 16GB, which limits you to two 7B models — too small to be useful on complex tasks. The minimum viable VRAM for architect mode with 14B+ models is 32GB. Below that, single-model runs strictly better. <h2 id="the-scaffolding-experiment">The Scaffolding Experiment</h2> After the format A/B produced clear winners, I wanted to understand what was really limiting the failing models on the csv-parser task. The task asked models to build a C# console app and test project from scratch — which means creating <code>.csproj</code> files, a solution file, adding project references, restoring NuGet packages, and then writing correct C#. That&rsquo;s two separate problems: .NET project plumbing and C# logic. I split them apart. The scaffolded version of the task pre-creates everything: both <code>.csproj</code> files with correct <code>net10.0</code> targets, a <code>Program.cs</code> entry point the model doesn&rsquo;t touch, a stub <code>CsvProcessor.cs</code> with a TODO comment, a test project with a NuGet reference already wired, and stub test method shells. <code>dotnet restore</code> runs before the model starts. The model&rsquo;s job is to implement one static method and fill in five test bodies. <table> <thead> <tr> <th>Model</th> <th>From-scratch</th> <th>Scaffolded</th> <th>Change</th> </tr> </thead> <tbody> <tr> <td>devstral:latest</td> <td>70/100</td> <td>90/100</td> <td>+20</td> </tr> <tr> <td>qwen3-coder:30b</td> <td>0/100</td> <td>90/100</td> <td>+90</td> </tr> <tr> <td>cogito:14b</td> <td>0/100</td> <td>10/100</td> <td>+10</td> </tr> <tr> <td>granite4:32b-a9b-h</td> <td>0/100</td> <td>10/100</td> <td>+10</td> </tr> </tbody> </table> qwen3-coder was never broken. Its 0/100 on the from-scratch task was entirely a scaffolding failure. It doesn&rsquo;t know how to create a .NET solution structure from the command line — that&rsquo;s a DevOps problem, not a C# problem. Given the structure, it writes correct C# and correct tests in one shot, in 56 seconds. That&rsquo;s four times faster than devstral on the same task. cogito:14b and granite4:32b-a9b-h still fail on the scaffolded version. Their problem is C# reasoning, not project structure. The scaffolding experiment drew a clean line between the two failure modes. The practical implication: if you&rsquo;re deploying these models on an existing codebase — the actual real-world use case — the scaffolding problem doesn&rsquo;t exist. The codebase is already there. qwen3-coder becomes a genuine competitor to devstral for existing-codebase work. <h2 id="where-the-leaderboard-stands">Where the Leaderboard Stands</h2> After format configuration, context window fixes, and scaffolding experiments, the picture looks like this: For sweet-spot tasks (one or two files, existing codebase, 80-120 lines of code): qwen3-coder:30b at 26 seconds, cogito:14b at 11 seconds on both formats, devstral at 53 seconds, mistral-small3.2:24b at 44 seconds, and qwen2.5-coder:14b at 73 seconds. Five models that work reliably. For multi-file from scratch: devstral:latest, confirmed against eight challengers. No other local model in this weight class completes the csv-parser task reliably regardless of configuration. Eliminated regardless of configuration: gemma4 (all variants), glm-4.7-flash, qwen2.5:14b, qwen3:14b, qwen3.5:27b, deepseek-r1, gpt-oss, magistral — all timeout or fail in both formats. These aren&rsquo;t configuration problems. They&rsquo;re either the wrong model type (thinking models on a 16GB budget), capability gaps, or both. The 6x performance variation claim from the research turned out to be conservative in at least one case. qwen3-coder went from zero to perfect. You can&rsquo;t express that as a multiplier. <hr> Next up: <a href="/erichexter/2026/06/06/local-llm-bench-part-5-final-picks/">Part 5</a> — expanding the model pool, three surprise entries that research told me to skip, and the final leaderboard after 23 models across six weeks of testing. Eric Hexter Single-Shot Lies 2026-05-31T12:00:00Z 2026-05-31T12:00:00Z https://lostechies.com/erichexter/2026/05/31/local-llm-bench-part-3-single-shot-lies/ Part 3 of 5 in the <a href="/erichexter/2026/05/25/local-llm-bench-part-1-which-models-can-chat/">Local LLM Bench series</a>. gemma4:latest scored 10/10 on every test I built. Perfect chat response. Perfect code generation. Perfect tool call. Perfect instruction following. I ran it twice to be sure. Same result. So naturally, when it came time to run the first real agentic coding task, that was the model I reached for. It produced zero lines of useful code in ten minutes. That&rsquo;s the story of Phase 8, and it changed everything about how I think about model evaluation. <h2 id="the-task">The Task</h2> The agentic benchmark I built is a CSV parser in C#. A console app that reads a file with Name and Score columns, prints the top 3 scores descending, ties broken alphabetically. Verify with <code>dotnet test</code>. The task is sized to what I&rsquo;d call &ldquo;stretch tier&rdquo; — two projects, roughly 150 lines of code, multi-file, requires the model to scaffold a .NET solution from scratch and then implement correct logic. A competent human developer does this in about ten minutes. The harness is aider 0.86.2 installed via <code>uv tools</code>, running headless with <code>--yes-always --exit --message-file</code>. Scoring: 60 points if the verify command passes, 20 if the model finishes in two iterations or fewer, 10 for no compile errors, 10 for clean edit format. 100 points maximum. I ran six models: the top performers from Phase 4&rsquo;s single-shot benchmark plus two new additions. <h2 id="the-results">The Results</h2> <table> <thead> <tr> <th>Model</th> <th>Score</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>devstral:latest</td> <td>70/100</td> <td>5 iterations, 147 seconds</td> </tr> <tr> <td>gemma4:latest</td> <td>20/100</td> <td>600 second timeout, 0 turns</td> </tr> <tr> <td>gemma4:26b</td> <td>20/100</td> <td>600 second timeout</td> </tr> <tr> <td>glm-4.7-flash</td> <td>20/100</td> <td>600 second timeout</td> </tr> <tr> <td>qwen2.5:14b</td> <td>10/100</td> <td>91 seconds, never recovered</td> </tr> <tr> <td>qwen3-coder:30b</td> <td>0/100</td> <td>77 seconds, garbled output</td> </tr> </tbody> </table> One passes. Five fail. The model that aced every single-shot test I designed hits its ten-minute wall and produces nothing. The model that topped the leaderboard with a perfect score is the first casualty. devstral is, notably, marketed specifically for agentic coding loops. That framing turned out to matter. <h2 id="what-went-wrong-with-gemma4">What Went Wrong With gemma4</h2> gemma4:latest doesn&rsquo;t fail because it can&rsquo;t write C#. It fails because it doesn&rsquo;t understand that it&rsquo;s supposed to be writing files. When aider sends it a task, it responds with a description of what the code should look like, or it writes a fenced code block in prose, or it explains the approach in detail without producing any actual edits. I watched this happen in real time and it took longer than I&rsquo;d like to admit before I understood what I was seeing. These responses look helpful if you&rsquo;re reading them as a chat assistant. aider can&rsquo;t do anything with them — it&rsquo;s waiting for structured edit blocks that follow its protocol, not a tutorial. The single-shot benchmark rewarded exactly the behavior that makes gemma4 useless in an agentic loop. &ldquo;Write a Python function that checks if a number is prime&rdquo; — gemma4 produces clean, correct Python instantly. But that task has one shot, one context, one output. There&rsquo;s no concept of a multi-step session, no expectation that the model needs to write files into a directory, no loop where the model gets feedback and tries again. Ask gemma4 to run a ten-minute coding session and it has no mental model for what &ldquo;running a coding session&rdquo; means. It&rsquo;s a very good chat assistant. That&rsquo;s not the same thing. <h2 id="what-went-wrong-with-qwen3-coder">What Went Wrong With qwen3-coder</h2> qwen3-coder:30b scores 0/100, which looks worse than the timeout failures. It&rsquo;s actually more interesting. The model ran for 77 seconds before aider gave up, which means it produced output — just output that aider silently rejected as malformed edits. The code was probably fine. The format wasn&rsquo;t. This is a harness compatibility problem, not a capability problem. aider expects edit blocks in specific formats — either a <code>diff</code>-style patch or a <code>whole</code>-file replacement. qwen3-coder was emitting something that resembled neither cleanly enough for aider to parse. aider&rsquo;s response to a malformed edit is to silently skip it, log nothing useful, and eventually exit. From the score sheet, it looks like the model produced nothing. That&rsquo;s not what happened. This distinction matters, because it&rsquo;s a clue. If the failure is format mismatch rather than capability, changing the format instruction should fix it. I filed that away and moved on. <h2 id="what-it-means">What It Means</h2> The research literature on agentic coding benchmarks describes a roughly 17% pass rate for 14-30B parameter models on what they call &ldquo;stretch tier&rdquo; tasks: multi-file, 150+ lines of code, multiple tool-call iterations. My six-model run hit 1-in-6. Exactly 17%. That number didn&rsquo;t come from luck. It came from the same thing the research describes: most models that can answer questions well don&rsquo;t have a working mental model of &ldquo;I am operating a computer, I need to write files, I need to keep doing work until a test passes.&rdquo; Those are different cognitive tasks. Single-shot chat benchmarks don&rsquo;t distinguish between them. The models that time out aren&rsquo;t slower or dumber than devstral. They&rsquo;re not designed for this. gemma4 is optimized to produce a high-quality response to a question. devstral is optimized to take a task and not stop until it&rsquo;s done. The training objectives are different. The behavior is different. The single-shot score captures none of that. <h2 id="where-this-leaves-us">Where This Leaves Us</h2> devstral finished the task with 70/100. It needed five iterations instead of two (losing 20 points on the efficiency score), but it shipped working code. None of the other five models produced a single passing test. The 70/100 score isn&rsquo;t a ceiling — it&rsquo;s a baseline. devstral used the default aider configuration with no tuning. It worked anyway. The question is whether anything else can be made to work, or whether devstral is the only local model that can do this at all. qwen3-coder&rsquo;s format failure points toward an answer. If the problem is configuration, not capability, then changing the configuration should change the result. That&rsquo;s the experiment Part 4 runs. <hr> Next up: <a href="/erichexter/2026/06/03/local-llm-bench-part-4-harness-optimization/">Part 4</a> — one config change takes a model from 0/100 to 100/100, and the harness turns out to matter more than the model. Eric Hexter Building a .NET 10 Benchmark Harness 2026-05-28T12:00:00Z 2026-05-28T12:00:00Z https://lostechies.com/erichexter/2026/05/28/local-llm-bench-part-2-building-the-harness/ Part 2 of 5 in the <a href="/erichexter/2026/05/25/local-llm-bench-part-1-which-models-can-chat/">Local LLM Bench series</a>. The PowerShell script from part one did its job. It surfaced the think-mode problem, sorted out which models could call tools, and gave me rough latency numbers. But it could not tell me whether the code models wrote was actually correct — I was reading output and deciding it looked fine, which is not the same thing as running it. What I needed was a harness that ran models against defined tasks, verified the outputs mechanically, and produced a repeatable score. I&rsquo;m a C# developer. .NET 10 was already on the machine. The choice was not a choice. <h2 id="architecture">Architecture</h2> The project is a .NET 10 console application. The core pieces are: OllamaRunner is a thin HTTP wrapper around Ollama&rsquo;s <code>/api/generate</code> and <code>/api/chat</code> endpoints. Every request goes out with <code>temperature=0</code>, <code>seed=42</code>, and <code>think=false</code>. Temperature zero makes results deterministic enough to compare across runs. The seed locks that in further. The <code>think</code> flag is false by default — models that need it explicitly will be detected and handled. RoslynEvaluator handles the <code>SumEvens</code> code test in-process. It takes whatever the model returns, strips any markdown fences, wraps the bare method in a class, and hands it to the Roslyn CSharp scripting API to compile and execute. If it compiles and <code>SumEvens(new[] {1,2,3,4,5})</code> returns 6, the model passes. This runs entirely in memory with no disk I/O and no subprocess. TempProjectRunner is where it gets more serious. This component scaffolds actual temporary <code>dotnet</code> projects, writes model-generated code into them, builds them with <code>dotnet build</code>, and runs them with <code>dotnet run</code>. It checks stdout for the expected output. For the test suite portion, it scaffolds a second project alongside the first, adds a project reference, drops in model-generated xUnit test code, and runs <code>dotnet test</code>. Every project is cleaned up from the temp directory when the run completes. Scorer orchestrates the sequence — chat test, code test, tool test, instruction test, reasoning test, JSON output test, sequence test, Hello World test — and assembles the results into a <code>ModelResult</code> record. ModelResult is a straightforward C# record type. Every boolean metric is a property; <code>TotalScore</code> is a computed getter that sums them. The record also carries timing in milliseconds for each test category and a <code>ThinkRequired</code> flag that is informational only and does not affect the score. ConsoleReporter prints the final table to the terminal with ANSI color coding. ResultStore writes the raw results to <code>results/model-results.json</code> and a human-readable markdown ledger to <code>results/RESULTS.md</code> after each run. <h2 id="the-code-tests">The Code Tests</h2> The first code test is <code>SumEvens</code>: write a C# method that takes <code>IEnumerable&lt;int&gt;</code> and returns the sum of even numbers. Return only the method, no class, no namespace, no explanation. This is deliberately narrow. The narrow scope is the point — it is testing whether a model can follow output constraints and write code that compiles and produces correct results, not whether it can write impressive prose around the code. RoslynEvaluator wraps the method in a class, invokes it with <code>{1, 2, 3, 4, 5}</code>, and checks that the result is 6. Compile error means the model scores zero on both compile and correct. Compiles but returns the wrong number means compile point awarded, correct point denied. Compiles and returns 6 means full credit. <h2 id="hello-world-the-real-test">Hello World: The Real Test</h2> The Hello World test is where I learned something useful. The prompt asks the model to write a complete C# console application: a <code>Greeter</code> class with a public static <code>GetGreeting()</code> method that returns <code>&quot;Hello, World!&quot;</code>, plus a Main method or top-level statements that calls it and prints the result. Separately, it asks the model to write xUnit tests for that <code>Greeter</code> class. TempProjectRunner scaffolds a <code>dotnet new console</code> project, replaces <code>Program.cs</code> with whatever the model generated, runs <code>dotnet build</code>, then <code>dotnet run</code>, and checks stdout for <code>&quot;Hello, World!&quot;</code>. For the test portion, it scaffolds a <code>dotnet new xunit</code> project in the same temp directory, adds a project reference to the app, drops in the model&rsquo;s test code as <code>GreeterTests.cs</code>, runs <code>dotnet build</code>, and then <code>dotnet test</code>. This turns out to be an excellent proxy for whether a model understands C# project structure. Writing a method is straightforward. Writing a complete application that builds from scratch against a specific framework target, with a class in a form that a separately compiled test project can reference — that is a different problem. Models that understand C# project conventions get it right on the first try. Models that pattern-match on superficial features tend to include the wrong using statements, declare the class in a namespace that the test code does not account for, or produce an entry point that conflicts with the <code>Greeter</code> class definition. Each step is gated: if the app does not compile, neither the output check nor the test run happens. If the tests do not compile, the pass/fail result is not recorded. Partial credit is possible — a model can build the app but write tests that compile and then fail at runtime, earning two of the four Hello World points. <h2 id="scoring">Scoring</h2> The 10-point scoring breakdown for the initial complete run: <table> <thead> <tr> <th>Category</th> <th>Points</th> </tr> </thead> <tbody> <tr> <td>Chat response (non-empty, sensible)</td> <td>1</td> </tr> <tr> <td>SumEvens compiles</td> <td>1</td> </tr> <tr> <td>SumEvens correct</td> <td>1</td> </tr> <tr> <td>Tool call supported (not HTTP 400)</td> <td>1</td> </tr> <tr> <td>Tool call valid (structured, correct function)</td> <td>1</td> </tr> <tr> <td>Instruction followed (exactly three words)</td> <td>1</td> </tr> <tr> <td>Hello World app compiles</td> <td>1</td> </tr> <tr> <td>Hello World app correct output</td> <td>1</td> </tr> <tr> <td>Hello World tests compile</td> <td>1</td> </tr> <tr> <td>Hello World tests pass</td> <td>1</td> </tr> </tbody> </table> After the initial runs I extended the suite with three more tests, bringing the maximum to 13: a reasoning test (a word problem with an exact numeric answer — $4.50, no other text), a JSON output test (produce a valid JSON array of at least three programming language names), and a sequence test (output the numbers 1 through 5, one per line, nothing else). All three are binary pass/fail with no partial credit. The reasoning and sequence tests catch models that ignore output constraints even when the constraint is explicit. Several did. <h2 id="unit-tests">Unit Tests</h2> The test project covers 13 cases across five test classes. <code>ModelResultTests</code> verifies that the scoring logic is correct — all true returns the expected sum, all false returns zero, <code>ThinkRequired</code> does not affect the score. <code>RoslynEvaluatorTests</code> covers the markdown fence stripping and three evaluation cases: correct implementation, wrong result, and garbage input. <code>ScorerTests</code> uses a <code>MockRunner</code> that replays canned responses and verifies that the Scorer assembles the <code>ModelResult</code> correctly for the pass case, the tool-rejected case, and the instruction-failure case. <code>ConsoleReporterTests</code> confirms that <code>PrintTable</code> does not throw with null prior results or when a model has regressed since the previous run. None of these tests require a running Ollama instance. The mock runner pattern makes the Scorer fully testable without any external dependencies. <h2 id="first-complete-run">First Complete Run</h2> Thirteen models, ten metrics each. This is what came back: <table> <thead> <tr> <th>Model</th> <th>Score</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>gemma4:latest</td> <td>10/10</td> <td>Clean sweep</td> </tr> <tr> <td>glm-4.7-flash</td> <td>9/10</td> <td></td> </tr> <tr> <td>gemma4:26b</td> <td>8/10</td> <td></td> </tr> <tr> <td>qwen2.5:14b</td> <td>8/10</td> <td></td> </tr> <tr> <td>devstral:latest</td> <td>7/10</td> <td></td> </tr> <tr> <td>qwen3-coder:30b</td> <td>7/10</td> <td></td> </tr> <tr> <td>qwen3:14b</td> <td>7/10</td> <td></td> </tr> <tr> <td>mistral:latest</td> <td>6/10</td> <td></td> </tr> <tr> <td>gpt-oss:20b</td> <td>5/10</td> <td>think_required detected</td> </tr> <tr> <td>phi4:14b</td> <td>5/10</td> <td></td> </tr> <tr> <td>llava:7b</td> <td>5/10</td> <td></td> </tr> <tr> <td>qwen2.5-coder:14b</td> <td>4/10</td> <td></td> </tr> <tr> <td>qwen3:30b</td> <td>3/10</td> <td></td> </tr> </tbody> </table> gemma4:latest — a ~12B parameter model — scores 10 out of 10. It answers the chat question, writes <code>SumEvens</code> correctly, emits a proper tool call, follows the three-word instruction, builds the Hello World app, writes tests that compile and pass, gets the math problem right, produces valid JSON, and outputs the sequence with no extra text. On every metric the harness defines, it is the best model in the pool by a clean margin over everything larger than it. The result is worth sitting with. A model less than half the size of qwen3:30b outscores it by seven points. glm-4.7-flash is a 30B MoE and comes in second at 9/10. The coding-focused variants — qwen2.5-coder and qwen3-coder — score lower than their general-purpose counterparts at similar sizes. The obvious interpretation is that gemma4:latest is simply the best model here. The problem is that the harness measures what I built the harness to measure. Before drawing that conclusion, I need to know whether these metrics are the right metrics. <hr> The full source is at <a href="https://github.com/erichexter/ollama-model-bench">github.com/erichexter/ollama-model-bench</a>. <hr> Next up: <a href="/erichexter/2026/05/31/local-llm-bench-part-3-single-shot-lies/">Part 3</a> digs into what the scores actually mean — and why gemma4:latest&rsquo;s clean sweep turned out to be almost entirely beside the point. Eric Hexter Search — The Evolution of the Karpathy LLM Wiki 2026-05-26T12:00:00Z 2026-05-26T12:00:00Z https://lostechies.com/erichexter/2026/05/26/search-evolution-of-the-karpathy-llm-wiki/ My LLM notes wiki outgrew file reads. Agents were pulling entire files to find a single relevant section — burning tokens on context that didn&rsquo;t matter, missing things that were buried three pages deep. The corpus had just grown past the point where IO-based access was practical. The fix was search. And since agents need tools, the obvious move was to build it as an MCP server. But if you&rsquo;re building search anyway, plain keyword matching felt like leaving half the value on the table — too easy to miss conceptual matches that don&rsquo;t share exact terms. So: something old and something new. SQLite already has FTS5. sqlite-vec adds HNSW vector search as a loadable extension. Ollama runs the embedding model locally. Put them together and you get hybrid RAG on hardware you already own, exposed as an MCP tool any agent in the fleet can call. This post covers how it&rsquo;s built — starting from what the agent sees and working inward to the SQL and vector embeddings. <hr> <h2 id="what-the-agent-sees">What the Agent Sees</h2> From the agent&rsquo;s perspective, this is just an MCP server with a set of tools. Point an <code>.mcp.json</code> at the host and the tools are available. No setup, no SDK, no awareness of what&rsquo;s running underneath. The primary tool is <code>search_knowledge</code>: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">{ &#34;method&#34;: &#34;tools/call&#34;, &#34;params&#34;: { &#34;name&#34;: &#34;search_knowledge&#34;, &#34;arguments&#34;: { &#34;query&#34;: &#34;attention mechanism scaled dot product&#34;, &#34;top_k&#34;: 5, &#34;hybrid_alpha&#34;: 0.6, &#34;sources&#34;: [&#34;karpathy-wiki&#34;] } } } </code></pre></div>The response comes back as ranked chunks with source context: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json">{ &#34;content&#34;: [{ &#34;type&#34;: &#34;text&#34;, &#34;text&#34;: &#34;[ { \&#34;text\&#34;: \&#34;Scaled dot-product attention divides the dot products by √d_k to prevent vanishing gradients in high dimensions...\&#34;, \&#34;source\&#34;: \&#34;karpathy-wiki\&#34;, \&#34;relPath\&#34;: \&#34;transformers/attention.md\&#34;, \&#34;score\&#34;: 0.91, \&#34;frontmatter\&#34;: { \&#34;tags\&#34;: [\&#34;attention\&#34;, \&#34;transformers\&#34;] } }, ... ]&#34; }] } </code></pre></div>The agent gets ranked text chunks, source file paths, and scores. It doesn&rsquo;t need to know whether the result came from a vector search or keyword search — that&rsquo;s the server&rsquo;s problem. <h3 id="the-full-tool-set">The Full Tool Set</h3> Seven tools in total. <code>search_knowledge</code> covers 95% of use. <table> <thead> <tr> <th>Tool</th> <th>Purpose</th> </tr> </thead> <tbody> <tr> <td><code>search_knowledge</code></td> <td>Hybrid vec+FTS search across one or more sources.</td> </tr> <tr> <td><code>get_page</code></td> <td>Retrieve a full page by source + relative path. Use when search returns a partial chunk and you want the full document.</td> </tr> <tr> <td><code>list_sources</code></td> <td>Lists indexed sources with page/chunk counts and last-indexed timestamps.</td> </tr> <tr> <td><code>get_stats</code></td> <td>Query counts and latencies over 1h / 24h / 7d / 30d windows.</td> </tr> <tr> <td><code>get_query_log</code></td> <td>Recent query history. Useful for understanding what agents are actually asking.</td> </tr> <tr> <td><code>refresh_ingest</code></td> <td>Trigger immediate re-indexing for a source after a write.</td> </tr> <tr> <td><code>ping</code></td> <td>Returns current UTC. Health check.</td> </tr> </tbody> </table> <code>list_sources</code> is underrated as a diagnostic. A 200 response from the API tells you nothing about whether the index is populated. If results are poor, check <code>pageCount &gt; 0</code> and that <code>lastIndexed</code> is recent before assuming the search logic is wrong. <h3 id="the-hybrid_alpha-parameter">The <code>hybrid_alpha</code> Parameter</h3> This is the control knob for the blend between vector search and full-text search. <ul> <li><code>0.0</code> — pure FTS (BM25 keyword ranking)</li> <li><code>1.0</code> — pure vector (semantic similarity)</li> <li><code>0.5</code> — equal blend (default)</li> </ul> In practice, <code>0.6</code>–<code>0.7</code> (vector-weighted) works better for conceptual queries: &ldquo;how does attention scale with sequence length.&rdquo; Drop toward <code>0.3</code> when you need an exact term match that the embedding model might paraphrase: specific function names, error codes, version numbers. <hr> <h2 id="how-the-search-works">How the Search Works</h2> When <code>search_knowledge</code> is called, the server runs two queries in parallel and merges the results. <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-csharp" data-lang="csharp">var vectorTask = SearchByVector(embeddingVector, topK * 2, sources); var ftsTask = SearchByFts(query, topK * 2, sources); await Task.WhenAll(vectorTask, ftsTask); var merged = Merge(vectorTask.Result, ftsTask.Result, hybridAlpha, topK); </code></pre></div>The merge step normalizes each result list&rsquo;s scores to <code>[0, 1]</code>, applies the alpha weight, sums scores per chunk (a chunk can appear in both lists), and returns the top K. Normalization matters — BM25 and HNSW distance are on completely different scales. Skip it and one path dominates every query regardless of alpha. Before either query runs, the search query itself gets embedded: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-http" data-lang="http">POST http://&lt;ollama-host&gt;:11434/api/embeddings Content-Type: application/json { &#34;model&#34;: &#34;nomic-embed-text:latest&#34;, &#34;prompt&#34;: &#34;attention mechanism scaled dot product&#34; } </code></pre></div>That gives back a 768-dimensional float vector — what the vector search runs against. <h3 id="the-vector-query">The Vector Query</h3> sqlite-vec exposes vector search through a virtual table with a <code>MATCH</code> clause. Under the hood it&rsquo;s doing an approximate nearest-neighbor scan via HNSW: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql">SELECT c.id, c.body, c.source, c.rel_path, c.frontmatter, cv.distance FROM chunk_vecs cv JOIN chunks c ON c.id = cv.chunk_id WHERE cv.embedding MATCH :embedding AND cv.k = :k AND (:sources IS NULL OR c.source IN :sources) ORDER BY cv.distance; </code></pre></div><code>distance</code> here is L2 distance — lower is closer. sqlite-vec handles all the index internals; from the query side it looks like a regular SQL query. <h3 id="the-fts-query">The FTS Query</h3> Standard SQLite FTS5 with BM25 ranking: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql">SELECT c.id, c.body, c.source, c.rel_path, c.frontmatter, bm25(chunk_fts) AS fts_score FROM chunk_fts JOIN chunks c ON c.id = chunk_fts.rowid WHERE chunk_fts MATCH :query ORDER BY bm25(chunk_fts) LIMIT :k; </code></pre></div>FTS5&rsquo;s <code>MATCH</code> supports phrase queries, prefix matching, and boolean operators. For agent queries coming in as natural language, the server sanitizes the input to a simple term query before passing it to MATCH. <hr> <h2 id="the-data-model">The Data Model</h2> Three tables carry the retrieval workload: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql">-- Chunked text with metadata CREATE TABLE chunks ( id INTEGER PRIMARY KEY, page_id INTEGER NOT NULL REFERENCES pages(id), chunk_index INTEGER NOT NULL, body TEXT NOT NULL, token_count INTEGER, source TEXT, rel_path TEXT, frontmatter TEXT ); -- Vector index (sqlite-vec extension) CREATE VIRTUAL TABLE chunk_vecs USING vec0( chunk_id INTEGER PRIMARY KEY, embedding FLOAT[768] ); -- Full-text search index (FTS5, built into SQLite) CREATE VIRTUAL TABLE chunk_fts USING fts5( body, source UNINDEXED, rel_path UNINDEXED, content=&#39;chunks&#39;, content_rowid=&#39;id&#39; ); </code></pre></div><code>chunk_vecs</code> is a <a href="https://github.com/asg017/sqlite-vec">sqlite-vec</a> <code>vec0</code> virtual table — INSERT a row with the chunk ID and its 768-dim embedding, sqlite-vec maintains the HNSW index internally. <code>chunk_fts</code> is a content-backed FTS5 table that stays in sync with <code>chunks</code> via triggers. Supporting tables: <code>pages</code> (source files with hash-based change detection), <code>indexer_runs</code> (ingest audit log), <code>query_log</code> (query history for observability). One SQLite file. No separate processes, no network hops between storage components, no backup complexity. <hr> <h2 id="the-write-path">The Write Path</h2> When a document is added or updated in the source directory, the indexer picks it up: <ol> <li>SHA-256 hash the file. Compare against <code>pages.content_hash</code>. Skip if unchanged.</li> <li>Parse YAML frontmatter. Extract the body.</li> <li>Split into chunks — 512-token target, 64-token overlap, break on paragraph boundaries where possible.</li> <li>For each chunk: POST to Ollama <code>/api/embeddings</code>. Receive a 768-dim float array.</li> <li>INSERT into <code>chunks</code>. INSERT into <code>chunk_vecs</code>. FTS5 trigger handles <code>chunk_fts</code> sync.</li> <li>Update <code>pages.content_hash</code> and <code>indexed_at</code>.</li> <li>Write a row to <code>indexer_runs</code>.</li> </ol> <code>nomic-embed-text</code> is 137M parameters — fast on a GPU host, single-digit milliseconds per chunk. The indexer pipelines requests; Ollama queues them. <hr> <h2 id="gotchas">Gotchas</h2> The embed model context limit is a silent failure. <code>nomic-embed-text</code> has an 8K token context window. Chunks that exceed it are silently not embedded — present in <code>chunks</code>, retrievable via <code>get_page</code>, invisible to vector search. No error from Ollama. Enforce the chunk size limit at ingest time. Symptom check: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql">SELECT p.rel_path, p.source, LENGTH(p.content) AS content_len FROM pages p LEFT JOIN chunks c ON c.page_id = p.id WHERE c.id IS NULL; </code></pre></div>Any row here is a page with no chunks. Stale bind mount after remount. If the CIFS mount backing the source directory remounts — after a network blip or server reboot — the container holds a file descriptor to the old empty mount point. The API returns 200. The indexer runs. It finds zero files. Nothing crashes, nothing complains. Restart the container after any storage remount. Shallow health checks miss the real failure mode. <code>GET /ping → 200</code> stays green with an empty index. Real health check: call <code>list_sources</code>, assert <code>pageCount &gt; 0</code> with a recent <code>lastIndexed</code>. You&rsquo;re monitoring the retrieval system, not just the process. <hr> <h2 id="what-this-gets-you">What This Gets You</h2> ~11K chunks, query results under 100ms on commodity hardware. The Ollama embedding call is the only network hop on the hot path — ~10ms on a GPU host for a short query. The SQLite ANN index is not the bottleneck. Hybrid search earns its keep in practice. Pure vector drifts on exact version numbers, function names, and error codes. Pure FTS misses conceptual synonyms. The blend handles both without tuning a separate retriever per query type. The MCP wrapper means any agent that speaks the protocol can call it without any awareness of the storage layer. Add a source, re-index, done — consumers don&rsquo;t change. Most databases can store embeddings at this point. The reason to reach for SQLite + sqlite-vec specifically is that you probably already have it, it requires no new infrastructure, and the FTS5 index is already there. The hybrid approach — run both searches, blend by alpha — transfers to any store that can handle both. The schema and the search logic are the portable parts. Eric Hexter Which Local Models Can Actually Code? 2026-05-25T12:00:00Z 2026-05-25T12:00:00Z https://lostechies.com/erichexter/2026/05/25/local-llm-bench-part-1-which-models-can-chat/ Part 1 of 5 in the <a href="/erichexter/2026/05/25/local-llm-bench-part-1-which-models-can-chat/">Local LLM Bench series</a>. I had ten local models installed and no good answer to a simple question: which of them could actually do useful work? Chat demos are easy to fake. I wanted to know whether these models could write working code, call tools correctly, and follow instructions without needing hand-holding. The only way to find out was to run them. <h2 id="the-setup">The Setup</h2> Machine is an Alienware Windows 11 box with an RTX 5080 carrying 16GB of VRAM. Ollama is running locally, serving the following ten models: <ul> <li>mistral:latest (7B)</li> <li>llava:7b (7B, vision)</li> <li>gemma4:latest (~12B)</li> <li>gemma4:26b (26B)</li> <li>qwen3:14b (14B)</li> <li>qwen3:30b (30B)</li> <li>phi4:14b (14B)</li> <li>qwen2.5:14b (14B)</li> <li>qwen2.5-coder:14b (14B, coding-focused)</li> <li>glm-4.7-flash (30B MoE)</li> </ul> The size range alone tells you the hardware story. Anything under about 20B fits in VRAM comfortably. The 26B and 30B models spill onto system RAM — which you feel in the latency numbers. <h2 id="first-pass-two-prompts-powershell">First Pass: Two Prompts, PowerShell</h2> The first script was about as minimal as it gets. Two prompts per model: &ldquo;What is the capital of France?&rdquo; to confirm the model is responding at all, and &ldquo;Write an <code>is_prime()</code> function in Python&rdquo; as a basic code generation check. No scoring, no verification — just checking that something came back. Most models answered both prompts without incident. Then I hit the bigger ones. gemma4:26b, glm-4.7-flash, and qwen3:30b all returned empty responses. Not errors — the HTTP calls succeeded, Ollama said everything was fine, the responses just contained no text. That took longer than it should have, and the answer was different for each model. <h2 id="the-think-mode-wall">The Think-Mode Wall</h2> qwen3 models support a reasoning mode where the model works through a problem step by step before producing visible output. The reasoning tokens live inside <code>&lt;think&gt;...&lt;/think&gt;</code> blocks and don&rsquo;t count against the response. What does count against the response is the token budget, and when I was requesting with a tight <code>num_predict</code> limit, the model was spending the entire budget on internal reasoning and returning nothing to the caller. glm-4.7-flash has its own variant of the same mode — different model family, same symptom. The fix for both: add <code>&quot;think&quot;: false</code> to the request body. With that flag set, qwen3:14b went from returning a blank response to producing clean, working code in about 2 seconds. The qwen3 and glm models followed. gemma4:26b&rsquo;s blank responses were a separate problem entirely. At 26B it spills to RAM, and with a tight <code>num_predict</code> budget and slow generation speed, the script&rsquo;s read timeout was firing before any tokens arrived. More headroom fixed it. The lesson here is that &ldquo;model returned empty string&rdquo; and &ldquo;model failed&rdquo; are not the same thing, and you have to understand what each model family expects before you can interpret the output. <h2 id="tool-calling-where-things-got-interesting">Tool-Calling: Where Things Got Interesting</h2> Once the basic chat and code tests were passing, I added a tool-calling test. The prompt was &ldquo;What&rsquo;s the weather in Paris?&rdquo; with a <code>get_weather</code> function schema attached to the request. A model that handles tool calling correctly should stop generating text and instead emit a structured <code>tool_calls</code> object pointing at <code>get_weather</code> with the right argument. A model that doesn&rsquo;t understand the protocol either returns prose (&ldquo;I don&rsquo;t have access to weather data&rdquo;), returns a JSON blob as plain text, or refuses the request entirely with an HTTP 400. The results split into three clear buckets. mistral, gemma4 (both sizes), qwen3:14b, qwen2.5:14b, and glm-4.7-flash all produced proper structured <code>tool_calls</code>. That is the expected behavior — the model uses the tool schema as intended. qwen2.5-coder:14b was the interesting failure. It returned what looked like a tool call, but as a raw JSON string embedded in the message content rather than as a structured <code>tool_calls</code> entry. The model clearly understood what was being asked; it just didn&rsquo;t output it in the right format. A &ldquo;coder&rdquo; model is not necessarily a &ldquo;tool-aware&rdquo; model. They are different capabilities. llava:7b and phi4:14b both returned HTTP 400 on any request that included the <code>tools</code> field. Those models simply do not accept the parameter — the API rejects it before the model even sees the prompt. llava makes sense here: it is a vision model, not a chat/agent model. phi4 is less obvious. <h2 id="mid-phase-additions">Mid-Phase Additions</h2> While working through these tests I pulled in three more models that had come up in research as strong candidates for coding benchmarks: devstral:latest (22B, Devstral Small — Mistral&rsquo;s coding-focused release), qwen3-coder:30b (~30B, Qwen&rsquo;s coding-tuned variant), and gpt-oss:20b (~20B). All three were added before the formal scoring phase started. <h2 id="the-baseline-table">The Baseline Table</h2> Here is where every model stood after the initial phase — response times are wall-clock from the PowerShell script, rounded to the nearest second: <table> <thead> <tr> <th>Model</th> <th>Size</th> <th>Chat</th> <th>Code</th> <th>Tool call</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>mistral:latest</td> <td>7B</td> <td>3s</td> <td>1s</td> <td>proper</td> <td></td> </tr> <tr> <td>llava:7b</td> <td>7B</td> <td>4s</td> <td>&lt;1s</td> <td>rejected</td> <td>Vision model</td> </tr> <tr> <td>gemma4:latest</td> <td>~12B</td> <td>6s</td> <td>1s</td> <td>proper</td> <td></td> </tr> <tr> <td>qwen3:14b</td> <td>14B</td> <td>4s</td> <td>1s</td> <td>proper</td> <td>think=false required</td> </tr> <tr> <td>phi4:14b</td> <td>14B</td> <td>5s</td> <td>1s</td> <td>rejected</td> <td></td> </tr> <tr> <td>qwen2.5:14b</td> <td>14B</td> <td>6s</td> <td>1s</td> <td>proper</td> <td></td> </tr> <tr> <td>qwen2.5-coder:14b</td> <td>14B</td> <td>6s</td> <td>1s</td> <td>text (not structured)</td> <td>&ldquo;coder&rdquo; does not mean tool-aware</td> </tr> <tr> <td>gemma4:26b</td> <td>26B</td> <td>9s</td> <td>3s</td> <td>proper</td> <td>Partial CPU offload</td> </tr> <tr> <td>glm-4.7-flash</td> <td>30B MoE</td> <td>8s</td> <td>4s</td> <td>proper</td> <td></td> </tr> <tr> <td>qwen3:30b</td> <td>30B</td> <td>14s</td> <td>8s</td> <td>proper</td> <td>Slowest in pool</td> </tr> </tbody> </table> The latency numbers tell one story — size matters, mostly predictably. The tool-call column tells another: ten models, three different behaviors from the same input, and two of them would silently fail in any agentic loop that expected structured output. <h2 id="what-works-actually-means">What &ldquo;Works&rdquo; Actually Means</h2> The issue with this baseline is that &ldquo;passes&rdquo; hides a lot. A model that returns a tool call in the message content instead of the <code>tool_calls</code> field looks fine until your application tries to deserialize the response. A model that works at <code>num_predict=300</code> might silently truncate at <code>num_predict=100</code>. A model that answers &ldquo;capital of France&rdquo; correctly might write Python <code>is_prime()</code> that has an off-by-one error nobody noticed because nobody ran it. Everything in this phase was manual inspection. I was reading outputs and deciding they looked reasonable. That is not a test; that is a vibe check. The only way to actually know whether a model can write working code is to compile and run the code. Which meant building something more serious. <hr> Next up: <a href="/erichexter/2026/05/28/local-llm-bench-part-2-building-the-harness/">Part 2</a> covers building the .NET 10 benchmark harness — including a scoring system that actually executes model-generated C# and runs the tests. Eric Hexter