<?xml version="1.0" encoding="UTF-8" standalone="no"?><rss xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" version="2.0">

<channel>
	<title>Deep Fried Bytes</title>
	<atom:link href="http://deepfriedbytes.com/feed/" rel="self" type="application/rss+xml"/>
	<link>https://deepfriedbytes.com/</link>
	<description>Deep Fried Bytes is an audio talk show with a Southern flavor hosted by technologists and developers Keith Elder and Chris Woodruff. The show discusses a wide range of topics including application development, operating systems and technology in general. Anything is fair game if it plugs into the wall or takes a battery.</description>
	<lastBuildDate>Wed, 11 Nov 2026 21:32:53 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1</generator>

<image>
	<url>https://deepfriedbytes.com/wp-content/uploads/2025/07/cropped-cropped-Deep-Fried-Bytes-32x32.png</url>
	<title>Blog about a digital future</title>
	<link>https://deepfriedbytes.com/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<itunes:explicit>no</itunes:explicit><copyright>2008 by Deep Fried Bytes, All rights reserved</copyright><itunes:image href="http://deepfriedbytes.com/images/deepfried_feedimage.png"/><itunes:keywords>technology,windows,apple,linux,osx,net,c,vb,net,home,server,ipod,zune,sql,server,programmer,developer</itunes:keywords><itunes:summary>Deep Fried Bytes is an audio talk show with a Southern flavor hosted by technologists and developers Keith Elder and Chris Woodruff. The show discusses a wide range of topics including application development, operating systems and technology in general. Anything is fair game if it plugs into the wall or takes a battery.</itunes:summary><itunes:subtitle>Everything tastes better deep fried, especially technology!</itunes:subtitle><itunes:category text="Technology"/><itunes:category text="Technology"><itunes:category text="Podcasting"/></itunes:category><itunes:category text="Technology"><itunes:category text="Gadgets"/></itunes:category><itunes:category text="Technology"><itunes:category text="Tech News"/></itunes:category><itunes:author>Keith Elder &amp; Chris Woodruff</itunes:author><itunes:owner><itunes:email>comments@deepfriedbytes.com</itunes:email><itunes:name>Keith Elder &amp; Chris Woodruff</itunes:name></itunes:owner><item>
		<title>The AI Vision Maintenance Tax Solo Freelancers Miss</title>
		<link>https://deepfriedbytes.com/the-ai-vision-maintenance-tax-solo-freelancers-miss/</link>
		
		
		<pubDate>Wed, 11 Nov 2026 07:17:36 +0000</pubDate>
				<category><![CDATA[AI Computer Vision]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[API]]></category>
		<category><![CDATA[Computer Vision]]></category>
		<category><![CDATA[Machine Learning Apps]]></category>
		<category><![CDATA[Software Engineering]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/the-ai-vision-maintenance-tax-solo-freelancers-miss/</guid>

					<description><![CDATA[<p>A vision feature can be cheap to launch and expensive to keep honest. For a solo freelancer, the recurring work is not usually retraining; it is finding out that yesterday’s camera, lighting, API response, or customer expectations changed. I would choose a vision system whose failures I can inspect and reverse myself, even if a more sophisticated model wins the initial demo. The first model choice matters less than the exit path I treat the first-app comparison, Custom Vision Model or Pretrained API Which Fits Your First App, as a launch checklist rather than a maintenance plan. The choice between a pretrained API and a custom model matters, but neither choice tells you who will diagnose a bad result on a Tuesday night. If the client cannot show you the input, the version that processed it, and the decision that followed, you have inherited an incident without evidence. A pretrained vision API wins when labels are scarce and its existing categories closely match the job, because you can test a useful workflow before collecting a training set. It costs per request, exposes you to provider changes, and may impose retention or data-transfer terms you need to check. A custom model run with ONNX Runtime wins when the task is narrow, you have permission to retain representative examples, and an API’s outputs repeatedly miss the same requirement. It costs annotation time, evaluation time, and responsibility for serving a pinned model file. Those are different maintenance bills, not a free-versus-paid choice. My default would be a pretrained API behind one small adapter, with a manual-review path that remains usable if the API fails. I would not train a custom model merely to avoid an API bill, because a small monthly invoice is easier for one person to manage than an unmeasured labeling and retraining obligation. I would change that position only after saved examples show a persistent error the API cannot be configured to avoid. Make the adapter return your own stable fields, such as decision, confidence, provider, and model_version, rather than passing the provider’s full JSON response into the app. That boundary limits a provider migration to one place because the rest of the code depends on your contract. Pin the provider endpoint or model identifier where the service permits it, and keep a small set of approved test images; a green HTTP response proves transport worked, not that the answer stayed the same. There is also an exit cost in the images themselves. If your examples contain client data, confirm who may keep them before building a training archive, because a model you cannot legally or contractually retest is difficult to maintain. Record only what you need, set a deletion date, and keep a SHA-256 hash when you need to tell whether two files are identical without keeping duplicate copies. Every accepted prediction creates a small support obligation The tempting launch metric is “the model was correct on our test set.” The maintenance question is narrower: which mistakes reach the customer without review? A false positive that merely opens a review screen has a different cost from a false positive that sends an irreversible notification. Set the review threshold around the action’s cost, because the same model score can be acceptable for one action and unacceptable for another. Start with an explicit record for each decision: a timestamp, a pseudonymous input ID, the model or API version, the score, the threshold in force, the action taken, and whether a human later corrected it. SQLite with PRAGMA journal_mode=WAL is sufficient for a single-process pilot because it makes local writes and later inspection straightforward. If multiple workers will write to the same database across machines, use a managed database instead, because a shared SQLite file is the wrong coordination mechanism. Do not confuse a confidence score with a probability of being right. If a service returns 0.90, that number may be a ranking signal rather than a calibrated 90% chance, because providers and model architectures define scores differently. The threshold must be chosen against labeled examples from the actual workflow. A provisional threshold of 0.85 is a value to tune, not a recommendation to ship: compare errors above and below it before allowing automatic acceptance. A workable solo routine is to review a fixed sample of accepted decisions and all customer-reported mistakes each week. A sample of 30 reviewed accepted decisions is an operational starting count, not a statistically sufficient guarantee; small or changing traffic can make its error rate swing sharply. Log the denominator alongside the rate so “zero errors” cannot disguise “zero labels.” Include some low-confidence decisions in the review queue as well, because examining only accepted outputs will hide cases the system repeatedly sends to manual work. Keep the review record deliberately plain. A CSV export can carry IDs, scores, outcomes, and model versions; a separate, access-controlled folder can hold the few images you have permission to retain. Strip unnecessary EXIF metadata before sharing examples, because camera locations and timestamps can reveal information unrelated to the task. This is less elegant than an annotation platform, but it is easier to audit when you are the only operator. Monitoring must tell you when to stop trusting automation I use the production-metrics guide, How to prove your vision model works in production metrics, as a starting point for a stronger rule: a metric earns its place only if it changes what I do next. A dashboard of request counts cannot tell you whether accepted predictions are wrong, because correctness usually arrives later through review or complaint. Write the pause rule before launch so you are not inventing one while a client waits for an answer. The following Python 3.11 smoke test calculates the error rate among reviewed, automatically accepted examples. It runs as written and intentionally refuses to call a four-example sample healthy. Replace the inline rows with a reviewed CSV export before using it for an operational decision. python3 - &#60;&#60;'PY' import csv, io rows = csv.DictReader(io.StringIO("""score,correct 0.93,1 0.88,0 0.94,0 0.89,0""")) accepted = [r for r in rows if float(r["score"]) &#62;= 0.85] n = len(accepted) rate = sum(r["correct"] == "0" for r in accepted) / n if n else 0 print(f"sample={n} error={rate:.1%}") print("REVIEW" if n &#62;= 30 and rate &#62; 0.10 else "INSUFFICIENT LABELS" if n &#60; 30 else "OK") PY The 10% pause boundary is an example policy to set with the client, not evidence that a 9% error rate is safe. The script also needs an important production companion: report how many requests bypassed review, because an apparently good reviewed sample may not represent the requests the model accepted. If corrections arrive days late, calculate outcomes by decision date and mark recent cohorts incomplete; otherwise, late labels can make a new version appear better than it is. Track availability separately from correctness. Log HTTP status codes and latency at the adapter, and honor HTTP 429 and the Retry-After header rather than retrying immediately, because a burst of retries can prolong a provider rate limit. In an illustrative pilot, a measured p95 latency of 420 ms across 200 requests would be a baseline for that pilot only; it would not predict a future client’s traffic. If requests start timing out, fall back to manual review rather than silently treating missing predictions as negative results. I would not deploy Prometheus and OpenTelemetry just to watch one low-traffic freelancer project, because maintaining collectors and dashboards can consume more time than reading a daily export. A scheduled check, an emailed failure notice, and a weekly review appointment are enough until missed incidents show that the simpler arrangement is failing. A rollback plan saves more time than a retraining plan Many vision regressions start outside the model. A phone update can change image dimensions; an upload path can rotate pictures differently; a provider can change response fields. Keep a compact fixture set with expected decisions and image properties, then run it with pytest against the adapter before every release. Store the version of the adapter alongside the model identifier, because “same model” does not imply “same preprocessing.” For a self-hosted model, pin the ONNX file and its checksum, the ONNX Runtime version, and any preprocessing constants such as resize dimensions and color order. A Docker image pinned by digest helps reproduce the software environment because a mutable tag can point to different contents later. It does not preserve the customer’s input distribution, so keep separately approved examples that exercise glare, rotation, blur, and the kinds of mistakes reviewers have actually seen. For an API, save redacted request and response examples and test your adapter against them. Disable automatic acceptance with a configuration switch that does not require retraining or redeployment, because a freelancer needs a safe response even when the provider is unavailable. Tell the client what the manual path costs in time. An honest delay is preferable to an automatic answer you cannot validate. Keep the rollback small enough to rehearse: restore the previous adapter release, restore its threshold, process one known fixture, and confirm that new requests enter the intended review path. Run that rehearsal after a change rather than assuming a saved model file is a rollback; configuration, preprocessing, and routing can all change the result while the weights stay identical. The first maintenance task is to write the pause rule Before signing off on a demo, take ten recent inputs you are allowed to retain and record the output, version, threshold, and human verdict for each. Then write down who switches off automatic acceptance, what evidence triggers that switch, and how work continues manually. If that procedure cannot fit on one page, simplify the feature before promising the client it will run unattended.</p>
<p>The post <a href="https://deepfriedbytes.com/the-ai-vision-maintenance-tax-solo-freelancers-miss/">The AI Vision Maintenance Tax Solo Freelancers Miss</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A vision feature can be cheap to launch and expensive to keep honest. For a solo freelancer, the recurring work is not usually retraining; it is finding out that yesterday’s camera, lighting, API response, or customer expectations changed. I would choose a vision system whose failures I can inspect and reverse myself, even if a more sophisticated model wins the initial demo.</p>
<h2>The first model choice matters less than the exit path</h2>
<p>I treat the first-app comparison, <a href=/custom-vision-model-or-pretrained-api-which-fits-your-first-app/>Custom Vision Model or Pretrained API Which Fits Your First App</a>, as a launch checklist rather than a maintenance plan. The choice between a pretrained API and a custom model matters, but neither choice tells you who will diagnose a bad result on a Tuesday night. If the client cannot show you the input, the version that processed it, and the decision that followed, you have inherited an incident without evidence.</p>
<p><strong>A pretrained vision API wins</strong> when labels are scarce and its existing categories closely match the job, because you can test a useful workflow before collecting a training set. It costs per request, exposes you to provider changes, and may impose retention or data-transfer terms you need to check. <strong>A custom model run with ONNX Runtime wins</strong> when the task is narrow, you have permission to retain representative examples, and an API’s outputs repeatedly miss the same requirement. It costs annotation time, evaluation time, and responsibility for serving a pinned model file. Those are different maintenance bills, not a free-versus-paid choice.</p>
<p>My default would be a pretrained API behind one small adapter, with a manual-review path that remains usable if the API fails. I would <strong>not</strong> train a custom model merely to avoid an API bill, because a small monthly invoice is easier for one person to manage than an unmeasured labeling and retraining obligation. I would change that position only after saved examples show a persistent error the API cannot be configured to avoid.</p>
<p>Make the adapter return your own stable fields, such as <em>decision</em>, <em>confidence</em>, <em>provider</em>, and <em>model_version</em>, rather than passing the provider’s full JSON response into the app. That boundary limits a provider migration to one place because the rest of the code depends on your contract. Pin the provider endpoint or model identifier where the service permits it, and keep a small set of approved test images; a green HTTP response proves transport worked, not that the answer stayed the same.</p>
<p>There is also an exit cost in the images themselves. If your examples contain client data, confirm who may keep them before building a training archive, because a model you cannot legally or contractually retest is difficult to maintain. Record only what you need, set a deletion date, and keep a SHA-256 hash when you need to tell whether two files are identical without keeping duplicate copies.</p>
<h2>Every accepted prediction creates a small support obligation</h2>
<p>The tempting launch metric is “the model was correct on our test set.” The maintenance question is narrower: which mistakes reach the customer without review? A false positive that merely opens a review screen has a different cost from a false positive that sends an irreversible notification. Set the review threshold around the action’s cost, because the same model score can be acceptable for one action and unacceptable for another.</p>
<p>Start with an explicit record for each decision: a timestamp, a pseudonymous input ID, the model or API version, the score, the threshold in force, the action taken, and whether a human later corrected it. SQLite with <code>PRAGMA journal_mode=WAL</code> is sufficient for a single-process pilot because it makes local writes and later inspection straightforward. If multiple workers will write to the same database across machines, use a managed database instead, because a shared SQLite file is the wrong coordination mechanism.</p>
<p>Do not confuse a confidence score with a probability of being right. If a service returns <em>0.90</em>, that number may be a ranking signal rather than a calibrated 90% chance, because providers and model architectures define scores differently. The threshold must be chosen against labeled examples from the actual workflow. A provisional threshold of <em>0.85</em> is a value to tune, not a recommendation to ship: compare errors above and below it before allowing automatic acceptance.</p>
<p>A workable solo routine is to review a fixed sample of accepted decisions and all customer-reported mistakes each week. A sample of <em>30</em> reviewed accepted decisions is an operational starting count, not a statistically sufficient guarantee; small or changing traffic can make its error rate swing sharply. Log the denominator alongside the rate so “zero errors” cannot disguise “zero labels.” Include some low-confidence decisions in the review queue as well, because examining only accepted outputs will hide cases the system repeatedly sends to manual work.</p>
<p>Keep the review record deliberately plain. A CSV export can carry IDs, scores, outcomes, and model versions; a separate, access-controlled folder can hold the few images you have permission to retain. Strip unnecessary EXIF metadata before sharing examples, because camera locations and timestamps can reveal information unrelated to the task. This is less elegant than an annotation platform, but it is easier to audit when you are the only operator.</p>
<h2>Monitoring must tell you when to stop trusting automation</h2>
<p>I use the production-metrics guide, <a href=/how-to-prove-your-vision-model-works-in-production-metrics/>How to prove your vision model works in production metrics</a>, as a starting point for a stronger rule: a metric earns its place only if it changes what I do next. A dashboard of request counts cannot tell you whether accepted predictions are wrong, because correctness usually arrives later through review or complaint. Write the pause rule before launch so you are not inventing one while a client waits for an answer.</p>
<p>The following Python 3.11 smoke test calculates the error rate among reviewed, automatically accepted examples. It runs as written and intentionally refuses to call a four-example sample healthy. Replace the inline rows with a reviewed CSV export before using it for an operational decision.</p>
<pre>python3 - &lt;&lt;'PY'
import csv, io
rows = csv.DictReader(io.StringIO("""score,correct
0.93,1
0.88,0
0.94,0
0.89,0"""))
accepted = [r for r in rows if float(r["score"]) &gt;= 0.85]
n = len(accepted)
rate = sum(r["correct"] == "0" for r in accepted) / n if n else 0
print(f"sample={n} error={rate:.1%}")
print("REVIEW" if n &gt;= 30 and rate &gt; 0.10 else "INSUFFICIENT LABELS" if n &lt; 30 else "OK")
PY</pre>
<p>The <em>10%</em> pause boundary is an example policy to set with the client, not evidence that a 9% error rate is safe. The script also needs an important production companion: report how many requests bypassed review, because an apparently good reviewed sample may not represent the requests the model accepted. If corrections arrive days late, calculate outcomes by decision date and mark recent cohorts incomplete; otherwise, late labels can make a new version appear better than it is.</p>
<p>Track availability separately from correctness. Log HTTP status codes and latency at the adapter, and honor HTTP 429 and the <code>Retry-After</code> header rather than retrying immediately, because a burst of retries can prolong a provider rate limit. In an illustrative pilot, a measured p95 latency of <em>420 ms across 200 requests</em> would be a baseline for that pilot only; it would not predict a future client’s traffic. If requests start timing out, fall back to manual review rather than silently treating missing predictions as negative results.</p>
<p>I would not deploy Prometheus and OpenTelemetry just to watch one low-traffic freelancer project, because maintaining collectors and dashboards can consume more time than reading a daily export. A scheduled check, an emailed failure notice, and a weekly review appointment are enough until missed incidents show that the simpler arrangement is failing.</p>
<h2>A rollback plan saves more time than a retraining plan</h2>
<p>Many vision regressions start outside the model. A phone update can change image dimensions; an upload path can rotate pictures differently; a provider can change response fields. Keep a compact fixture set with expected decisions and image properties, then run it with pytest against the adapter before every release. Store the version of the adapter alongside the model identifier, because “same model” does not imply “same preprocessing.”</p>
<p>For a self-hosted model, pin the ONNX file and its checksum, the ONNX Runtime version, and any preprocessing constants such as resize dimensions and color order. A Docker image pinned by digest helps reproduce the software environment because a mutable tag can point to different contents later. It does not preserve the customer’s input distribution, so keep separately approved examples that exercise glare, rotation, blur, and the kinds of mistakes reviewers have actually seen.</p>
<p>For an API, save redacted request and response examples and test your adapter against them. Disable automatic acceptance with a configuration switch that does not require retraining or redeployment, because a freelancer needs a safe response even when the provider is unavailable. Tell the client what the manual path costs in time. An honest delay is preferable to an automatic answer you cannot validate.</p>
<p>Keep the rollback small enough to rehearse: restore the previous adapter release, restore its threshold, process one known fixture, and confirm that new requests enter the intended review path. Run that rehearsal after a change rather than assuming a saved model file is a rollback; configuration, preprocessing, and routing can all change the result while the weights stay identical.</p>
<h2>The first maintenance task is to write the pause rule</h2>
<p>Before signing off on a demo, take ten recent inputs you are allowed to retain and record the output, version, threshold, and human verdict for each. Then write down who switches off automatic acceptance, what evidence triggers that switch, and how work continues manually. If that procedure cannot fit on one page, simplify the feature before promising the client it will run unattended.</p>
<p>The post <a href="https://deepfriedbytes.com/the-ai-vision-maintenance-tax-solo-freelancers-miss/">The AI Vision Maintenance Tax Solo Freelancers Miss</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>From Empty Cluster to a Production-Ready GenAI Service</title>
		<link>https://deepfriedbytes.com/from-empty-cluster-to-a-production-ready-genai-service/</link>
		
		
		<pubDate>Tue, 03 Nov 2026 07:12:55 +0000</pubDate>
				<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[AI Web Solutions]]></category>
		<category><![CDATA[IT architecture]]></category>
		<category><![CDATA[Software Engineering]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/from-empty-cluster-to-a-production-ready-genai-service/</guid>

					<description><![CDATA[<p>A GenAI backend can pass a demo and still be unfit to operate: the answer looks plausible while its source document belongs to another tenant, its index is stale, or its retrieval call consumes the entire request budget. I would build the first production setup around an enforceable retrieval boundary, not model optimization, because access and failure behavior are harder to retrofit than prompts. The first deployable unit is an authorized document pipeline Begin with one source of truth: a document store whose records have a stable document ID, tenant ID, source version, and access policy. Give the ingestion worker read access to that source and write access to the retrieval database; do not give the request-serving API permission to rewrite source documents. That separation makes a bad reindex recoverable without allowing a compromised API process to alter the evidence it retrieves. For a first deployment, I would use PostgreSQL 16 with pgvector, an object store for original documents, and a queue such as Amazon SQS for change events. PostgreSQL wins here because the team can inspect document metadata, transaction state, and retrieval candidates with ordinary SQL. A dedicated vector service such as Qdrant wins when vector volume or search load has outgrown that arrangement; it costs another service to secure, back up, and reconcile with document permissions. Neither option removes the need to authorize each result. Make ingestion idempotent before generating an embedding. An event should identify a document and its source version, not contain an untrusted copy of its text. The worker fetches the current document, checks its deletion state, splits it into chunks, embeds those chunks, and replaces the previous version in one database transaction. A uniqueness constraint on document ID, source version, and chunk position prevents an SQS redelivery from creating duplicates. On deletion, remove the chunks and record the processed version; otherwise, a delayed older event can restore text that should no longer be searchable. For an initial tokenizer setting, try chunks of 400 tokens with 60 tokens of overlap; both are values to tune against your documents and evaluation queries, not universal limits. Record the splitter version alongside the embedding model ID, since changing either can make old and new vectors incomparable. OpenAI publishes a default dimension of 1,536 for text-embedding-3-small; choose the table dimension from the embedding configuration actually deployed, and reject vectors with the wrong dimension at ingestion. The model-choice framing (RAG vs Fine-Tuning in Your First GenAI Backend) is premature as a launch gate because neither choice specifies who may retrieve a chunk. I would not put all tenants into a shared vector search and ask the model to ignore unauthorized results: once the text has reached the prompt, a prompt instruction is not an access control. A small database you can recreate beats an elaborate search cluster you cannot audit Bring up a local retrieval database first. The following Bash snippet starts a PostgreSQL 16 image with pgvector and creates the minimum chunk table. It binds the port to loopback for local development; it is not a production networking or password configuration. docker run -d --name rag-db -e POSTGRES_PASSWORD=localonly \ -p 127.0.0.1:5432:5432 pgvector/pgvector:pg16 until docker exec rag-db pg_isready -U postgres &#62;/dev/null 2&#62;&#38;1; do sleep 1; done docker exec -i rag-db psql -U postgres -v ON_ERROR_STOP=1 &#60;&#60;'SQL' CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE chunks ( tenant_id text NOT NULL, document_id text NOT NULL, chunk_id text PRIMARY KEY, source_version text NOT NULL, body text NOT NULL, embedding vector(1536) NOT NULL, indexed_at timestamptz NOT NULL DEFAULT now() ); CREATE INDEX chunks_tenant ON chunks (tenant_id, document_id); SQL Pin the container image by digest in deployment so a rebuild cannot silently change database binaries. Put PostgreSQL on a private network, use a secret manager instead of the local password, enable TLS for database connections, and run migrations with a separate role. Back up both the database and original documents: vectors can be regenerated, but rebuilding them takes time and requires the exact source versions and embedding configuration. Next, implement a query that joins chunks to authorized documents before selecting the nearest matches. Use the authenticated principal from the API gateway or OIDC token to resolve allowed document IDs on the server; never accept a tenant ID from the request body as proof of access. A tenant predicate is a useful first filter, but document-level permissions still matter when people within one tenant have different access. Consider PostgreSQL row-level security as a second guard, with tests run under the restricted application role; table owners can bypass policies unless configured appropriately. Start with exact pgvector cosine-distance search using the &#60;=&#62; operator over the authorized candidate set. It may be fast enough when that set is small, and its behavior is easier to verify than an approximate index. Add an HNSW index only after query plans and load tests show the exact scan is the bottleneck: approximate search can return fewer useful matches after a selective permission filter, so test result quality as well as latency. Keep the retrieved chunk IDs and source versions with each response for audit and reproducible debugging. A working setup is more than a successful SQL query. The request path should authenticate the caller, embed the question, fetch authorized chunks, construct a prompt with source identifiers, call the generation model, and return an answer with citations that map to those identifiers. If retrieval returns no authorized evidence, return an explicit “no supporting source found” response rather than letting the model answer from general knowledge; that behavior keeps a document-grounded feature from silently changing its contract. Timeouts and traces are part of the feature, not later hardening Set one end-to-end deadline and divide it among embedding, retrieval, and generation. As an initial service-level objective, 900 ms at p95 might be a target to test for the retrieval portion, not a claim about pgvector performance. Leave room for the model call and network transit in the overall deadline. Use cancellation across HTTP and database calls so a disconnected request does not keep consuming connections and model quota. Instrument the path with OpenTelemetry spans for authentication, query embedding, authorized-candidate lookup, vector search, prompt assembly, and model response. Export duration histograms and error counters to Prometheus; calculate p95 with histogram_quantile() over an appropriate rate window. Track candidate count, returned chunk count, PostgreSQL pool wait time, and the age of the newest indexed source version. These measurements distinguish a slow search from a queue backlog or a model-provider delay, which require different fixes. The scaling warning (Your GenAI Feature Scales Until Retrieval and Latency Collide) deserves a narrower operational test: retrieval and latency collide only when traces show which resource is saturated. Run a staging workload with, for example, 30 concurrent virtual users as an initial test setting, then vary concurrency while holding the query set and authorization distribution fixed. Include common queries, low-result queries, and tenants with very different document counts. A single aggregate p95 can hide the tenant whose authorized search space is much larger. Configure separate timeouts and concurrency limits for ingestion and serving. An embedding provider outage should stop new indexing without exhausting API workers, while an SQS dead-letter queue should retain failed events for inspection. During a generation-provider outage, do not automatically retry every live request: retries can multiply traffic precisely when capacity is scarce. Return a bounded error, preserve the trace ID, and let the client retry according to its own backoff policy. A release is safe only when access, freshness, and recovery have tests Build a small evaluation set before opening traffic. Include questions with a known supporting chunk, questions with no supporting chunk, and questions where a tempting matching chunk belongs to another principal. Test the retrieved IDs separately from the generated prose; a fluent answer cannot prove that retrieval honored permissions. For each deployment, run these cases against the same source versions and record whether the expected chunk appears in the authorized top results. Add three failure drills to the deployment checklist. First, revoke document access after it has been indexed and verify that the next request cannot retrieve it; this tests authorization at query time rather than assuming index updates are instantaneous. Second, delete a document and replay an older ingestion event to confirm that version checks prevent resurrection. Third, stop the embedding provider and verify that serving continues from existing indexed data while ingestion failures go to a visible queue. Define freshness as an observable delay between a committed source version and its searchable version. A five-minute maximum lag is one possible acceptance threshold to adjust to the update rate and user expectation; alert on sustained breaches rather than every short queue spike. Also alert when the dead-letter queue grows, database connections approach their configured limit, or authorized-result counts drop abruptly after a deployment. Each alert should point to a runbook action: pause ingestion, roll back a splitter change, drain a bad event, or restore a database snapshot. Roll out to a small group with citations and trace IDs visible to support staff. Keep the previous embedding model and index available until the new version passes access tests and query evaluation, because changing embedding versions in place makes regressions hard to isolate. If costs or latency rise, inspect span data and query plans before buying a faster vector service; moving the index will not repair an authorization join, an unbounded connection pool, or a slow provider call. The first production action is to prove a forbidden document stays forbidden Create two test principals with overlapping source material but different document permissions. Index one document each, then issue the same question under both identities and inspect the retrieved chunk IDs, not merely the answers. Run that test after an access revocation as well. If it fails, stop there: tuning prompts, indexes, and latency against a retrieval path that leaks evidence only makes the wrong setup faster.</p>
<p>The post <a href="https://deepfriedbytes.com/from-empty-cluster-to-a-production-ready-genai-service/">From Empty Cluster to a Production-Ready GenAI Service</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A GenAI backend can pass a demo and still be unfit to operate: the answer looks plausible while its source document belongs to another tenant, its index is stale, or its retrieval call consumes the entire request budget. I would build the first production setup around an enforceable retrieval boundary, not model optimization, because access and failure behavior are harder to retrofit than prompts.</p>
<h2>The first deployable unit is an authorized document pipeline</h2>
<p>Begin with one source of truth: a document store whose records have a stable document ID, tenant ID, source version, and access policy. Give the ingestion worker read access to that source and write access to the retrieval database; do not give the request-serving API permission to rewrite source documents. That separation makes a bad reindex recoverable without allowing a compromised API process to alter the evidence it retrieves.</p>
<p>For a first deployment, I would use PostgreSQL 16 with pgvector, an object store for original documents, and a queue such as Amazon SQS for change events. PostgreSQL wins here because the team can inspect document metadata, transaction state, and retrieval candidates with ordinary SQL. A dedicated vector service such as Qdrant wins when vector volume or search load has outgrown that arrangement; it costs another service to secure, back up, and reconcile with document permissions. Neither option removes the need to authorize each result.</p>
<p>Make ingestion idempotent before generating an embedding. An event should identify a document and its source version, not contain an untrusted copy of its text. The worker fetches the current document, checks its deletion state, splits it into chunks, embeds those chunks, and replaces the previous version in one database transaction. A uniqueness constraint on document ID, source version, and chunk position prevents an SQS redelivery from creating duplicates. On deletion, remove the chunks and record the processed version; otherwise, a delayed older event can restore text that should no longer be searchable.</p>
<p>For an initial tokenizer setting, try chunks of <strong>400 tokens</strong> with <strong>60 tokens</strong> of overlap; both are values to tune against your documents and evaluation queries, not universal limits. Record the splitter version alongside the embedding model ID, since changing either can make old and new vectors incomparable. OpenAI publishes a default dimension of <strong>1,536</strong> for <code>text-embedding-3-small</code>; choose the table dimension from the embedding configuration actually deployed, and reject vectors with the wrong dimension at ingestion.</p>
<p>The model-choice framing (<a href=/rag-vs-fine-tuning-in-your-first-genai-backend/>RAG vs Fine-Tuning in Your First GenAI Backend</a>) is premature as a launch gate because neither choice specifies who may retrieve a chunk. I would <strong>not</strong> put all tenants into a shared vector search and ask the model to ignore unauthorized results: once the text has reached the prompt, a prompt instruction is not an access control.</p>
<h2>A small database you can recreate beats an elaborate search cluster you cannot audit</h2>
<p>Bring up a local retrieval database first. The following Bash snippet starts a PostgreSQL 16 image with pgvector and creates the minimum chunk table. It binds the port to loopback for local development; it is not a production networking or password configuration.</p>
<pre>docker run -d --name rag-db -e POSTGRES_PASSWORD=localonly \
  -p 127.0.0.1:5432:5432 pgvector/pgvector:pg16
until docker exec rag-db pg_isready -U postgres &gt;/dev/null 2&gt;&amp;1; do sleep 1; done
docker exec -i rag-db psql -U postgres -v ON_ERROR_STOP=1 &lt;&lt;'SQL'
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (
  tenant_id text NOT NULL,
  document_id text NOT NULL,
  chunk_id text PRIMARY KEY,
  source_version text NOT NULL,
  body text NOT NULL,
  embedding vector(1536) NOT NULL,
  indexed_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX chunks_tenant ON chunks (tenant_id, document_id);
SQL</pre>
<p>Pin the container image by digest in deployment so a rebuild cannot silently change database binaries. Put PostgreSQL on a private network, use a secret manager instead of the local password, enable TLS for database connections, and run migrations with a separate role. Back up both the database and original documents: vectors can be regenerated, but rebuilding them takes time and requires the exact source versions and embedding configuration.</p>
<p>Next, implement a query that joins chunks to authorized documents <em>before</em> selecting the nearest matches. Use the authenticated principal from the API gateway or OIDC token to resolve allowed document IDs on the server; never accept a tenant ID from the request body as proof of access. A tenant predicate is a useful first filter, but document-level permissions still matter when people within one tenant have different access. Consider PostgreSQL row-level security as a second guard, with tests run under the restricted application role; table owners can bypass policies unless configured appropriately.</p>
<p>Start with exact pgvector cosine-distance search using the <code>&lt;=&gt;</code> operator over the authorized candidate set. It may be fast enough when that set is small, and its behavior is easier to verify than an approximate index. Add an HNSW index only after query plans and load tests show the exact scan is the bottleneck: approximate search can return fewer useful matches after a selective permission filter, so test result quality as well as latency. Keep the retrieved chunk IDs and source versions with each response for audit and reproducible debugging.</p>
<p>A working setup is more than a successful SQL query. The request path should authenticate the caller, embed the question, fetch authorized chunks, construct a prompt with source identifiers, call the generation model, and return an answer with citations that map to those identifiers. If retrieval returns no authorized evidence, return an explicit “no supporting source found” response rather than letting the model answer from general knowledge; that behavior keeps a document-grounded feature from silently changing its contract.</p>
<h2>Timeouts and traces are part of the feature, not later hardening</h2>
<p>Set one end-to-end deadline and divide it among embedding, retrieval, and generation. As an initial service-level objective, <strong>900 ms at p95</strong> might be a target to test for the retrieval portion, not a claim about pgvector performance. Leave room for the model call and network transit in the overall deadline. Use cancellation across HTTP and database calls so a disconnected request does not keep consuming connections and model quota.</p>
<p>Instrument the path with OpenTelemetry spans for authentication, query embedding, authorized-candidate lookup, vector search, prompt assembly, and model response. Export duration histograms and error counters to Prometheus; calculate p95 with <code>histogram_quantile()</code> over an appropriate rate window. Track candidate count, returned chunk count, PostgreSQL pool wait time, and the age of the newest indexed source version. These measurements distinguish a slow search from a queue backlog or a model-provider delay, which require different fixes.</p>
<p>The scaling warning (<a href=/your-genai-feature-scales-until-retrieval-and-latency-collide/>Your GenAI Feature Scales Until Retrieval and Latency Collide</a>) deserves a narrower operational test: retrieval and latency collide only when traces show which resource is saturated. Run a staging workload with, for example, <strong>30 concurrent virtual users</strong> as an initial test setting, then vary concurrency while holding the query set and authorization distribution fixed. Include common queries, low-result queries, and tenants with very different document counts. A single aggregate p95 can hide the tenant whose authorized search space is much larger.</p>
<p>Configure separate timeouts and concurrency limits for ingestion and serving. An embedding provider outage should stop new indexing without exhausting API workers, while an SQS dead-letter queue should retain failed events for inspection. During a generation-provider outage, do not automatically retry every live request: retries can multiply traffic precisely when capacity is scarce. Return a bounded error, preserve the trace ID, and let the client retry according to its own backoff policy.</p>
<h2>A release is safe only when access, freshness, and recovery have tests</h2>
<p>Build a small evaluation set before opening traffic. Include questions with a known supporting chunk, questions with no supporting chunk, and questions where a tempting matching chunk belongs to another principal. Test the retrieved IDs separately from the generated prose; a fluent answer cannot prove that retrieval honored permissions. For each deployment, run these cases against the same source versions and record whether the expected chunk appears in the authorized top results.</p>
<p>Add three failure drills to the deployment checklist. First, revoke document access after it has been indexed and verify that the next request cannot retrieve it; this tests authorization at query time rather than assuming index updates are instantaneous. Second, delete a document and replay an older ingestion event to confirm that version checks prevent resurrection. Third, stop the embedding provider and verify that serving continues from existing indexed data while ingestion failures go to a visible queue.</p>
<p>Define freshness as an observable delay between a committed source version and its searchable version. A <strong>five-minute</strong> maximum lag is one possible acceptance threshold to adjust to the update rate and user expectation; alert on sustained breaches rather than every short queue spike. Also alert when the dead-letter queue grows, database connections approach their configured limit, or authorized-result counts drop abruptly after a deployment. Each alert should point to a runbook action: pause ingestion, roll back a splitter change, drain a bad event, or restore a database snapshot.</p>
<p>Roll out to a small group with citations and trace IDs visible to support staff. Keep the previous embedding model and index available until the new version passes access tests and query evaluation, because changing embedding versions in place makes regressions hard to isolate. If costs or latency rise, inspect span data and query plans before buying a faster vector service; moving the index will not repair an authorization join, an unbounded connection pool, or a slow provider call.</p>
<h2>The first production action is to prove a forbidden document stays forbidden</h2>
<p>Create two test principals with overlapping source material but different document permissions. Index one document each, then issue the same question under both identities and inspect the retrieved chunk IDs, not merely the answers. Run that test after an access revocation as well. If it fails, stop there: tuning prompts, indexes, and latency against a retrieval path that leaks evidence only makes the wrong setup faster.</p>
<p>The post <a href="https://deepfriedbytes.com/from-empty-cluster-to-a-production-ready-genai-service/">From Empty Cluster to a Production-Ready GenAI Service</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>5 Robotics Team Mistakes That Cost Junior Devs Weeks</title>
		<link>https://deepfriedbytes.com/5-robotics-team-mistakes-that-cost-junior-devs-weeks/</link>
		
		
		<pubDate>Thu, 29 Oct 2026 10:52:49 +0000</pubDate>
				<category><![CDATA[Robotics]]></category>
		<category><![CDATA[Development Trends]]></category>
		<category><![CDATA[IT architecture]]></category>
		<category><![CDATA[Software Engineering]]></category>
		<category><![CDATA[technology]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/5-robotics-team-mistakes-that-cost-junior-devs-weeks/</guid>

					<description><![CDATA[<p>A robot that moves on a developer’s laptop has passed an onboarding exercise, not a release test. I would not start an automation project by adding a planner or splitting nodes across machines, because neither fixes the failures a short demo rarely exposes: stale sensor data, inconsistent clocks, and commands that outlive the connection that sent them. Teams pay for that confusion in debugging time first and unsafe motion later. A moving bot is a poor release criterion because the demo hides failure modes Read ROS 2 on a laptop in 60 minutes from install to moving bot, then replay the resulting demo with missing odometry before you call it ready. The quickstart’s moving-bot milestone is useful for confirming that ROS 2, a launch file, and a command path work together. It does not establish what happens when one of those pieces stops working during motion. A common team mistake is to turn that first successful run into an implicit interface contract. A developer sees a cmd_vel message produce movement and assumes the robot will stop when the publisher exits. That assumption depends on the motor controller’s behavior, the command timeout, and whether another publisher is still active. If nobody has specified those conditions, the cost is a late discovery during integration: someone must trace which process owns motion while the robot is already behaving unexpectedly. Write down the contract before changing the architecture. For each motion command, identify its owner, the conditions under which it expires, and the component that enforces expiration. For each input, identify how old it may be before the controller rejects it. These are testable statements; “the robot responds smoothly” is not, because two observers can watch the same run and disagree about whether it passed. ROS 2 Jazzy Jalisco is a practical version to pin for a new bench exercise, but the version alone does not make the exercise reproducible because middleware, dependencies, and launch parameters also affect behavior. Record the ROS distribution, the DDS implementation, and the configuration committed with the test. A teammate should be able to replay the same fault without reconstructing your shell history. Make one failure test as routine as the happy-path demo: start motion at a deliberately capped speed, stop publishing odometry, and record the time until the controller inhibits motion. Treat 0.5 m/s as a trial cap to set for your own platform, not as a universally safe speed. If the test has no expected outcome, the team has found a missing requirement rather than a flaky robot. QoS defaults can turn a live sensor into a silent data gap ROS 2 uses DDS quality-of-service policies, and a publisher being visible on the graph does not prove a subscriber can receive its messages. A reliable subscriber, for example, is incompatible with a publisher offering only best-effort delivery. A junior developer may spend hours inspecting callback code when the real fault is a policy mismatch at discovery. Start with what the running graph actually offers. The following bench probe publishes a string and prints the endpoint details on a machine with ROS 2 Jazzy installed and sourced from the standard path: source /opt/ros/jazzy/setup.bash ros2 topic pub -r 10 /probe std_msgs/msg/String '{data: alive}' &#62;/tmp/ros-probe.log 2&#62;&#38;1 &#38; pub_pid=$! trap 'kill "$pub_pid" 2&#62;/dev/null &#124;&#124; true' EXIT sleep 2 ros2 topic info --verbose /probe ros2 topic echo --once /probe std_msgs/msg/String kill "$pub_pid" wait "$pub_pid" 2&#62;/dev/null &#124;&#124; true The 10 Hz rate here is a bench setting to make the probe easy to observe, not a recommendation for a sensor. On an actual sensor_msgs/msg/LaserScan topic, inspect reliability, durability, and history on both endpoints with ros2 topic info &#8211;verbose. Then check whether the subscriber is receiving timely scans; a matching QoS policy cannot guarantee fresh data if the publisher itself is delayed. Best effort versus reliable is a choice with different costs. Best effort often wins for frequent sensor samples when a newer sample can replace a lost one, because retransmitting an old sample may add little value. It costs you occasional missing samples, which the consumer must tolerate and measure. Reliable delivery wins when losing an individual message would break the application’s intended behavior; it costs buffering and potentially more delay under a poor connection. Neither setting is a substitute for rejecting stale input, because a delivered message can still arrive too late to use. Define freshness at the consumer, not just frequency at the publisher. An initial 100 ms maximum scan age can be a value to tune during bench tests; choose it against the robot’s speed, sensor timing, and stopping behavior. Log message header time, receipt time, and whether the controller accepted the sample. Without those fields, a team can “fix” a missing-data incident by changing QoS while leaving the late-data problem intact. Clock and frame mistakes often masquerade as planner defects When a navigation run fails, teams often tune the planner first because the visible symptom is a bad path. That can waste days if the underlying error is a transform at the wrong time or a frame with the wrong meaning. A planner cannot make a sound decision from a pose that was assembled using inconsistent timestamps. Use the ROS coordinate-frame conventions in REP-105 as a check on meaning: map, odom, and base_link are not interchangeable labels. The odom frame supports locally continuous motion estimates, while map can change as localization corrects its estimate. If a team silently treats a map correction as physical movement, a controller may react to a jump the robot never made. Inspect the transform tree with tf2 tools before changing planner parameters. Time needs the same discipline. A simulation using Gazebo and use_sim_time:=true may publish /clock, while a hardware node using its system clock may stamp nav_msgs/msg/Odometry differently. The resulting transform lookup failures can appear intermittent because they depend on which messages arrived and when. Check the use_sim_time parameter for every node involved, then compare header stamps with the clock each node uses. Pick a provisional 50 ms transform-age budget for a bench test and adjust it from observed timing and the platform’s motion constraints. Record both the measured age distribution and the count of rejected lookups; reporting only an average hides the late samples most likely to disrupt a run. rosbag2 helps here because a recorded failure gives another developer the same message timing to inspect, although playback still needs the clock and parameters set deliberately. The cost of skipping these checks is a misleading fix. Increasing a planner’s tolerance might make one run look better because it accepts older data, while reducing the time available to detect a genuinely stale pose. Change a tolerance only after you can show which timestamp or transform caused the rejection. A watchdog that only logs failures does not stop a robot ROS 2 DDS deadline and liveliness policies can reveal that an endpoint missed an expected update or disappeared. They do not, by themselves, guarantee that a motor stops, because detection still has to trigger an action through a functioning path. Treat a warning in a terminal as diagnostic evidence, not as a safety mechanism. Trace the entire stop path with the team: sensor or command expires, a supervisor detects it, the controller rejects further motion, and an actuator-level timeout or independent stop path enforces the result. If the same computer and network connection are required both to command motion and to stop it, losing that connection can defeat the planned response. A separate hardware emergency stop or motor-controller timeout may be warranted, but claiming compliance with ISO 13849 or IEC 61508 requires a safety process and evidence, not the presence of a ROS node. A provisional 200 ms command timeout is a test setting, not a universal stopping limit. Tune it against command frequency, network behavior, and the time the platform needs to stop. Then interrupt the publisher, disconnect the network, and kill the supervisor in separate trials. Those are different faults: a robot that stops after one may keep moving after another. Measure elapsed time from the last accepted command to inhibited motion, and record whether the motor controller or the ROS supervisor caused the stop. Run 30 induced dropouts as an initial experiment size rather than treating a clean run as proof of safety. Even zero observed late stops in that set would establish only what happened under those tested conditions. Put the fault procedure in launch_testing or a hardware test script so the next developer can repeat it after a controller change. Keep security separate from this stop claim. ROS_DOMAIN_ID can help separate discovery domains, and SROS2 can configure DDS security, but neither enforces a physical stop when a command path fails. Mixing those concerns creates a costly review error: the team sees an access-control measure and assumes it has also addressed motion safety. Trend-led rewrites charge interest before they remove a fault Read Robotics Software Development Trends for Smart Automation, then ask which proposed change would eliminate a failure your tests have reproduced. Trend discussions can suggest useful approaches, but adopting an architecture because it is current gives a junior developer more interfaces to debug before the team knows which interface is failing. Consider ROS 2 composable nodes versus separate processes. Composition can win when nodes exchange large messages on one computer and measured transfer overhead is a bottleneck; its cost is shared process fate and harder isolation when one component crashes. Separate processes can win when fault isolation and independent restarts matter more; their cost is extra interprocess communication and deployment configuration. Choose after a trace identifies the bottleneck or a fault test identifies the needed isolation. Otherwise, the rewrite risks changing failure behavior without improving it. The same discipline applies before switching between Cyclone DDS and Fast DDS. Middleware differences can matter on a particular network, but changing implementations while also changing QoS and node layout makes the result difficult to attribute. Hold the launch configuration constant, run the same rosbag2 input or repeatable hardware scenario, and compare the metric that prompted the change. Use colcon test and pytest for ordinary regressions, while keeping hardware fault results distinct from simulator results because a simulator cannot establish how a real actuator stops. On your next task, take one motion-related topic and write down its publisher, subscriber, QoS policies, timestamp source, expiry condition, and stop owner. Run one dropout and save the log before proposing a new node or middleware setting. If you cannot identify who rejects an expired command, that is the first defect to fix.</p>
<p>The post <a href="https://deepfriedbytes.com/5-robotics-team-mistakes-that-cost-junior-devs-weeks/">5 Robotics Team Mistakes That Cost Junior Devs Weeks</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A robot that moves on a developer’s laptop has passed an onboarding exercise, not a release test. I would not start an automation project by adding a planner or splitting nodes across machines, because neither fixes the failures a short demo rarely exposes: stale sensor data, inconsistent clocks, and commands that outlive the connection that sent them. Teams pay for that confusion in debugging time first and unsafe motion later.</p>
<h2>A moving bot is a poor release criterion because the demo hides failure modes</h2>
<p>Read <a href=/ros-2-on-a-laptop-in-60-minutes-from-install-to-moving-bot/>ROS 2 on a laptop in 60 minutes from install to moving bot</a>, then replay the resulting demo with missing odometry before you call it ready. The quickstart’s moving-bot milestone is useful for confirming that ROS 2, a launch file, and a command path work together. It does not establish what happens when one of those pieces stops working during motion.</p>
<p>A common team mistake is to turn that first successful run into an implicit interface contract. A developer sees a <em>cmd_vel</em> message produce movement and assumes the robot will stop when the publisher exits. That assumption depends on the motor controller’s behavior, the command timeout, and whether another publisher is still active. If nobody has specified those conditions, the cost is a late discovery during integration: someone must trace which process owns motion while the robot is already behaving unexpectedly.</p>
<p>Write down the contract before changing the architecture. For each motion command, identify its owner, the conditions under which it expires, and the component that enforces expiration. For each input, identify how old it may be before the controller rejects it. These are testable statements; “the robot responds smoothly” is not, because two observers can watch the same run and disagree about whether it passed.</p>
<p>ROS 2 Jazzy Jalisco is a practical version to pin for a new bench exercise, but the version alone does not make the exercise reproducible because middleware, dependencies, and launch parameters also affect behavior. Record the ROS distribution, the DDS implementation, and the configuration committed with the test. A teammate should be able to replay the same fault without reconstructing your shell history.</p>
<p>Make one failure test as routine as the happy-path demo: start motion at a deliberately capped speed, stop publishing odometry, and record the time until the controller inhibits motion. Treat <strong>0.5 m/s</strong> as a trial cap to set for your own platform, not as a universally safe speed. If the test has no expected outcome, the team has found a missing requirement rather than a flaky robot.</p>
<h2>QoS defaults can turn a live sensor into a silent data gap</h2>
<p>ROS 2 uses DDS quality-of-service policies, and a publisher being visible on the graph does not prove a subscriber can receive its messages. A reliable subscriber, for example, is incompatible with a publisher offering only best-effort delivery. A junior developer may spend hours inspecting callback code when the real fault is a policy mismatch at discovery.</p>
<p>Start with what the running graph actually offers. The following bench probe publishes a string and prints the endpoint details on a machine with ROS 2 Jazzy installed and sourced from the standard path:</p>
<pre>source /opt/ros/jazzy/setup.bash
ros2 topic pub -r 10 /probe std_msgs/msg/String '{data: alive}' &gt;/tmp/ros-probe.log 2&gt;&amp;1 &amp;
pub_pid=$!
trap 'kill "$pub_pid" 2&gt;/dev/null || true' EXIT
sleep 2
ros2 topic info --verbose /probe
ros2 topic echo --once /probe std_msgs/msg/String
kill "$pub_pid"
wait "$pub_pid" 2&gt;/dev/null || true</pre>
<p>The <strong>10 Hz</strong> rate here is a bench setting to make the probe easy to observe, not a recommendation for a sensor. On an actual <em>sensor_msgs/msg/LaserScan</em> topic, inspect reliability, durability, and history on both endpoints with <em>ros2 topic info &#8211;verbose</em>. Then check whether the subscriber is receiving timely scans; a matching QoS policy cannot guarantee fresh data if the publisher itself is delayed.</p>
<p><strong>Best effort versus reliable</strong> is a choice with different costs. Best effort often wins for frequent sensor samples when a newer sample can replace a lost one, because retransmitting an old sample may add little value. It costs you occasional missing samples, which the consumer must tolerate and measure. Reliable delivery wins when losing an individual message would break the application’s intended behavior; it costs buffering and potentially more delay under a poor connection. Neither setting is a substitute for rejecting stale input, because a delivered message can still arrive too late to use.</p>
<p>Define freshness at the consumer, not just frequency at the publisher. An initial <strong>100 ms</strong> maximum scan age can be a value to tune during bench tests; choose it against the robot’s speed, sensor timing, and stopping behavior. Log message header time, receipt time, and whether the controller accepted the sample. Without those fields, a team can “fix” a missing-data incident by changing QoS while leaving the late-data problem intact.</p>
<h2>Clock and frame mistakes often masquerade as planner defects</h2>
<p>When a navigation run fails, teams often tune the planner first because the visible symptom is a bad path. That can waste days if the underlying error is a transform at the wrong time or a frame with the wrong meaning. A planner cannot make a sound decision from a pose that was assembled using inconsistent timestamps.</p>
<p>Use the ROS coordinate-frame conventions in REP-105 as a check on meaning: <em>map</em>, <em>odom</em>, and <em>base_link</em> are not interchangeable labels. The <em>odom</em> frame supports locally continuous motion estimates, while <em>map</em> can change as localization corrects its estimate. If a team silently treats a map correction as physical movement, a controller may react to a jump the robot never made. Inspect the transform tree with <em>tf2</em> tools before changing planner parameters.</p>
<p>Time needs the same discipline. A simulation using Gazebo and <em>use_sim_time:=true</em> may publish <em>/clock</em>, while a hardware node using its system clock may stamp <em>nav_msgs/msg/Odometry</em> differently. The resulting transform lookup failures can appear intermittent because they depend on which messages arrived and when. Check the <em>use_sim_time</em> parameter for every node involved, then compare header stamps with the clock each node uses.</p>
<p>Pick a provisional <strong>50 ms</strong> transform-age budget for a bench test and adjust it from observed timing and the platform’s motion constraints. Record both the measured age distribution and the count of rejected lookups; reporting only an average hides the late samples most likely to disrupt a run. <em>rosbag2</em> helps here because a recorded failure gives another developer the same message timing to inspect, although playback still needs the clock and parameters set deliberately.</p>
<p>The cost of skipping these checks is a misleading fix. Increasing a planner’s tolerance might make one run look better because it accepts older data, while reducing the time available to detect a genuinely stale pose. Change a tolerance only after you can show which timestamp or transform caused the rejection.</p>
<h2>A watchdog that only logs failures does not stop a robot</h2>
<p>ROS 2 DDS deadline and liveliness policies can reveal that an endpoint missed an expected update or disappeared. They do not, by themselves, guarantee that a motor stops, because detection still has to trigger an action through a functioning path. Treat a warning in a terminal as diagnostic evidence, not as a safety mechanism.</p>
<p>Trace the entire stop path with the team: sensor or command expires, a supervisor detects it, the controller rejects further motion, and an actuator-level timeout or independent stop path enforces the result. If the same computer and network connection are required both to command motion and to stop it, losing that connection can defeat the planned response. A separate hardware emergency stop or motor-controller timeout may be warranted, but claiming compliance with ISO 13849 or IEC 61508 requires a safety process and evidence, not the presence of a ROS node.</p>
<p>A provisional <strong>200 ms</strong> command timeout is a test setting, not a universal stopping limit. Tune it against command frequency, network behavior, and the time the platform needs to stop. Then interrupt the publisher, disconnect the network, and kill the supervisor in separate trials. Those are different faults: a robot that stops after one may keep moving after another.</p>
<p>Measure elapsed time from the last accepted command to inhibited motion, and record whether the motor controller or the ROS supervisor caused the stop. Run <strong>30 induced dropouts</strong> as an initial experiment size rather than treating a clean run as proof of safety. Even zero observed late stops in that set would establish only what happened under those tested conditions. Put the fault procedure in <em>launch_testing</em> or a hardware test script so the next developer can repeat it after a controller change.</p>
<p>Keep security separate from this stop claim. <em>ROS_DOMAIN_ID</em> can help separate discovery domains, and SROS2 can configure DDS security, but neither enforces a physical stop when a command path fails. Mixing those concerns creates a costly review error: the team sees an access-control measure and assumes it has also addressed motion safety.</p>
<h2>Trend-led rewrites charge interest before they remove a fault</h2>
<p>Read <a href=/robotics-software-development-trends-for-smart-automation/>Robotics Software Development Trends for Smart Automation</a>, then ask which proposed change would eliminate a failure your tests have reproduced. Trend discussions can suggest useful approaches, but adopting an architecture because it is current gives a junior developer more interfaces to debug before the team knows which interface is failing.</p>
<p>Consider <strong>ROS 2 composable nodes versus separate processes</strong>. Composition can win when nodes exchange large messages on one computer and measured transfer overhead is a bottleneck; its cost is shared process fate and harder isolation when one component crashes. Separate processes can win when fault isolation and independent restarts matter more; their cost is extra interprocess communication and deployment configuration. Choose after a trace identifies the bottleneck or a fault test identifies the needed isolation. Otherwise, the rewrite risks changing failure behavior without improving it.</p>
<p>The same discipline applies before switching between Cyclone DDS and Fast DDS. Middleware differences can matter on a particular network, but changing implementations while also changing QoS and node layout makes the result difficult to attribute. Hold the launch configuration constant, run the same <em>rosbag2</em> input or repeatable hardware scenario, and compare the metric that prompted the change. Use <em>colcon test</em> and <em>pytest</em> for ordinary regressions, while keeping hardware fault results distinct from simulator results because a simulator cannot establish how a real actuator stops.</p>
<p>On your next task, take one motion-related topic and write down its publisher, subscriber, QoS policies, timestamp source, expiry condition, and stop owner. Run one dropout and save the log before proposing a new node or middleware setting. If you cannot identify who rejects an expired command, that is the first defect to fix.</p>
<p>The post <a href="https://deepfriedbytes.com/5-robotics-team-mistakes-that-cost-junior-devs-weeks/">5 Robotics Team Mistakes That Cost Junior Devs Weeks</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>2024 Crypto Runbooks Fail Production in 2026</title>
		<link>https://deepfriedbytes.com/2024-crypto-runbooks-fail-production-in-2026/</link>
		
		
		<pubDate>Thu, 22 Oct 2026 07:47:32 +0000</pubDate>
				<category><![CDATA[Cryptocurrencies]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[AI Web Solutions]]></category>
		<category><![CDATA[Blockchain]]></category>
		<category><![CDATA[Digital ecosystems]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/2024-crypto-runbooks-fail-production-in-2026/</guid>

					<description><![CDATA[<p>A wallet request can time out after it has been broadcast, leaving the caller unable to tell whether retrying will create a second transfer. Since 2024, delegated accounts and more varied transaction paths have made that uncertainty harder to contain. My position: a production wallet needs a durable intent record before it needs another signing integration. If no component owns recovery, scaling workers only scales the ambiguity. The old build-or-buy boundary no longer matches the recovery boundary I disagree with Your 2024 Crypto Build-vs-Buy Playbook Is Obsolete if its sourcing question is treated as the production decision: buying signing, RPC, or account-abstraction services does not buy an answer to “Did this customer request execute?” Your team still has to join the HTTP request, the signed operation, the broadcast attempt, and the eventual chain result into one recoverable record. That distinction has become sharper over the last two years. Ethereum’s Pectra upgrade activated on May 7, 2025—a network date, not a performance claim—and introduced EIP-7702, which lets an externally owned account delegate execution to code. An EIP-7702-capable wallet can therefore have behavior that an older “EOA means one simple transaction” runbook does not describe. ERC-4337 adds another operational path: a UserOperation goes through a bundler and an EntryPoint rather than through the ordinary EOA transaction submission path. EntryPoint v0.7 matters as a version boundary because tooling that serializes a UserOperation for a different EntryPoint version can produce an operation the intended path will not accept. These mechanisms do not make every wallet request interchangeable. EIP-5792’s wallet_sendCalls describes a wallet-facing call-batching interface, while eth_sendRawTransaction broadcasts an already signed EVM transaction; a successful response from either interface is not proof of final execution because inclusion and chain history can still change. A paymaster may also be involved in an ERC-4337 operation, so rebuilding an apparently identical request after a timeout can change its signed contents. Store the exact bytes or canonical operation fields that were submitted, not merely the API payload that started the process. I would not make a vendor webhook the system of record, because a delayed, duplicated, or missing webhook cannot establish what your API already promised the caller. Use webhooks as evidence to reconcile against receipts and your durable intent record. Keep the vendor’s request ID, the chain ID, the signer or smart-account address, and the transaction or UserOperation hash together; without that mapping, an on-call engineer must guess which customer action a chain event represents. Idempotency now has to survive a signer and a broadcaster At 10x users, your crypto wallet breaks at idempotency names a real failure mode, but I would not use traffic growth as the trigger for fixing it: one timeout between signing and persistence is enough to make a retry unsafe. The operational contract should be narrower and testable: the same authenticated request key and the same request body identify one wallet intent, while a reused key with a different body is rejected. That contract prevents your API from deliberately creating a second intent; it does not claim that a blockchain provides exactly-once execution. Put a uniqueness constraint around the intent before dispatching work. This PostgreSQL example runs in psql; its sample hash is a placeholder, so production code must compute the digest from a canonical request representation before insertion. CREATE TABLE wallet_intents ( tenant text NOT NULL, chain_id bigint NOT NULL, idem_key text NOT NULL, body_hash text NOT NULL, state text NOT NULL DEFAULT 'accepted', tx_hash text, PRIMARY KEY (tenant, chain_id, idem_key) ); INSERT INTO wallet_intents (tenant, chain_id, idem_key, body_hash) VALUES ('demo', 1, 'request-7', 'sha256:sample') ON CONFLICT (tenant, chain_id, idem_key) DO NOTHING; SELECT body_hash, state, tx_hash FROM wallet_intents WHERE tenant = 'demo' AND chain_id = 1 AND idem_key = 'request-7'; The insert makes concurrent acceptance of the same key converge on one row because PostgreSQL enforces the primary key. It does not finish the API contract by itself: after a conflict, compare the stored body_hash with the incoming hash and return a conflict if they differ. Treat that check and any state claim as database transactions, because two workers that merely read “accepted” can both decide to sign. A worker should claim a dispatch state with a conditional update, record the resulting signed payload or hash, and only then proceed through the broadcast state machine. Nonce management needs its own boundary. For an EOA, reserve a nonce per chain ID plus signer under database coordination; asking two RPC endpoints for eth_getTransactionCount with pending is not a lock, because their mempools need not contain the same transactions. Never allocate a fresh nonce just because eth_sendRawTransaction timed out: rebroadcasting the same signed bytes preserves the transaction hash, whereas a newly signed transaction can create a second payment. Replacement under EIP-1559 is a deliberate state transition with the same nonce and changed fee fields, not an automatic retry of the customer intent. For ERC-4337, track the UserOperation hash, EntryPoint address, smart account, and bundler submission separately from the business intent. Calling eth_sendUserOperation again with altered operation fields is not equivalent to replaying the same submission, because the operation identifier depends on what was signed. A receipt lookup, followed by a check of the relevant chain state, is the recovery path when the submitter loses its response. An accepted API response should mean “durably recorded,” not “paid,” because a signer or bundler can still reject work after acceptance. PostgreSQL beats a Redis lock for the source of truth There are two plausible defaults for coordinating wallet workers: a PostgreSQL intent row with a conditional claim, or a Redis SET key value NX PX lock. PostgreSQL wins when the claim and intent transition must remain auditable together, because its transaction can make the state change and ownership decision atomic. Its cost is write latency and contention on a busy signer or intent partition. Redis wins for short-lived, replaceable coordination where another durable record already defines the truth, because acquiring an expiring lock is simple and avoids a database write on that fast path. Its cost is that lock expiry or loss does not tell you whether the former holder signed or broadcast anything. For money-moving work, I would choose PostgreSQL as the authority and use Redis only as an optional scheduling optimization, because a worker crash must not turn an expired lock into permission to create a new transaction. Start with a 5-minute lease as a tuning input for a worker claim, but make expiry return the intent to reconciliation, not directly to signing. That rule matters when a process broadcasts successfully and dies before writing the response: the next worker must inspect the stored signed payload, receipt, nonce, and account state before deciding what action remains. The state machine should distinguish accepted, claimed, signed, broadcast, included, final, and needs_review. Those names are useful because they separate your own durable actions from observations about the chain. A transaction receipt showing inclusion is not the same thing as your application’s finality policy; a reorganization can remove a recently observed receipt. Configure that policy per chain rather than copying an Ethereum mainnet confirmation count to an L2, because the chains can expose different settlement and finality signals. Resist collapsing a stuck item into “failed” merely because the provider returned an error. An RPC timeout says that your client did not receive an answer, not that a node did not receive the bytes. Conversely, a nonce-too-low response does not by itself prove this intent succeeded, because another transaction from the signer may have consumed the nonce. The useful on-call question is whether the exact operation is known to be included, still eligible for broadcast, deliberately replaced, or genuinely unaccounted for. A recovery drill is more valuable than a throughput benchmark Put an end-to-end trace around intent acceptance, worker claim, signing, submission, and reconciliation with OpenTelemetry, but keep raw signing material out of spans because tracing backends are not key stores. Propagate an internal intent ID through Kubernetes jobs or queue messages; do not use the transaction hash as the only correlation ID, because it may not exist when the request first fails. With Prometheus, count state transitions and expose the age of the oldest non-final intent by chain and signer. Avoid a metric label containing every idempotency key, because unbounded label cardinality can overwhelm the monitoring system. As a proposed SLO rather than an observed result, target 99.9% availability for durably accepting valid wallet intents; measure chain completion separately because network fees and inclusion are outside the API acceptance boundary. Set an initial alert threshold when the oldest claimed or broadcast intent has made no verifiable progress for 15 minutes, then tune it against the chains and fee policies you operate. Alert on an aging state, not merely a low transaction count, because a quiet wallet can have no transactions while a single stranded transfer still needs intervention. Run a fault drill before raising Kubernetes Horizontal Pod Autoscaler worker limits. Inject a 30-second test timeout after submission but before the broadcaster records its response, then retry the same API key while two workers are active. The expected result is one intent row, no second nonce allocation for that intent, and a reconciliation path that either finds the submitted operation or marks the evidence insufficient for automated resubmission. That test catches a more consequential defect than a requests-per-second benchmark because it exercises the moment at which the caller, database, and chain can disagree. Budget two independently operated RPC providers as an operational starting point for investigation, not as a voting oracle. Comparing their receipt and pending-nonce views helps expose provider-local gaps, but disagreement should pause an irreversible decision rather than cause a third broadcast. Record which provider saw each submission and which block and receipt informed each transition, so the next on-call engineer can reconstruct the decision without relying on a dashboard’s current view. Prove one timeout cannot become two payments Start by selecting one production transfer path and finding the exact point where signed bytes can leave your process before their identity is durable. Add a fault-injection test at that boundary and retry the customer request with the same key. If the test creates a second nonce or operation, stop increasing worker concurrency: fix the intent record and reconciliation transition first. That is a release gate an on-call engineer can verify, not a promise that the network will never fail.</p>
<p>The post <a href="https://deepfriedbytes.com/2024-crypto-runbooks-fail-production-in-2026/">2024 Crypto Runbooks Fail Production in 2026</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A wallet request can time out after it has been broadcast, leaving the caller unable to tell whether retrying will create a second transfer. Since 2024, delegated accounts and more varied transaction paths have made that uncertainty harder to contain. My position: a production wallet needs a durable intent record before it needs another signing integration. If no component owns recovery, scaling workers only scales the ambiguity.</p>
<h2>The old build-or-buy boundary no longer matches the recovery boundary</h2>
<p>I disagree with <a href=/your-2024-crypto-build-vs-buy-playbook-is-obsolete/>Your 2024 Crypto Build-vs-Buy Playbook Is Obsolete</a> if its sourcing question is treated as the production decision: buying signing, RPC, or account-abstraction services does not buy an answer to “Did this customer request execute?” Your team still has to join the HTTP request, the signed operation, the broadcast attempt, and the eventual chain result into one recoverable record.</p>
<p>That distinction has become sharper over the last two years. Ethereum’s Pectra upgrade activated on May 7, 2025—a network date, not a performance claim—and introduced EIP-7702, which lets an externally owned account delegate execution to code. An EIP-7702-capable wallet can therefore have behavior that an older “EOA means one simple transaction” runbook does not describe. ERC-4337 adds another operational path: a UserOperation goes through a bundler and an EntryPoint rather than through the ordinary EOA transaction submission path. EntryPoint v0.7 matters as a version boundary because tooling that serializes a UserOperation for a different EntryPoint version can produce an operation the intended path will not accept.</p>
<p>These mechanisms do not make every wallet request interchangeable. EIP-5792’s <em>wallet_sendCalls</em> describes a wallet-facing call-batching interface, while <em>eth_sendRawTransaction</em> broadcasts an already signed EVM transaction; a successful response from either interface is not proof of final execution because inclusion and chain history can still change. A paymaster may also be involved in an ERC-4337 operation, so rebuilding an apparently identical request after a timeout can change its signed contents. Store the exact bytes or canonical operation fields that were submitted, not merely the API payload that started the process.</p>
<p>I would <strong>not</strong> make a vendor webhook the system of record, because a delayed, duplicated, or missing webhook cannot establish what your API already promised the caller. Use webhooks as evidence to reconcile against receipts and your durable intent record. Keep the vendor’s request ID, the chain ID, the signer or smart-account address, and the transaction or UserOperation hash together; without that mapping, an on-call engineer must guess which customer action a chain event represents.</p>
<h2>Idempotency now has to survive a signer and a broadcaster</h2>
<p><a href=/at-10x-users-your-crypto-wallet-breaks-at-idempotency/>At 10x users, your crypto wallet breaks at idempotency</a> names a real failure mode, but I would not use traffic growth as the trigger for fixing it: one timeout between signing and persistence is enough to make a retry unsafe. The operational contract should be narrower and testable: the same authenticated request key and the same request body identify one wallet intent, while a reused key with a different body is rejected. That contract prevents your API from deliberately creating a second intent; it does not claim that a blockchain provides exactly-once execution.</p>
<p>Put a uniqueness constraint around the intent before dispatching work. This PostgreSQL example runs in <em>psql</em>; its sample hash is a placeholder, so production code must compute the digest from a canonical request representation before insertion.</p>
<pre>CREATE TABLE wallet_intents (
  tenant text NOT NULL, chain_id bigint NOT NULL,
  idem_key text NOT NULL, body_hash text NOT NULL,
  state text NOT NULL DEFAULT 'accepted',
  tx_hash text, PRIMARY KEY (tenant, chain_id, idem_key)
);
INSERT INTO wallet_intents (tenant, chain_id, idem_key, body_hash)
VALUES ('demo', 1, 'request-7', 'sha256:sample')
ON CONFLICT (tenant, chain_id, idem_key) DO NOTHING;
SELECT body_hash, state, tx_hash FROM wallet_intents
WHERE tenant = 'demo' AND chain_id = 1 AND idem_key = 'request-7';</pre>
<p>The insert makes concurrent acceptance of the same key converge on one row because PostgreSQL enforces the primary key. It does <strong>not</strong> finish the API contract by itself: after a conflict, compare the stored <em>body_hash</em> with the incoming hash and return a conflict if they differ. Treat that check and any state claim as database transactions, because two workers that merely read “accepted” can both decide to sign. A worker should claim a dispatch state with a conditional update, record the resulting signed payload or hash, and only then proceed through the broadcast state machine.</p>
<p>Nonce management needs its own boundary. For an EOA, reserve a nonce per <em>chain ID plus signer</em> under database coordination; asking two RPC endpoints for <em>eth_getTransactionCount</em> with <em>pending</em> is not a lock, because their mempools need not contain the same transactions. Never allocate a fresh nonce just because <em>eth_sendRawTransaction</em> timed out: rebroadcasting the same signed bytes preserves the transaction hash, whereas a newly signed transaction can create a second payment. Replacement under EIP-1559 is a deliberate state transition with the same nonce and changed fee fields, not an automatic retry of the customer intent.</p>
<p>For ERC-4337, track the UserOperation hash, EntryPoint address, smart account, and bundler submission separately from the business intent. Calling <em>eth_sendUserOperation</em> again with altered operation fields is not equivalent to replaying the same submission, because the operation identifier depends on what was signed. A receipt lookup, followed by a check of the relevant chain state, is the recovery path when the submitter loses its response. An accepted API response should mean “durably recorded,” not “paid,” because a signer or bundler can still reject work after acceptance.</p>
<h2>PostgreSQL beats a Redis lock for the source of truth</h2>
<p>There are two plausible defaults for coordinating wallet workers: a PostgreSQL intent row with a conditional claim, or a Redis <em>SET key value NX PX</em> lock. PostgreSQL wins when the claim and intent transition must remain auditable together, because its transaction can make the state change and ownership decision atomic. Its cost is write latency and contention on a busy signer or intent partition. Redis wins for short-lived, replaceable coordination where another durable record already defines the truth, because acquiring an expiring lock is simple and avoids a database write on that fast path. Its cost is that lock expiry or loss does not tell you whether the former holder signed or broadcast anything.</p>
<p>For money-moving work, I would choose PostgreSQL as the authority and use Redis only as an optional scheduling optimization, because a worker crash must not turn an expired lock into permission to create a new transaction. Start with a <strong>5-minute lease as a tuning input</strong> for a worker claim, but make expiry return the intent to <em>reconciliation</em>, not directly to <em>signing</em>. That rule matters when a process broadcasts successfully and dies before writing the response: the next worker must inspect the stored signed payload, receipt, nonce, and account state before deciding what action remains.</p>
<p>The state machine should distinguish <em>accepted</em>, <em>claimed</em>, <em>signed</em>, <em>broadcast</em>, <em>included</em>, <em>final</em>, and <em>needs_review</em>. Those names are useful because they separate your own durable actions from observations about the chain. A transaction receipt showing inclusion is not the same thing as your application’s finality policy; a reorganization can remove a recently observed receipt. Configure that policy per chain rather than copying an Ethereum mainnet confirmation count to an L2, because the chains can expose different settlement and finality signals.</p>
<p>Resist collapsing a stuck item into “failed” merely because the provider returned an error. An RPC timeout says that your client did not receive an answer, not that a node did not receive the bytes. Conversely, a nonce-too-low response does not by itself prove this intent succeeded, because another transaction from the signer may have consumed the nonce. The useful on-call question is whether the exact operation is known to be included, still eligible for broadcast, deliberately replaced, or genuinely unaccounted for.</p>
<h2>A recovery drill is more valuable than a throughput benchmark</h2>
<p>Put an end-to-end trace around intent acceptance, worker claim, signing, submission, and reconciliation with OpenTelemetry, but keep raw signing material out of spans because tracing backends are not key stores. Propagate an internal intent ID through Kubernetes jobs or queue messages; do not use the transaction hash as the only correlation ID, because it may not exist when the request first fails. With Prometheus, count state transitions and expose the age of the oldest non-final intent by chain and signer. Avoid a metric label containing every idempotency key, because unbounded label cardinality can overwhelm the monitoring system.</p>
<p>As a <strong>proposed SLO rather than an observed result</strong>, target 99.9% availability for durably accepting valid wallet intents; measure chain completion separately because network fees and inclusion are outside the API acceptance boundary. Set an <strong>initial alert threshold</strong> when the oldest claimed or broadcast intent has made no verifiable progress for 15 minutes, then tune it against the chains and fee policies you operate. Alert on an aging state, not merely a low transaction count, because a quiet wallet can have no transactions while a single stranded transfer still needs intervention.</p>
<p>Run a fault drill before raising Kubernetes Horizontal Pod Autoscaler worker limits. Inject a <strong>30-second test timeout</strong> after submission but before the broadcaster records its response, then retry the same API key while two workers are active. The expected result is one intent row, no second nonce allocation for that intent, and a reconciliation path that either finds the submitted operation or marks the evidence insufficient for automated resubmission. That test catches a more consequential defect than a requests-per-second benchmark because it exercises the moment at which the caller, database, and chain can disagree.</p>
<p>Budget <strong>two independently operated RPC providers as an operational starting point</strong> for investigation, not as a voting oracle. Comparing their receipt and pending-nonce views helps expose provider-local gaps, but disagreement should pause an irreversible decision rather than cause a third broadcast. Record which provider saw each submission and which block and receipt informed each transition, so the next on-call engineer can reconstruct the decision without relying on a dashboard’s current view.</p>
<h2>Prove one timeout cannot become two payments</h2>
<p>Start by selecting one production transfer path and finding the exact point where signed bytes can leave your process before their identity is durable. Add a fault-injection test at that boundary and retry the customer request with the same key. If the test creates a second nonce or operation, stop increasing worker concurrency: fix the intent record and reconciliation transition first. That is a release gate an on-call engineer can verify, not a promise that the network will never fail.</p>
<p>The post <a href="https://deepfriedbytes.com/2024-crypto-runbooks-fail-production-in-2026/">2024 Crypto Runbooks Fail Production in 2026</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>CTO, Score Robotics Stacks Before You Build or Buy</title>
		<link>https://deepfriedbytes.com/cto-score-robotics-stacks-before-you-build-or-buy/</link>
		
		
		<pubDate>Tue, 13 Oct 2026 09:41:24 +0000</pubDate>
				<category><![CDATA[Robotics]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[Machine Learning Apps]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/cto-score-robotics-stacks-before-you-build-or-buy/</guid>

					<description><![CDATA[<p>A vision API can win a demo and lose the procurement decision: the invoice may be predictable while data retention, camera drift, and migration remain untested. For a first production app, I would buy inference only from a vendor that passes a priced exit test. If the contract or interface prevents that test, I would start with a licensed model running in-house, even if its initial accuracy is lower. A vendor trial should start with an exit test, not a leaderboard Before asking vendors for accuracy claims, write down what you would need to replace them. That means access to the original images where your rights permit it, human-reviewed labels, model outputs, confidence values, timestamps, and enough request metadata to replay a decision. A supplier that returns only a category name makes replacement harder because you cannot reconstruct thresholds or diagnose disagreements after the fact. Ask each candidate to process the same held-out images through the interface you would actually deploy. A useful trial set includes ordinary images, low-light captures, motion blur, empty frames, and examples from every camera type you expect to support. Set 2,000 labeled images as an initial trial size to tune, then increase it if important slices contain too few examples to estimate error rates. Keep near-duplicate frames from the same event together when splitting data; otherwise, a test can reward recognition of a scene rather than performance on a new one. Record results by slice, not just in one aggregate score. A measured 95% overall agreement in your trial would still be unacceptable if one camera’s false-accept rate exceeded the limit you set for that camera. For each vendor, report false accepts and false rejects at a threshold selected on a separate development set. Do not move the threshold after inspecting held-out results, because doing so turns the test set into another tuning set. The simplest comparison artifact is a JSON Lines file with one reviewed truth value and one prediction per candidate. The following script runs with Python’s standard library; give it an eval.jsonl file whose rows contain slice, truth, api, and onnx fields. Use the same categorical output vocabulary for both candidates. python3 - &#60;&#60;'PY' import json from collections import defaultdict rows = [json.loads(s) for s in open("eval.jsonl") if s.strip()] by = defaultdict(lambda: [0, 0]) for r in rows: for name in ("api", "onnx"): key = (name, r["slice"]) by[key][0] += int(r[name] != r["truth"]) by[key][1] += 1 for (name, sl), (errors, n) in sorted(by.items()): print(name, sl, f"errors={errors}/{n}", f"rate={errors/n:.3f}") PY That script counts disagreements, not detection quality; use intersection over union and class-specific precision and recall when the app depends on bounding boxes. Make a human review procedure part of the trial, since “ground truth” produced by one of the competing services would favor that service. I would not accept a vendor’s public benchmark as the procurement test, because its image distribution, annotation rules, and decision threshold are unlikely to match yours. Amazon Rekognition wins on setup; ONNX Runtime wins on control Compare two concrete options: Amazon Rekognition DetectLabels as the bought service and ONNX Runtime 1.20 running a properly licensed vision model as the in-house path. Rekognition wins when its labels match your decision, request volume is uncertain, and you need an operational endpoint quickly; you pay per applicable API request, plus engineering time for mapping labels, handling failures, and reviewing AWS service terms. ONNX Runtime wins when you need a fixed preprocessing path, local inference, or tighter control over output formats; you pay for model selection, compute, deployment, monitoring, and an owner for every regression. Neither option should receive credit for work it does not perform. Rekognition’s MinConfidence parameter can filter returned labels, but a setting of 80 is a threshold to tune against your error costs, not proof that returned results are correct 80% of the time. ONNX Runtime executes a model efficiently, but it does not supply an appropriate model, usable training data, or a license to ship the weights. If you consider an Ultralytics YOLO11 export, review its AGPL-3.0 and enterprise licensing paths with counsel before treating an ONNX file as unrestricted. Use OpenCV 4.10 to make resizing, color conversion, and image orientation explicit in the trial harness. Record the exported ONNX opset, such as 17 where compatible, and the ONNX Runtime execution provider, such as CPUExecutionProvider or CUDAExecutionProvider; these settings matter because a benchmark from one preprocessing and hardware path does not establish production latency on another. If deployment needs NVIDIA TensorRT 10 or Triton Inference Server 2.50, test the exported graph there before approving a model, since a successful notebook inference does not prove the target runtime accepts every operation. Set a 200 ms p95 server-side inference budget as a provisional design target if that fits your app, and measure it under the concurrency you intend to support. Separate decode, preprocessing, inference, and network time using OpenTelemetry traces; a vendor may have fast model execution while upload and response transfer dominate the user-visible wait. Track a Prometheus latency histogram rather than an average, because the p95 estimated with histogram_quantile() will expose slow requests that an average conceals. Read Custom Vision Model or Pretrained API Which Fits Your First App before soliciting model proposals. I would make reversibility the earlier decision, because an excellent model is a poor purchase if you cannot reproduce its inputs and outputs elsewhere. The contract can erase an apparent accuracy advantage Run the technical trial in parallel with a contract review. Ask whether uploaded images are retained, whether they can be used to improve the supplier’s services, where processing occurs, how deletion is verified, and what changes require notice. Get the answers in the governing agreement or service terms, not only in a sales email, because your team will need enforceable terms if policies change. Price three years of use with a volume model you can explain. For example, treat 1 million images per month as a planning scenario to revise, and include retries, duplicate submissions, minimum commitments, storage, outbound transfer, and human review. Price the in-house alternative with compute, a spare-capacity allowance, model maintenance, and on-call time. Compare both at low and high volume: per-request pricing can be attractive while demand is uncertain, whereas owned capacity becomes easier to justify when utilization is steady. Then ask the vendor to perform an exit drill. Give its team a written request for your available images, annotations, outputs, and configuration, and time how long it takes to receive a usable export. Require stable identifiers so exported decisions can be joined to your records. A promise that “data is portable” is weak evidence because portability depends on file formats, identifiers, rights, and the effort needed to replay decisions. Read CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated after pricing that exit. A revised build-versus-buy argument is useful, but I would reject any shortcut from “buying is faster” to “this supplier is replaceable”; the second claim requires an export and replay test. Open-source choices need a contract review too. Check model-weight terms separately from framework terms, and retain a software bill of materials using SPDX 3.0 or CycloneDX 1.6 if your organization supports those formats. ONNX Runtime’s license does not grant rights to every model loaded into it, so record the provenance of weights, training datasets where known, and any commercial-use restrictions before counting self-hosting as the safer option. A purchase decision should have a failure gate and an owner Turn the trial into a short approval memo with gates that can fail. One gate should cover error by image slice; another should cover latency under load; a third should cover deletion, export, and contract terms. Assign an owner to each gate, since an unowned exception is likely to become an implicit approval. Set a 30-day shadow period as an initial operating window to adjust, during which the candidate produces results without controlling the final decision. During shadow operation, log the model or API version, request identifier, preprocessing version, threshold, and human override. Sample disagreements for review rather than sending every image back to an annotator, because review cost can overwhelm a low per-request inference price. Give reviewers a documented rule for ambiguous images; inconsistent labels can make either option appear unstable even when its predictions have not changed. Write the rollback trigger before launch. For example, choose a 1% false-accept ceiling for a high-cost decision only if your product owner agrees that it reflects the real cost of errors; otherwise, set a different value and document why. Trigger rollback on a sustained breach in the affected slice, not merely on an overall average. Keep the previous inference path callable until you have tested rollback, because a feature flag that points to a retired endpoint is not a recovery plan. Finally, decide who can change the system. A managed API still needs an owner for changed label taxonomies, quotas, authentication, and vendor notices. A self-hosted model needs an owner for patches, capacity, model updates, and drift investigations. Approve the option whose ongoing duties have named people and budget, since an apparently cheaper path can become expensive when those duties surface during an incident. The first approval artifact should be a paired test set Tomorrow, select one real decision your app makes and collect its first reviewed, consent-appropriate image sample. Define the output schema and error cost before inviting either vendor demos or model proposals. Then send both candidates the same test and ask for a priced export. The winner is the option your team can operate and replace, not the one with the most persuasive demo.</p>
<p>The post <a href="https://deepfriedbytes.com/cto-score-robotics-stacks-before-you-build-or-buy/">CTO, Score Robotics Stacks Before You Build or Buy</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A vision API can win a demo and lose the procurement decision: the invoice may be predictable while data retention, camera drift, and migration remain untested. For a first production app, I would buy inference only from a vendor that passes a priced exit test. If the contract or interface prevents that test, I would start with a licensed model running in-house, even if its initial accuracy is lower.</p>
<h2>A vendor trial should start with an exit test, not a leaderboard</h2>
<p>Before asking vendors for accuracy claims, write down what you would need to replace them. That means access to the original images where your rights permit it, human-reviewed labels, model outputs, confidence values, timestamps, and enough request metadata to replay a decision. A supplier that returns only a category name makes replacement harder because you cannot reconstruct thresholds or diagnose disagreements after the fact.</p>
<p>Ask each candidate to process the same held-out images through the interface you would actually deploy. A useful trial set includes ordinary images, low-light captures, motion blur, empty frames, and examples from every camera type you expect to support. Set <strong>2,000 labeled images</strong> as an initial trial size to tune, then increase it if important slices contain too few examples to estimate error rates. Keep near-duplicate frames from the same event together when splitting data; otherwise, a test can reward recognition of a scene rather than performance on a new one.</p>
<p>Record results by slice, not just in one aggregate score. A measured <strong>95% overall agreement</strong> in your trial would still be unacceptable if one camera’s false-accept rate exceeded the limit you set for that camera. For each vendor, report false accepts and false rejects at a threshold selected on a separate development set. Do not move the threshold after inspecting held-out results, because doing so turns the test set into another tuning set.</p>
<p>The simplest comparison artifact is a JSON Lines file with one reviewed truth value and one prediction per candidate. The following script runs with Python’s standard library; give it an <code>eval.jsonl</code> file whose rows contain <code>slice</code>, <code>truth</code>, <code>api</code>, and <code>onnx</code> fields. Use the same categorical output vocabulary for both candidates.</p>
<pre>python3 - &lt;&lt;'PY'
import json
from collections import defaultdict
rows = [json.loads(s) for s in open("eval.jsonl") if s.strip()]
by = defaultdict(lambda: [0, 0])
for r in rows:
    for name in ("api", "onnx"):
        key = (name, r["slice"])
        by[key][0] += int(r[name] != r["truth"])
        by[key][1] += 1
for (name, sl), (errors, n) in sorted(by.items()):
    print(name, sl, f"errors={errors}/{n}", f"rate={errors/n:.3f}")
PY</pre>
<p>That script counts disagreements, not detection quality; use intersection over union and class-specific precision and recall when the app depends on bounding boxes. Make a human review procedure part of the trial, since “ground truth” produced by one of the competing services would favor that service. I would <strong>not</strong> accept a vendor’s public benchmark as the procurement test, because its image distribution, annotation rules, and decision threshold are unlikely to match yours.</p>
<h2>Amazon Rekognition wins on setup; ONNX Runtime wins on control</h2>
<p>Compare two concrete options: <strong>Amazon Rekognition DetectLabels</strong> as the bought service and <strong>ONNX Runtime 1.20</strong> running a properly licensed vision model as the in-house path. Rekognition wins when its labels match your decision, request volume is uncertain, and you need an operational endpoint quickly; you pay per applicable API request, plus engineering time for mapping labels, handling failures, and reviewing AWS service terms. ONNX Runtime wins when you need a fixed preprocessing path, local inference, or tighter control over output formats; you pay for model selection, compute, deployment, monitoring, and an owner for every regression.</p>
<p>Neither option should receive credit for work it does not perform. Rekognition’s <code>MinConfidence</code> parameter can filter returned labels, but a setting of <strong>80</strong> is a threshold to tune against your error costs, not proof that returned results are correct 80% of the time. ONNX Runtime executes a model efficiently, but it does not supply an appropriate model, usable training data, or a license to ship the weights. If you consider an Ultralytics YOLO11 export, review its AGPL-3.0 and enterprise licensing paths with counsel before treating an ONNX file as unrestricted.</p>
<p>Use OpenCV 4.10 to make resizing, color conversion, and image orientation explicit in the trial harness. Record the exported ONNX opset, such as <strong>17</strong> where compatible, and the ONNX Runtime execution provider, such as <code>CPUExecutionProvider</code> or <code>CUDAExecutionProvider</code>; these settings matter because a benchmark from one preprocessing and hardware path does not establish production latency on another. If deployment needs NVIDIA TensorRT 10 or Triton Inference Server 2.50, test the exported graph there before approving a model, since a successful notebook inference does not prove the target runtime accepts every operation.</p>
<p>Set a <strong>200 ms p95</strong> server-side inference budget as a provisional design target if that fits your app, and measure it under the concurrency you intend to support. Separate decode, preprocessing, inference, and network time using OpenTelemetry traces; a vendor may have fast model execution while upload and response transfer dominate the user-visible wait. Track a Prometheus latency histogram rather than an average, because the p95 estimated with <code>histogram_quantile()</code> will expose slow requests that an average conceals.</p>
<p>Read <a href=/custom-vision-model-or-pretrained-api-which-fits-your-first-app/>Custom Vision Model or Pretrained API Which Fits Your First App</a> before soliciting model proposals. I would make reversibility the earlier decision, because an excellent model is a poor purchase if you cannot reproduce its inputs and outputs elsewhere.</p>
<h2>The contract can erase an apparent accuracy advantage</h2>
<p>Run the technical trial in parallel with a contract review. Ask whether uploaded images are retained, whether they can be used to improve the supplier’s services, where processing occurs, how deletion is verified, and what changes require notice. Get the answers in the governing agreement or service terms, not only in a sales email, because your team will need enforceable terms if policies change.</p>
<p>Price three years of use with a volume model you can explain. For example, treat <strong>1 million images per month</strong> as a planning scenario to revise, and include retries, duplicate submissions, minimum commitments, storage, outbound transfer, and human review. Price the in-house alternative with compute, a spare-capacity allowance, model maintenance, and on-call time. Compare both at low and high volume: per-request pricing can be attractive while demand is uncertain, whereas owned capacity becomes easier to justify when utilization is steady.</p>
<p>Then ask the vendor to perform an exit drill. Give its team a written request for your available images, annotations, outputs, and configuration, and time how long it takes to receive a usable export. Require stable identifiers so exported decisions can be joined to your records. A promise that “data is portable” is weak evidence because portability depends on file formats, identifiers, rights, and the effort needed to replay decisions.</p>
<p>Read <a href=/ctos-your-2024-genai-build-vs-buy-playbook-is-outdated/>CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated</a> after pricing that exit. A revised build-versus-buy argument is useful, but I would reject any shortcut from “buying is faster” to “this supplier is replaceable”; the second claim requires an export and replay test.</p>
<p>Open-source choices need a contract review too. Check model-weight terms separately from framework terms, and retain a software bill of materials using SPDX 3.0 or CycloneDX 1.6 if your organization supports those formats. ONNX Runtime’s license does not grant rights to every model loaded into it, so record the provenance of weights, training datasets where known, and any commercial-use restrictions before counting self-hosting as the safer option.</p>
<h2>A purchase decision should have a failure gate and an owner</h2>
<p>Turn the trial into a short approval memo with gates that can fail. One gate should cover error by image slice; another should cover latency under load; a third should cover deletion, export, and contract terms. Assign an owner to each gate, since an unowned exception is likely to become an implicit approval. Set a <strong>30-day</strong> shadow period as an initial operating window to adjust, during which the candidate produces results without controlling the final decision.</p>
<p>During shadow operation, log the model or API version, request identifier, preprocessing version, threshold, and human override. Sample disagreements for review rather than sending every image back to an annotator, because review cost can overwhelm a low per-request inference price. Give reviewers a documented rule for ambiguous images; inconsistent labels can make either option appear unstable even when its predictions have not changed.</p>
<p>Write the rollback trigger before launch. For example, choose a <strong>1% false-accept ceiling</strong> for a high-cost decision only if your product owner agrees that it reflects the real cost of errors; otherwise, set a different value and document why. Trigger rollback on a sustained breach in the affected slice, not merely on an overall average. Keep the previous inference path callable until you have tested rollback, because a feature flag that points to a retired endpoint is not a recovery plan.</p>
<p>Finally, decide who can change the system. A managed API still needs an owner for changed label taxonomies, quotas, authentication, and vendor notices. A self-hosted model needs an owner for patches, capacity, model updates, and drift investigations. Approve the option whose ongoing duties have named people and budget, since an apparently cheaper path can become expensive when those duties surface during an incident.</p>
<h2>The first approval artifact should be a paired test set</h2>
<p>Tomorrow, select one real decision your app makes and collect its first reviewed, consent-appropriate image sample. Define the output schema and error cost before inviting either vendor demos or model proposals. Then send both candidates the same test and ask for a priced export. The winner is the option your team can operate and replace, not the one with the most persuasive demo.</p>
<p>The post <a href="https://deepfriedbytes.com/cto-score-robotics-stacks-before-you-build-or-buy/">CTO, Score Robotics Stacks Before You Build or Buy</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>RAG vs Fine-Tuning in Your First GenAI Backend</title>
		<link>https://deepfriedbytes.com/rag-vs-fine-tuning-in-your-first-genai-backend/</link>
		
		
		<pubDate>Tue, 06 Oct 2026 09:58:34 +0000</pubDate>
				<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[AI Web Solutions]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/rag-vs-fine-tuning-in-your-first-genai-backend/</guid>

					<description><![CDATA[<p>A coding assistant does not need to understand an entire repository before it can answer a developer’s question. It needs to find the right code, show where it found it, and avoid presenting stale guesses as facts. For a first implementation, I would choose live repository search over a vector index. Embeddings become worthwhile when real questions repeatedly defeat symbol and text search—not simply because the feature uses generative AI. Live repository search is the better default for code questions Consider a developer asking, “Where do we reject an expired session token?” The likely answer has a function name, an error string, a test, or a call site that ripgrep can find in the current checkout. A coding assistant can search those candidates, read bounded excerpts, and ask the model to explain the path from request handling to rejection. This approach wins when questions mention identifiers, configuration keys, endpoints, stack traces, or behavior with recognizable vocabulary, because those clues already exist as searchable text. Live search also gives a backend developer a familiar consistency model: the assistant reads the same revision the developer is inspecting. git supplies the commit SHA; rg supplies file paths and line numbers; Tree-sitter can expand a hit to its containing function when a few matching lines are insufficient. No background indexing job has to catch up after a merge. Treat Generative AI for Software Development: Practical Use Cases as a menu of candidate tasks rather than evidence that every coding task needs an embeddings index. The first version can be deliberately small. This Python 3 script searches a checked-out repository for a literal phrase, excludes two common dependency directories, and prints a bounded set of matches. Run it from the repository root after installing ripgrep: #!/usr/bin/env python3 import subprocess import sys if len(sys.argv) != 2: raise SystemExit("usage: python3 search.py PHRASE") cmd = ["rg", "-n", "-F", "-m", "2", "--glob", "!vendor/**", "--glob", "!node_modules/**", "--", sys.argv[1], "."] result = subprocess.run(cmd, text=True, capture_output=True) if result.returncode not in (0, 1): raise SystemExit(result.stderr) print("\n".join(result.stdout.splitlines()[:40])) Here, -F would normally be written as an inline flag, but the important behavior is that ripgrep treats the query as a literal phrase rather than a regular expression. The 40-line output cap is a starting limit to tune against answer quality and prompt size, not a measured optimum. In a service, avoid collecting every match before applying that cap: search within authorized paths, stream results, and stop once enough distinct files have been inspected. Add a second pass for call sites or tests only when the first pass leaves an unresolved question. I would not embed the entire repository on day one, because an index introduces update, deletion, and access-control work before there is evidence that lexical search misses important answers. Live search has costs too: repeated disk reads, poor matches for questions that use different words from the code, and the need to rank noisy hits. Those costs are visible in a request trace, which makes them easier to justify or fix than an index built on assumption. Vector retrieval wins when developers ask about concepts the code never names Now change the question to, “Why can a cancelled job still produce an invoice?” The relevant implementation may be spread across a queue consumer, a state transition, and an event handler without using the words “cancelled” and “invoice” together. A preindexed vector search can surface semantically related passages that literal matching misses. It is more likely to win across several repositories, long design documents, or renamed subsystems, because users cannot reliably guess the identifiers that tie those sources together. The competing approaches have distinct bills. Live search with ripgrep wins for fresh, identifier-rich questions and costs request-time I/O plus ranking logic. Preindexed retrieval with pgvector wins when conceptual discovery repeatedly matters and costs embedding generation, storage, reindexing, permission filtering, and investigation of stale results. A vector hit is only a candidate, because similarity does not establish that a passage describes the current behavior. If you add an index, make its unit of retrieval useful to a programmer. Chunking at arbitrary character boundaries can separate a condition from the branch it guards, so prefer a function or similarly coherent unit identified with Tree-sitter. Store repository, path, revision, symbol, and access scope beside each chunk. PostgreSQL with pgvector can hold embeddings and metadata together; an HNSW index can accelerate approximate nearest-neighbor search, although its recall and query cost still need testing. The hnsw.ef_search setting is a search-effort knob to tune rather than a value to copy from another workload. Do not force a choice between semantic and exact matches inside the indexed approach. A developer who pastes SessionExpiredError expects the exact symbol to rank highly, so combine vector candidates with BM25 results from a search engine such as Elasticsearch, then rerank the merged set. An initial top-k of 8 is an experiment setting: increase it if relevant files are missed, or decrease it if excerpts crowd out the question and answer. Keep the final context tied to a revision so the model can cite code the developer can still open. I agree with Your GenAI Feature Scales Until Retrieval and Latency Collide that retrieval latency becomes a production constraint, but index freshness deserves equal attention because a fast answer from yesterday’s code can be worse than a slower answer from the current commit. The vector approach earns its extra machinery only when it improves grounded answers under both constraints. Answer quality, not search relevance alone, should decide the winner A search result can look relevant and still lead the model to the wrong conclusion. Build an evaluation set from questions developers actually ask in code review, incident follow-up, and onboarding. For each question, record the revision, acceptable source locations, and what a correct answer must say. A 30-question pilot is a planning size for the first comparison, not a statistically persuasive benchmark; its purpose is to expose failure modes before anyone optimizes a leaderboard number. Run both retrieval approaches against the same repository revisions and the same answer model. Count whether a necessary file appears in the retrieved set using Recall@k, and use MRR to notice when it appears only after several distractions. Neither metric says whether the final explanation is sound, so also check whether the answer cites lines that support its claims, distinguishes tests from production behavior, and says “I cannot establish that” when the evidence is incomplete. Reviewers should mark a correct conclusion with an unsupported citation as a failure, because the assistant has given the developer no reliable way to verify it. Include adversarially ordinary questions: a symbol renamed last week, documentation that contradicts implementation, and two functions with similar names in different services. Re-run the set after a merge to see whether indexed results still point at the intended revision. If vector retrieval improves conceptual questions but regresses exact-symbol questions, add hybrid ranking before declaring it the winner; that fix targets the observed failure rather than replacing a useful search method wholesale. Measure the request path with OpenTelemetry spans for search, excerpt loading, model input preparation, and generation. Use p95 latency rather than only the mean, because slow repository reads or index queries affect the developer waiting on an answer even when most requests are quick. An 800 ms retrieval budget is an example service objective to adjust after measuring your own environment, not a claim about either approach’s speed. Record input tokens and retrieval calls as well, since an apparently cheap search can become expensive when it sends large, repetitive excerpts to the model. The decision rule should be hard to game: adopt the index for a question class only if it raises supported-answer quality without breaching that class’s latency and freshness requirements. Keep live search available for exact identifiers even after adding vectors, because a semantic ranking model has no reason to treat an exact error code as more important than a nearby conceptual match unless the system explicitly tells it to. Permissions and freshness can overturn a retrieval win A repository assistant is also an authorization system. Live search should run only over paths the requester may read; an indexed service must apply the same restriction before returning candidates. Filtering forbidden excerpts after a model has seen them is too late, because the answer can disclose their content. Store access metadata with indexed chunks, invalidate it when permissions change, and test that an unauthorized user cannot retrieve a restricted file through either exact or semantic queries. Freshness needs an explicit contract. Suppose an index is allowed to lag the main branch by 15 minutes; treat that as a proposed policy ceiling, not a property of the indexing software. Attach the indexed commit SHA to every result and compare it with the requested revision. If they differ beyond the permitted lag, fall back to live search or state plainly that the answer uses an older revision. Silent fallback to stale context makes a code explanation difficult to debug, because the developer and assistant may be reasoning about different programs. Retrieved repository text is untrusted input, even when it comes from a file the developer can read. A comment can tell an assistant to ignore instructions or reveal credentials; it remains source material, not an instruction from the user. Keep tool permissions separate from the language model’s retrieved context, and avoid passing secrets or whole environment files merely because they matched a search. These controls apply to both approaches, while a persistent index adds the obligation to remove deleted or newly restricted content from stored copies. Start by collecting real developer questions and implementing bounded, permission-aware live search at a pinned commit. Ask reviewers to mark the files and lines each answer actually needed. Once that baseline exists, test an indexed retriever on the questions it misses. If vectors cannot produce more supported answers at an acceptable freshness and latency cost, keep the simpler system and improve its ranking instead.</p>
<p>The post <a href="https://deepfriedbytes.com/rag-vs-fine-tuning-in-your-first-genai-backend/">RAG vs Fine-Tuning in Your First GenAI Backend</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A coding assistant does not need to understand an entire repository before it can answer a developer’s question. It needs to find the right code, show where it found it, and avoid presenting stale guesses as facts. For a first implementation, I would choose live repository search over a vector index. Embeddings become worthwhile when real questions repeatedly defeat symbol and text search—not simply because the feature uses generative AI.</p>
<h2>Live repository search is the better default for code questions</h2>
<p>Consider a developer asking, “Where do we reject an expired session token?” The likely answer has a function name, an error string, a test, or a call site that <strong>ripgrep</strong> can find in the current checkout. A coding assistant can search those candidates, read bounded excerpts, and ask the model to explain the path from request handling to rejection. This approach wins when questions mention identifiers, configuration keys, endpoints, stack traces, or behavior with recognizable vocabulary, because those clues already exist as searchable text.</p>
<p>Live search also gives a backend developer a familiar consistency model: the assistant reads the same revision the developer is inspecting. <strong>git</strong> supplies the commit SHA; <strong>rg</strong> supplies file paths and line numbers; <strong>Tree-sitter</strong> can expand a hit to its containing function when a few matching lines are insufficient. No background indexing job has to catch up after a merge. Treat <a href=/generative-ai-for-software-development-practical-use-cases/>Generative AI for Software Development: Practical Use Cases</a> as a menu of candidate tasks rather than evidence that every coding task needs an embeddings index.</p>
<p>The first version can be deliberately small. This Python 3 script searches a checked-out repository for a literal phrase, excludes two common dependency directories, and prints a bounded set of matches. Run it from the repository root after installing <strong>ripgrep</strong>:</p>
<pre>#!/usr/bin/env python3
import subprocess
import sys
if len(sys.argv) != 2:
    raise SystemExit("usage: python3 search.py PHRASE")
cmd = ["rg", "-n", "-F", "-m", "2", "--glob", "!vendor/**",
       "--glob", "!node_modules/**", "--", sys.argv[1], "."]
result = subprocess.run(cmd, text=True, capture_output=True)
if result.returncode not in (0, 1):
    raise SystemExit(result.stderr)
print("\n".join(result.stdout.splitlines()[:40]))</pre>
<p>Here, <code>-F</code> would normally be written as an inline flag, but the important behavior is that ripgrep treats the query as a literal phrase rather than a regular expression. The <strong>40-line output cap</strong> is a starting limit to tune against answer quality and prompt size, not a measured optimum. In a service, avoid collecting every match before applying that cap: search within authorized paths, stream results, and stop once enough distinct files have been inspected. Add a second pass for call sites or tests only when the first pass leaves an unresolved question.</p>
<p>I would <strong>not</strong> embed the entire repository on day one, because an index introduces update, deletion, and access-control work before there is evidence that lexical search misses important answers. Live search has costs too: repeated disk reads, poor matches for questions that use different words from the code, and the need to rank noisy hits. Those costs are visible in a request trace, which makes them easier to justify or fix than an index built on assumption.</p>
<h2>Vector retrieval wins when developers ask about concepts the code never names</h2>
<p>Now change the question to, “Why can a cancelled job still produce an invoice?” The relevant implementation may be spread across a queue consumer, a state transition, and an event handler without using the words “cancelled” and “invoice” together. A preindexed vector search can surface semantically related passages that literal matching misses. It is more likely to win across several repositories, long design documents, or renamed subsystems, because users cannot reliably guess the identifiers that tie those sources together.</p>
<p>The competing approaches have distinct bills. <strong>Live search with ripgrep</strong> wins for fresh, identifier-rich questions and costs request-time I/O plus ranking logic. <strong>Preindexed retrieval with pgvector</strong> wins when conceptual discovery repeatedly matters and costs embedding generation, storage, reindexing, permission filtering, and investigation of stale results. A vector hit is only a candidate, because similarity does not establish that a passage describes the current behavior.</p>
<p>If you add an index, make its unit of retrieval useful to a programmer. Chunking at arbitrary character boundaries can separate a condition from the branch it guards, so prefer a function or similarly coherent unit identified with Tree-sitter. Store repository, path, revision, symbol, and access scope beside each chunk. <strong>PostgreSQL</strong> with <strong>pgvector</strong> can hold embeddings and metadata together; an HNSW index can accelerate approximate nearest-neighbor search, although its recall and query cost still need testing. The <strong>hnsw.ef_search</strong> setting is a search-effort knob to tune rather than a value to copy from another workload.</p>
<p>Do not force a choice between semantic and exact matches inside the indexed approach. A developer who pastes <em>SessionExpiredError</em> expects the exact symbol to rank highly, so combine vector candidates with <strong>BM25</strong> results from a search engine such as <strong>Elasticsearch</strong>, then rerank the merged set. An initial <strong>top-k of 8</strong> is an experiment setting: increase it if relevant files are missed, or decrease it if excerpts crowd out the question and answer. Keep the final context tied to a revision so the model can cite code the developer can still open.</p>
<p>I agree with <a href=/your-genai-feature-scales-until-retrieval-and-latency-collide/>Your GenAI Feature Scales Until Retrieval and Latency Collide</a> that retrieval latency becomes a production constraint, but index freshness deserves equal attention because a fast answer from yesterday’s code can be worse than a slower answer from the current commit. The vector approach earns its extra machinery only when it improves grounded answers under both constraints.</p>
<h2>Answer quality, not search relevance alone, should decide the winner</h2>
<p>A search result can look relevant and still lead the model to the wrong conclusion. Build an evaluation set from questions developers actually ask in code review, incident follow-up, and onboarding. For each question, record the revision, acceptable source locations, and what a correct answer must say. A <strong>30-question pilot</strong> is a planning size for the first comparison, not a statistically persuasive benchmark; its purpose is to expose failure modes before anyone optimizes a leaderboard number.</p>
<p>Run both retrieval approaches against the same repository revisions and the same answer model. Count whether a necessary file appears in the retrieved set using <strong>Recall@k</strong>, and use <strong>MRR</strong> to notice when it appears only after several distractions. Neither metric says whether the final explanation is sound, so also check whether the answer cites lines that support its claims, distinguishes tests from production behavior, and says “I cannot establish that” when the evidence is incomplete. Reviewers should mark a correct conclusion with an unsupported citation as a failure, because the assistant has given the developer no reliable way to verify it.</p>
<p>Include adversarially ordinary questions: a symbol renamed last week, documentation that contradicts implementation, and two functions with similar names in different services. Re-run the set after a merge to see whether indexed results still point at the intended revision. If vector retrieval improves conceptual questions but regresses exact-symbol questions, add hybrid ranking before declaring it the winner; that fix targets the observed failure rather than replacing a useful search method wholesale.</p>
<p>Measure the request path with <strong>OpenTelemetry</strong> spans for search, excerpt loading, model input preparation, and generation. Use <strong>p95 latency</strong> rather than only the mean, because slow repository reads or index queries affect the developer waiting on an answer even when most requests are quick. An <strong>800 ms retrieval budget</strong> is an example service objective to adjust after measuring your own environment, not a claim about either approach’s speed. Record input tokens and retrieval calls as well, since an apparently cheap search can become expensive when it sends large, repetitive excerpts to the model.</p>
<p>The decision rule should be hard to game: adopt the index for a question class only if it raises supported-answer quality without breaching that class’s latency and freshness requirements. Keep live search available for exact identifiers even after adding vectors, because a semantic ranking model has no reason to treat an exact error code as more important than a nearby conceptual match unless the system explicitly tells it to.</p>
<h2>Permissions and freshness can overturn a retrieval win</h2>
<p>A repository assistant is also an authorization system. Live search should run only over paths the requester may read; an indexed service must apply the same restriction <em>before</em> returning candidates. Filtering forbidden excerpts after a model has seen them is too late, because the answer can disclose their content. Store access metadata with indexed chunks, invalidate it when permissions change, and test that an unauthorized user cannot retrieve a restricted file through either exact or semantic queries.</p>
<p>Freshness needs an explicit contract. Suppose an index is allowed to lag the main branch by <strong>15 minutes</strong>; treat that as a proposed policy ceiling, not a property of the indexing software. Attach the indexed commit SHA to every result and compare it with the requested revision. If they differ beyond the permitted lag, fall back to live search or state plainly that the answer uses an older revision. Silent fallback to stale context makes a code explanation difficult to debug, because the developer and assistant may be reasoning about different programs.</p>
<p>Retrieved repository text is untrusted input, even when it comes from a file the developer can read. A comment can tell an assistant to ignore instructions or reveal credentials; it remains source material, not an instruction from the user. Keep tool permissions separate from the language model’s retrieved context, and avoid passing secrets or whole environment files merely because they matched a search. These controls apply to both approaches, while a persistent index adds the obligation to remove deleted or newly restricted content from stored copies.</p>
<p>Start by collecting real developer questions and implementing bounded, permission-aware live search at a pinned commit. Ask reviewers to mark the files and lines each answer actually needed. Once that baseline exists, test an indexed retriever on the questions it misses. If vectors cannot produce more supported answers at an acceptable freshness and latency cost, keep the simpler system and improve its ranking instead.</p>
<p>The post <a href="https://deepfriedbytes.com/rag-vs-fine-tuning-in-your-first-genai-backend/">RAG vs Fine-Tuning in Your First GenAI Backend</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated</title>
		<link>https://deepfriedbytes.com/ctos-your-2024-genai-build-vs-buy-playbook-is-outdated/</link>
		
		
		<pubDate>Wed, 30 Sep 2026 11:05:16 +0000</pubDate>
				<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[AI Web Solutions]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/ctos-your-2024-genai-build-vs-buy-playbook-is-outdated/</guid>

					<description><![CDATA[<p>“Build or buy the AI” is the wrong question for a CTO in 2026. The useful boundary is narrower: buy model capability until your own evidence shows that owning inference or training improves the economics. Build the evaluation, data controls, and fallback paths yourself. Those assets remain valuable when an API price changes, an open-weight model improves, or a supplier’s terms stop fitting your product. The default has shifted from owning a model to owning the decision boundary Two years ago, advice to prototype with an API and then train a custom model often sounded like a natural progression. I would no longer treat training as the next stage. Better multimodal APIs have made a purchased baseline more useful, while serving software such as vLLM has made self-hosting a credible alternative without making training necessary. These changes separate three decisions that older build-versus-buy advice bundled together: whose weights to use, where inference runs, and who controls the product’s acceptance criteria. My default is to buy access to a capable model, own the evaluation set, and keep the inference boundary replaceable. I would not train a proprietary vision model or fine-tune a language model merely because a pilot works, because pilot traffic rarely reveals enough about failure distribution or operating cost to justify an irreversible investment. Training becomes defensible when a labeled dataset represents production conditions, a purchased baseline misses a quantified requirement, and the expected gain exceeds annotation, retraining, and serving costs. This is not an argument for letting a supplier define success. A CTO should own the ground-truth examples, review rules, data-retention requirements, and release gate. For generated answers, that gate may require an answer to cite an authorized source and refuse when evidence is absent. For images, it may require a specified precision at a tolerable false-rejection rate. Those are product decisions; a vendor’s aggregate benchmark cannot substitute for them because its test distribution and error costs differ from yours. Custom Vision Model or Pretrained API Which Fits Your First App frames the first application as a model choice; I would make the first choice a reversible contract and test harness instead, because a promising first model says little about how cheaply the second one can be substituted. Put API calls behind an internal interface that records model identifier, prompt or preprocessing version, request cost, latency, and adjudicated outcome. That interface is worth building even if the supplier is never replaced. Recent platform changes have made old procurement shortcuts unreliable OpenAI’s Structured Outputs, introduced in 2024 with JSON Schema enforcement through strict: true on supported models, weakened the old claim that teams must fine-tune simply to get parseable fields. Schema conformity does not establish factual correctness, however, because a well-formed JSON object can still contain an invented value. Similarly, multimodal APIs can establish an image baseline quickly, but their availability does not prove that their terms permit every image to be sent off-site. The current procurement question is therefore less “Can an API do this?” and more “Which obligations survive a provider change?” Self-hosting has also changed. vLLM’s OpenAI-compatible server and Hugging Face Transformers reduce the work needed to try open-weight models, while ONNX Runtime can simplify deployment of some vision pipelines. None removes the cost of operating them: somebody still owns GPU capacity, upgrades, security patches, load testing, and incident response. An open-weight license must also be checked model by model, because “downloadable” does not establish permission for a particular commercial use. Older advice to compare only token or per-image prices is now especially weak. A quote omits retries, image resizing, prompt caching rules, minimum capacity, and human review. It can also hide a costly exit. Ask vendors for retention and deletion terms, region controls, model-version notification, rate limits, and an export path for logs and annotations. If personal data is involved, check the processor agreement against GDPR Article 28 rather than treating an API’s security page as a contract, because operational assurances and legal obligations are different artifacts. Do not infer that a standards badge settles model risk, either. ISO/IEC 42001:2023 describes an AI management system, and the NIST AI RMF 1.0 offers a risk-management framework; neither certifies that a specific response is correct for your application. Use those frameworks to organize ownership and evidence. Use your own test set to decide whether to ship. Pretrained APIs win the first test; self-hosting wins only under a proven constraint The explicit comparison is between a managed pretrained API and self-hosted open-weight inference. The managed API wins when demand is uncertain, model quality changes quickly, and a small team needs to test a product assumption: its cost is variable request spend, vendor dependence, and whatever data-handling restrictions the contract imposes. Self-hosting wins when sustained utilization, residency requirements, or a necessary model modification outweigh operations overhead: its cost is GPU capacity, engineering time, on-call responsibility, and often idle headroom. Neither option wins merely because its demo is faster. Set thresholds before running the comparison. For example, a p95 end-to-end latency target of 800 ms is a value to tune to the user interaction, not a universal benchmark. A 30-day shadow run is a proposed observation window, long enough to include at least some traffic variation but not a substitute for seasonal testing. A 95% precision floor might be an application’s chosen acceptance criterion when false positives trigger expensive review; another product may rationally favor recall. Make the assumptions explicit so a team cannot move the goalposts after seeing a preferred model’s results. Measure cost per accepted result, not cost per request. A cheap call that fails and is retried twice may cost more than one expensive successful call; a response requiring manual correction has another cost again. Record p50 and p95 latency, error rate, retry rate, abstention rate, and reviewer time by model version. OpenTelemetry can connect application traces to inference spans, but pin the semantic-convention version used by your instrumentation because GenAI attribute names have evolved. Export the underlying events as well as a dashboard so a provider switch does not erase your comparison. The following standard-library Python example runs without an SDK. Its 100 generated rows and dollar amounts are illustrative inputs, not measured performance; replace the rows with request logs to calculate a meaningful decision metric. python3 - &#60;&#60;'PY' import math rows = [(110 + (i * 37) % 390, 0.002 + (i % 4) * 0.001, i % 9 != 0) for i in range(100)] latencies = sorted(ms for ms, cost, accepted in rows) accepted = sum(ok for ms, cost, ok in rows) spend = sum(cost for ms, cost, ok in rows) p95 = latencies[math.ceil(0.95 * len(latencies)) - 1] print(f"p95_ms={p95}") print(f"accepted={accepted}/{len(rows)}") print(f"cost_per_accepted_usd={spend / accepted:.4f}") PY Run the same scoring code on both options’ logs, with reviewer decisions attached to identical requests. If a self-hosted model misses the acceptance floor, lower per-request spend is not a saving, because the missing quality must be paid for through retries, review, or lost users. Separate the evidence that changes a model decision from the evidence that changes a system decision Not every bad AI result calls for a different model. In a retrieval-backed feature, a correct model cannot cite a document that retrieval failed to supply. In a vision pipeline, a stronger classifier cannot recover detail discarded by an aggressive image resize. Before approving training or a hosting migration, require an error review that assigns each failure to input quality, retrieval or preprocessing, model behavior, policy, or application integration. The categories make spending accountable because each has a different plausible remedy. Your GenAI Feature Scales Until Retrieval and Latency Collide correctly warns that inference is not the whole response path; I would resist treating that collision as a reason to own model serving, because a slow search index remains slow after an API is replaced with a GPU. Instrument request ingress, retrieval, reranking, inference, and post-processing separately. For image requests, record decode and preprocessing time as well as inference. Compare p95 traces at the same concurrency, since a single-request benchmark conceals queueing. Keep model and system scorecards separate. A model scorecard can show precision, recall, calibration, and abstention on a versioned test set. A system scorecard should show end-to-end latency, cost per accepted result, review load, availability, and the proportion of requests excluded by data policy. If both improve after a change, trace the mechanism before attributing the gain to the model; otherwise, the next procurement decision may reward the wrong supplier. This separation also makes an exit plan credible. Store evaluation examples in a portable format, version transformations, and preserve enough request metadata to replay cases without retaining prohibited content. Test a second provider or a locally served model against the same gate before negotiating a long commitment. A fallback that has never passed that gate is a negotiating story, not an operational option. Run a shadow test before signing an annual contract Take one representative week of requests, remove data that cannot be used for testing, and have reviewers label a failure sample before comparing suppliers. Then run a managed API and a self-hosted candidate in shadow mode against the same acceptance rules. Ask finance for fully loaded costs and legal for the actual data terms. Sign for capacity only after the traces show which constraint ownership would remove.</p>
<p>The post <a href="https://deepfriedbytes.com/ctos-your-2024-genai-build-vs-buy-playbook-is-outdated/">CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>“Build or buy the AI” is the wrong question for a CTO in 2026. The useful boundary is narrower: buy model capability until your own evidence shows that owning inference or training improves the economics. Build the evaluation, data controls, and fallback paths yourself. Those assets remain valuable when an API price changes, an open-weight model improves, or a supplier’s terms stop fitting your product.</p>
<h2>The default has shifted from owning a model to owning the decision boundary</h2>
<p>Two years ago, advice to prototype with an API and then train a custom model often sounded like a natural progression. I would no longer treat training as the next stage. Better multimodal APIs have made a purchased baseline more useful, while serving software such as vLLM has made self-hosting a credible alternative without making training necessary. These changes separate three decisions that older build-versus-buy advice bundled together: whose weights to use, where inference runs, and who controls the product’s acceptance criteria.</p>
<p>My default is to buy access to a capable model, own the evaluation set, and keep the inference boundary replaceable. I would <strong>not</strong> train a proprietary vision model or fine-tune a language model merely because a pilot works, because pilot traffic rarely reveals enough about failure distribution or operating cost to justify an irreversible investment. Training becomes defensible when a labeled dataset represents production conditions, a purchased baseline misses a quantified requirement, and the expected gain exceeds annotation, retraining, and serving costs.</p>
<p>This is not an argument for letting a supplier define success. A CTO should own the ground-truth examples, review rules, data-retention requirements, and release gate. For generated answers, that gate may require an answer to cite an authorized source and refuse when evidence is absent. For images, it may require a specified precision at a tolerable false-rejection rate. Those are product decisions; a vendor’s aggregate benchmark cannot substitute for them because its test distribution and error costs differ from yours.</p>
<p><a href=/custom-vision-model-or-pretrained-api-which-fits-your-first-app/>Custom Vision Model or Pretrained API Which Fits Your First App</a> frames the first application as a model choice; I would make the first choice a reversible contract and test harness instead, because a promising first model says little about how cheaply the second one can be substituted. Put API calls behind an internal interface that records model identifier, prompt or preprocessing version, request cost, latency, and adjudicated outcome. That interface is worth building even if the supplier is never replaced.</p>
<h2>Recent platform changes have made old procurement shortcuts unreliable</h2>
<p>OpenAI’s Structured Outputs, introduced in 2024 with JSON Schema enforcement through <strong>strict: true</strong> on supported models, weakened the old claim that teams must fine-tune simply to get parseable fields. Schema conformity does not establish factual correctness, however, because a well-formed JSON object can still contain an invented value. Similarly, multimodal APIs can establish an image baseline quickly, but their availability does not prove that their terms permit every image to be sent off-site. The current procurement question is therefore less “Can an API do this?” and more “Which obligations survive a provider change?”</p>
<p>Self-hosting has also changed. vLLM’s OpenAI-compatible server and Hugging Face Transformers reduce the work needed to try open-weight models, while ONNX Runtime can simplify deployment of some vision pipelines. None removes the cost of operating them: somebody still owns GPU capacity, upgrades, security patches, load testing, and incident response. An open-weight license must also be checked model by model, because “downloadable” does not establish permission for a particular commercial use.</p>
<p>Older advice to compare only token or per-image prices is now especially weak. A quote omits retries, image resizing, prompt caching rules, minimum capacity, and human review. It can also hide a costly exit. Ask vendors for retention and deletion terms, region controls, model-version notification, rate limits, and an export path for logs and annotations. If personal data is involved, check the processor agreement against GDPR Article 28 rather than treating an API’s security page as a contract, because operational assurances and legal obligations are different artifacts.</p>
<p>Do not infer that a standards badge settles model risk, either. ISO/IEC 42001:2023 describes an AI management system, and the NIST AI RMF 1.0 offers a risk-management framework; neither certifies that a specific response is correct for your application. Use those frameworks to organize ownership and evidence. Use your own test set to decide whether to ship.</p>
<h2>Pretrained APIs win the first test; self-hosting wins only under a proven constraint</h2>
<p>The explicit comparison is between a <strong>managed pretrained API</strong> and <strong>self-hosted open-weight inference</strong>. The managed API wins when demand is uncertain, model quality changes quickly, and a small team needs to test a product assumption: its cost is variable request spend, vendor dependence, and whatever data-handling restrictions the contract imposes. Self-hosting wins when sustained utilization, residency requirements, or a necessary model modification outweigh operations overhead: its cost is GPU capacity, engineering time, on-call responsibility, and often idle headroom. Neither option wins merely because its demo is faster.</p>
<p>Set thresholds before running the comparison. For example, a <strong>p95 end-to-end latency target of 800 ms</strong> is a value to tune to the user interaction, not a universal benchmark. A <strong>30-day</strong> shadow run is a proposed observation window, long enough to include at least some traffic variation but not a substitute for seasonal testing. A <strong>95% precision floor</strong> might be an application’s chosen acceptance criterion when false positives trigger expensive review; another product may rationally favor recall. Make the assumptions explicit so a team cannot move the goalposts after seeing a preferred model’s results.</p>
<p>Measure cost per <em>accepted result</em>, not cost per request. A cheap call that fails and is retried twice may cost more than one expensive successful call; a response requiring manual correction has another cost again. Record p50 and p95 latency, error rate, retry rate, abstention rate, and reviewer time by model version. OpenTelemetry can connect application traces to inference spans, but pin the semantic-convention version used by your instrumentation because GenAI attribute names have evolved. Export the underlying events as well as a dashboard so a provider switch does not erase your comparison.</p>
<p>The following standard-library Python example runs without an SDK. Its 100 generated rows and dollar amounts are <em>illustrative inputs</em>, not measured performance; replace the rows with request logs to calculate a meaningful decision metric.</p>
<pre>python3 - &lt;&lt;'PY'
import math
rows = [(110 + (i * 37) % 390, 0.002 + (i % 4) * 0.001,
         i % 9 != 0) for i in range(100)]
latencies = sorted(ms for ms, cost, accepted in rows)
accepted = sum(ok for ms, cost, ok in rows)
spend = sum(cost for ms, cost, ok in rows)
p95 = latencies[math.ceil(0.95 * len(latencies)) - 1]
print(f"p95_ms={p95}")
print(f"accepted={accepted}/{len(rows)}")
print(f"cost_per_accepted_usd={spend / accepted:.4f}")
PY</pre>
<p>Run the same scoring code on both options’ logs, with reviewer decisions attached to identical requests. If a self-hosted model misses the acceptance floor, lower per-request spend is not a saving, because the missing quality must be paid for through retries, review, or lost users.</p>
<h2>Separate the evidence that changes a model decision from the evidence that changes a system decision</h2>
<p>Not every bad AI result calls for a different model. In a retrieval-backed feature, a correct model cannot cite a document that retrieval failed to supply. In a vision pipeline, a stronger classifier cannot recover detail discarded by an aggressive image resize. Before approving training or a hosting migration, require an error review that assigns each failure to input quality, retrieval or preprocessing, model behavior, policy, or application integration. The categories make spending accountable because each has a different plausible remedy.</p>
<p><a href=/your-genai-feature-scales-until-retrieval-and-latency-collide/>Your GenAI Feature Scales Until Retrieval and Latency Collide</a> correctly warns that inference is not the whole response path; I would resist treating that collision as a reason to own model serving, because a slow search index remains slow after an API is replaced with a GPU. Instrument request ingress, retrieval, reranking, inference, and post-processing separately. For image requests, record decode and preprocessing time as well as inference. Compare p95 traces at the same concurrency, since a single-request benchmark conceals queueing.</p>
<p>Keep model and system scorecards separate. A model scorecard can show precision, recall, calibration, and abstention on a versioned test set. A system scorecard should show end-to-end latency, cost per accepted result, review load, availability, and the proportion of requests excluded by data policy. If both improve after a change, trace the mechanism before attributing the gain to the model; otherwise, the next procurement decision may reward the wrong supplier.</p>
<p>This separation also makes an exit plan credible. Store evaluation examples in a portable format, version transformations, and preserve enough request metadata to replay cases without retaining prohibited content. Test a second provider or a locally served model against the same gate before negotiating a long commitment. A fallback that has never passed that gate is a negotiating story, not an operational option.</p>
<h2>Run a shadow test before signing an annual contract</h2>
<p>Take one representative week of requests, remove data that cannot be used for testing, and have reviewers label a failure sample before comparing suppliers. Then run a managed API and a self-hosted candidate in shadow mode against the same acceptance rules. Ask finance for fully loaded costs and legal for the actual data terms. Sign for capacity only after the traces show which constraint ownership would remove.</p>
<p>The post <a href="https://deepfriedbytes.com/ctos-your-2024-genai-build-vs-buy-playbook-is-outdated/">CTOs, Your 2024 GenAI Build-vs-Buy Playbook Is Outdated</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>Your 2024 Crypto Build-vs-Buy Playbook Is Obsolete</title>
		<link>https://deepfriedbytes.com/your-2024-crypto-build-vs-buy-playbook-is-obsolete/</link>
		
		
		<pubDate>Tue, 29 Sep 2026 07:36:09 +0000</pubDate>
				<category><![CDATA[Cryptocurrencies]]></category>
		<category><![CDATA[AI]]></category>
		<category><![CDATA[Blockchain]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/your-2024-crypto-build-vs-buy-playbook-is-obsolete/</guid>

					<description><![CDATA[<p>A wallet decision that looked sensible in 2024 can be an expensive mistake in 2026. Ethereum accounts can now delegate execution logic, while custody vendors offer increasingly complete signing and policy services. I would buy the signing and key-custody layer for most teams, then build the transaction policy and recovery process in-house. The exception is a company whose wallet infrastructure is itself the product. Ethereum’s account changes weakened the case for building a signer first Before Ethereum’s Pectra upgrade, a common argument for an in-house wallet stack was that externally owned accounts could not support enough application-specific behavior. That argument is weaker after EIP-7702, activated on Ethereum mainnet on May 7, 2025, according to the Ethereum Foundation’s published schedule. An EOA can authorize a delegation to contract code, allowing developers to design more flexible execution without first moving every user to a new smart-account address. Delegation does not remove the authority of the EOA’s underlying key, however, so it is not a substitute for protecting that key. This changes a CTO’s procurement question. “Can we build account abstraction?” is too broad to guide a decision; “Who controls the signing key, the delegated implementation, and the policy that approves a transfer?” exposes the actual boundaries. ERC-4337 remains relevant for teams using UserOperations, bundlers, and paymasters, while EIP-7702 offers another route to account behavior. Neither protocol decides who can authorize a recovery, who pays when sponsorship fails, or how an incident commander stops withdrawals. Older advice to choose one wallet architecture up front and hide it behind a generic “send transaction” API is now risky because that API obscures whether a request is an EOA transaction, an ERC-4337 UserOperation, or a delegated-account call. EIP-5792’s wallet_sendCalls can help represent batched calls where a wallet supports it, but wallet support and execution semantics must be checked rather than assumed. Put the operation type, chain identifier, expected signer, authorization method, and sponsorship decision into your internal request model. CAIP-2 chain identifiers can keep that model unambiguous when the same address appears on several networks. Cryptocurrency APIs for Developers: Build Secure Wallets is a useful starting point for wallet design, but its title’s build-first implication should not settle the custody decision. A capable team can write a viem 2.x client in days; demonstrating that its signing service will withstand a compromised application credential, a mistaken policy change, and a regional outage is a different project. I would not build a production key-signing service merely to avoid a vendor bill, because that bill is usually easier to bound than the consequences of an untested recovery path. Buy custody when it is a dependency; build it when control is the product The explicit choice is between a managed custody and signing API and an in-house signer using infrastructure such as AWS KMS or an HSM. The managed option wins when a wallet supports another product: it buys an established operational surface for key creation, signing controls, and access management. Its costs are vendor fees, integration work, exit planning, and the risk that a provider’s outage or policy model constrains your releases. The in-house option wins when you need to own signing semantics across providers or can justify dedicated cryptography and operations staff. Its costs include key ceremonies, access reviews, secure deployment, monitoring, and incident exercises, not just code. A cost comparison should price the same trust boundary on both sides. AWS KMS supports the ECC_SECG_P256K1 key specification used for Ethereum-compatible signatures, according to AWS documentation, but that feature alone is not a complete wallet policy engine. Your team still has to control who may call Sign, bind a signature request to an approved transaction, and prevent an application server from turning a broad signing permission into an unlimited withdrawal permission. Conversely, a managed provider is not automatically safe because it advertises multiparty computation: ask which actors can change policy, initiate recovery, or approve a destination, and test those answers in a sandbox. Use a comparable workload before comparing quotes. For example, model 50,000 active wallets and 10,000 signing requests per day as planning assumptions to replace with your forecast, not as claims about a typical deployment. Request a quote for that workload, including recovery events, webhook delivery, additional environments, and support. Alongside it, estimate engineering time for an in-house service, security review, on-call coverage, and an annual recovery exercise. The cheaper API price is not decisive if exporting accounts or recreating policies during an exit requires work your team has not funded. There is also a middle path worth stating precisely: buy key custody but own the approval policy outside the provider. A policy service can require a destination allowlist, a transaction simulation result, and a second approver before it requests a signature. Keep those decisions in an auditable store you control, then restrict the vendor credential so it cannot bypass them through an alternate endpoint. This arrangement wins when portability matters but key operations are not your differentiator; it costs an additional service and a careful review of every vendor permission. Do not describe it as “vendor-neutral” until you have run an exit test with another signer. A successful API call no longer proves that the intended account will execute it Integration advice used to concentrate on authenticating an API request, receiving a transaction hash, and watching for confirmation. That sequence misses an important question for delegated accounts: what code will the account use when the transaction executes? EIP-7702 delegation can change through an on-chain authorization, so an address classified once at onboarding cannot be treated as permanently unchanged. A transaction preview should record the chain, account state, target code, calldata, and block context; the approval service should recheck the facts it relies on immediately before submission. The following viem 2.x script checks whether an Ethereum mainnet address currently exposes an EIP-7702 delegation indicator. Save it as check.mjs, run npm install viem@2, and set ETH_RPC_URL and WALLET_ADDRESS before running node check.mjs: import { createPublicClient, http, isAddress } from 'viem'; import { mainnet } from 'viem/chains'; const address = process.env.WALLET_ADDRESS; if (!address &#124;&#124; !isAddress(address)) throw new Error('Set WALLET_ADDRESS'); const client = createPublicClient({ chain: mainnet, transport: http(process.env.ETH_RPC_URL) }); const code = await client.getCode({ address }); const delegated = /^0xef0100[0-9a-f]{40}$/i.test(code ?? ''); console.log({ address, delegated, implementation: delegated ? `0x${code.slice(8)}` : null }); This is an inspection aid, not an approval check: it reads current code but does not verify the implementation’s behavior, prove who authorized it, or guarantee the state at a later block. Where approval depends on a particular implementation, pin its address and reviewed code hash, simulate the proposed call against recent state, and fail closed if the delegation changes. Use a separate review for upgrades to that implementation. A passkey login under WebAuthn or a successful SIWE message under EIP-4361 proves something about authentication; neither, by itself, proves that the resulting transfer matches the user’s intent. I would use Cryptocurrency APIs for Developers Secure Wallet Integration for integration vocabulary, not as evidence that an SDK settles this trust boundary. ERC-1271 contract signatures, WalletConnect v2 sessions, provider permissions under EIP-1193, and delegated-account execution all give an application different ways to receive an apparently valid approval. The policy service must specify which of those approvals is acceptable for each operation, because treating every “signature verified” result as equivalent makes recovery and high-value transfers harder to defend. The procurement test should be an exit and incident exercise Over the last two years, the operational question has also become harder to postpone. In the EU, MiCA’s crypto-asset service provider provisions began applying on December 30, 2024; whether they apply to a particular wallet business depends on its services and jurisdiction. DORA has applied since January 17, 2025, to covered financial entities and affects their management of ICT third-party risk. Neither rule means every wallet team needs the same licence or contract. Both make “we will document our provider dependencies later” poor advice for a covered operation, because the dependency determines who can act during a failure. Before signing a custody contract, run a timed exercise rather than accepting a diagram. Give the vendor and your team a scenario in which a policy administrator is compromised while one signing region is unavailable. Set a test target of 30 minutes to disable new withdrawals; that is an internal target to tune against your risk, not a vendor performance claim. Record who has the authority to pause signing, how that authority is authenticated, and whether the action also blocks an attacker holding an existing API credential. Then restore service without quietly discarding the audit trail. Run a second exercise in which the provider relationship ends. Export what is actually exportable, identify accounts that cannot migrate without user action, and reproduce transaction policy with a replacement signer. Safe contracts, OpenZeppelin Contracts 5.x components, and standard EVM transaction formats may improve portability, but none guarantees that a vendor’s recovery roles or off-chain approval history will transfer. Ask for the data format and procedure in the contract, because an undocumented promise of “easy migration” cannot be tested during procurement. Finally, make the decision reversible where you can. Keep your application’s ledger, transaction intents, and policy decisions outside the signing provider, with identifiers that remain meaningful after a migration. Require receipts to link an approved intent to the exact signed payload and resulting on-chain transaction. This separation costs additional engineering now, but it lets the company change custody arrangements without rewriting the business rules that determine whether money may move. Start with the failure you would otherwise delegate Ask each shortlisted provider and your in-house team to demonstrate the same compromised-admin pause and account-exit exercise next week. Write down the authority, elapsed time, missing data, and manual steps for each attempt. Choose the option that leaves your team able to stop and explain a bad transfer—not the one whose SDK produces the first successful transaction hash.</p>
<p>The post <a href="https://deepfriedbytes.com/your-2024-crypto-build-vs-buy-playbook-is-obsolete/">Your 2024 Crypto Build-vs-Buy Playbook Is Obsolete</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>A wallet decision that looked sensible in 2024 can be an expensive mistake in 2026. Ethereum accounts can now delegate execution logic, while custody vendors offer increasingly complete signing and policy services. I would buy the signing and key-custody layer for most teams, then build the transaction policy and recovery process in-house. The exception is a company whose wallet infrastructure is itself the product.</p>
<h2>Ethereum’s account changes weakened the case for building a signer first</h2>
<p>Before Ethereum’s Pectra upgrade, a common argument for an in-house wallet stack was that externally owned accounts could not support enough application-specific behavior. That argument is weaker after EIP-7702, activated on Ethereum mainnet on May 7, 2025, according to the Ethereum Foundation’s published schedule. An EOA can authorize a delegation to contract code, allowing developers to design more flexible execution without first moving every user to a new smart-account address. Delegation does not remove the authority of the EOA’s underlying key, however, so it is not a substitute for protecting that key.</p>
<p>This changes a CTO’s procurement question. “Can we build account abstraction?” is too broad to guide a decision; “Who controls the signing key, the delegated implementation, and the policy that approves a transfer?” exposes the actual boundaries. ERC-4337 remains relevant for teams using UserOperations, bundlers, and paymasters, while EIP-7702 offers another route to account behavior. Neither protocol decides who can authorize a recovery, who pays when sponsorship fails, or how an incident commander stops withdrawals.</p>
<p>Older advice to choose one wallet architecture up front and hide it behind a generic “send transaction” API is now risky because that API obscures whether a request is an EOA transaction, an ERC-4337 UserOperation, or a delegated-account call. EIP-5792’s <em>wallet_sendCalls</em> can help represent batched calls where a wallet supports it, but wallet support and execution semantics must be checked rather than assumed. Put the operation type, chain identifier, expected signer, authorization method, and sponsorship decision into your internal request model. CAIP-2 chain identifiers can keep that model unambiguous when the same address appears on several networks.</p>
<p><a href=/cryptocurrency-apis-for-developers-build-secure-wallets/>Cryptocurrency APIs for Developers: Build Secure Wallets</a> is a useful starting point for wallet design, but its title’s build-first implication should not settle the custody decision. A capable team can write a viem 2.x client in days; demonstrating that its signing service will withstand a compromised application credential, a mistaken policy change, and a regional outage is a different project. I would not build a production key-signing service merely to avoid a vendor bill, because that bill is usually easier to bound than the consequences of an untested recovery path.</p>
<h2>Buy custody when it is a dependency; build it when control is the product</h2>
<p>The explicit choice is between a <strong>managed custody and signing API</strong> and an <strong>in-house signer using infrastructure such as AWS KMS or an HSM</strong>. The managed option wins when a wallet supports another product: it buys an established operational surface for key creation, signing controls, and access management. Its costs are vendor fees, integration work, exit planning, and the risk that a provider’s outage or policy model constrains your releases. The in-house option wins when you need to own signing semantics across providers or can justify dedicated cryptography and operations staff. Its costs include key ceremonies, access reviews, secure deployment, monitoring, and incident exercises, not just code.</p>
<p>A cost comparison should price the same trust boundary on both sides. AWS KMS supports the <em>ECC_SECG_P256K1</em> key specification used for Ethereum-compatible signatures, according to AWS documentation, but that feature alone is not a complete wallet policy engine. Your team still has to control who may call <em>Sign</em>, bind a signature request to an approved transaction, and prevent an application server from turning a broad signing permission into an unlimited withdrawal permission. Conversely, a managed provider is not automatically safe because it advertises multiparty computation: ask which actors can change policy, initiate recovery, or approve a destination, and test those answers in a sandbox.</p>
<p>Use a comparable workload before comparing quotes. For example, model 50,000 active wallets and 10,000 signing requests per day as <em>planning assumptions to replace with your forecast</em>, not as claims about a typical deployment. Request a quote for that workload, including recovery events, webhook delivery, additional environments, and support. Alongside it, estimate engineering time for an in-house service, security review, on-call coverage, and an annual recovery exercise. The cheaper API price is not decisive if exporting accounts or recreating policies during an exit requires work your team has not funded.</p>
<p>There is also a middle path worth stating precisely: buy key custody but own the approval policy outside the provider. A policy service can require a destination allowlist, a transaction simulation result, and a second approver before it requests a signature. Keep those decisions in an auditable store you control, then restrict the vendor credential so it cannot bypass them through an alternate endpoint. This arrangement wins when portability matters but key operations are not your differentiator; it costs an additional service and a careful review of every vendor permission. Do not describe it as “vendor-neutral” until you have run an exit test with another signer.</p>
<h2>A successful API call no longer proves that the intended account will execute it</h2>
<p>Integration advice used to concentrate on authenticating an API request, receiving a transaction hash, and watching for confirmation. That sequence misses an important question for delegated accounts: what code will the account use when the transaction executes? EIP-7702 delegation can change through an on-chain authorization, so an address classified once at onboarding cannot be treated as permanently unchanged. A transaction preview should record the chain, account state, target code, calldata, and block context; the approval service should recheck the facts it relies on immediately before submission.</p>
<p>The following viem 2.x script checks whether an Ethereum mainnet address currently exposes an EIP-7702 delegation indicator. Save it as <em>check.mjs</em>, run <em>npm install viem@2</em>, and set <em>ETH_RPC_URL</em> and <em>WALLET_ADDRESS</em> before running <em>node check.mjs</em>:</p>
<pre>import { createPublicClient, http, isAddress } from 'viem';
import { mainnet } from 'viem/chains';
const address = process.env.WALLET_ADDRESS;
if (!address || !isAddress(address)) throw new Error('Set WALLET_ADDRESS');
const client = createPublicClient({
  chain: mainnet, transport: http(process.env.ETH_RPC_URL)
});
const code = await client.getCode({ address });
const delegated = /^0xef0100[0-9a-f]{40}$/i.test(code ?? '');
console.log({ address, delegated,
  implementation: delegated ? `0x${code.slice(8)}` : null });</pre>
<p>This is an inspection aid, not an approval check: it reads current code but does not verify the implementation’s behavior, prove who authorized it, or guarantee the state at a later block. Where approval depends on a particular implementation, pin its address and reviewed code hash, simulate the proposed call against recent state, and fail closed if the delegation changes. Use a separate review for upgrades to that implementation. A passkey login under WebAuthn or a successful SIWE message under EIP-4361 proves something about authentication; neither, by itself, proves that the resulting transfer matches the user’s intent.</p>
<p>I would use <a href=/cryptocurrency-apis-for-developers-secure-wallet-integration/>Cryptocurrency APIs for Developers Secure Wallet Integration</a> for integration vocabulary, not as evidence that an SDK settles this trust boundary. ERC-1271 contract signatures, WalletConnect v2 sessions, provider permissions under EIP-1193, and delegated-account execution all give an application different ways to receive an apparently valid approval. The policy service must specify which of those approvals is acceptable for each operation, because treating every “signature verified” result as equivalent makes recovery and high-value transfers harder to defend.</p>
<h2>The procurement test should be an exit and incident exercise</h2>
<p>Over the last two years, the operational question has also become harder to postpone. In the EU, MiCA’s crypto-asset service provider provisions began applying on December 30, 2024; whether they apply to a particular wallet business depends on its services and jurisdiction. DORA has applied since January 17, 2025, to covered financial entities and affects their management of ICT third-party risk. Neither rule means every wallet team needs the same licence or contract. Both make “we will document our provider dependencies later” poor advice for a covered operation, because the dependency determines who can act during a failure.</p>
<p>Before signing a custody contract, run a timed exercise rather than accepting a diagram. Give the vendor and your team a scenario in which a policy administrator is compromised while one signing region is unavailable. Set a <em>test target</em> of 30 minutes to disable new withdrawals; that is an internal target to tune against your risk, not a vendor performance claim. Record who has the authority to pause signing, how that authority is authenticated, and whether the action also blocks an attacker holding an existing API credential. Then restore service without quietly discarding the audit trail.</p>
<p>Run a second exercise in which the provider relationship ends. Export what is actually exportable, identify accounts that cannot migrate without user action, and reproduce transaction policy with a replacement signer. Safe contracts, OpenZeppelin Contracts 5.x components, and standard EVM transaction formats may improve portability, but none guarantees that a vendor’s recovery roles or off-chain approval history will transfer. Ask for the data format and procedure in the contract, because an undocumented promise of “easy migration” cannot be tested during procurement.</p>
<p>Finally, make the decision reversible where you can. Keep your application’s ledger, transaction intents, and policy decisions outside the signing provider, with identifiers that remain meaningful after a migration. Require receipts to link an approved intent to the exact signed payload and resulting on-chain transaction. This separation costs additional engineering now, but it lets the company change custody arrangements without rewriting the business rules that determine whether money may move.</p>
<h2>Start with the failure you would otherwise delegate</h2>
<p>Ask each shortlisted provider and your in-house team to demonstrate the same compromised-admin pause and account-exit exercise next week. Write down the authority, elapsed time, missing data, and manual steps for each attempt. Choose the option that leaves your team able to stop and explain a bad transfer—not the one whose SDK produces the first successful transaction hash.</p>
<p>The post <a href="https://deepfriedbytes.com/your-2024-crypto-build-vs-buy-playbook-is-obsolete/">Your 2024 Crypto Build-vs-Buy Playbook Is Obsolete</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>Your GenAI Feature Scales Until Retrieval and Latency Collide</title>
		<link>https://deepfriedbytes.com/your-genai-feature-scales-until-retrieval-and-latency-collide/</link>
		
		
		<pubDate>Tue, 22 Sep 2026 08:25:05 +0000</pubDate>
				<category><![CDATA[Generative AI]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/your-genai-feature-scales-until-retrieval-and-latency-collide/</guid>

					<description><![CDATA[<p>Generative AI for software development does not fail first because the model writes bad code; it fails first because the organization cannot review, test, attribute, and govern the extra code volume. My position is deliberately narrow: a product manager should scope the rollout as a throughput and control-plane project, because assistant licenses are easy to buy and hard to operationalize. The first bottleneck is human review, because AI increases change volume before it increases trust The uncomfortable scaling limit is not prompt quality; it is reviewer capacity, because generated code still arrives as a pull request that someone accountable must understand. A team can move from “AI helped me write a function” to “AI produced a 900-line patch across six services” in one sprint, and the second case burns senior-engineer time faster than it saves junior-engineer time. Generative AI for Software Development: Practical Use Cases is useful as a catalogue of developer tasks, but I would challenge its implicit optimism because practical use cases become expensive when every generated diff requires architectural context, security review, and regression confidence. A product manager should treat “code accepted by maintainers” as the unit of value, because “code generated” measures activity rather than delivery. For planning, use a concrete capacity model. A reasonable tuning value is to cap AI-assisted pull requests at 500 added lines or 15 touched files until the team has evidence that reviews stay fast. A conservative planning baseline for a 30-engineer group is that only 6 to 8 people can reliably review cross-service changes, because domain ownership, on-call rotation, and release pressure shrink the real reviewer pool. The metrics should be mundane: pull request cycle time, review wait time, change failure rate, escaped defects, flaky-test rate, and DORA deployment frequency. DORA metrics are useful here because they connect engineering flow to release risk rather than celebrating assistant usage. I would also track p95 review wait time, because the average hides the exact class of large generated patches that usually cause escalation. I would not run a company-wide “enable Copilot for everyone and target 30% higher output” program, because usage targets reward larger diffs and make it harder to see whether the delivery system actually improved. The better first milestone is narrower: one product area, one repository family, one definition of an acceptable AI-assisted patch, and one set of review rules that can be enforced automatically. Policy entropy breaks next, because every assistant creates a second path for code to enter the system At small scale, a developer using GitHub Copilot Enterprise, Cursor, JetBrains AI Assistant 2024.3, Windsurf, OpenAI GPT-4.1, Anthropic Claude 3.5 Sonnet, Google Gemini 1.5 Pro, or Meta Llama 3.1 70B looks like a personal productivity choice. At scale, it becomes a policy surface, because each tool may touch source code, issue content, logs, secrets, and proprietary architecture notes. The product manager’s scoping problem is to decide which questions must be answered before expansion. Can prompts include customer data? Are generated snippets stored by the vendor? Which repositories are excluded? Which license-risk checks run before merge? Which models can be used for security-sensitive code? These questions are product work because unresolved policy becomes blocked adoption, not a legal side quest. Real tools make the controls specific. Semgrep 1.96 with &#8211;config=p/ci can catch common insecure patterns before review. CodeQL 2.20.0 can run language-aware queries and publish SARIF 2.1.0 results into GitHub code scanning. SonarQube 10.6 can enforce maintainability gates. OWASP ASVS 4.0.3 gives security reviewers a vocabulary for application controls. CycloneDX 1.6 or SPDX 2.3 can represent a software bill of materials, which matters because AI-generated dependency suggestions often look harmless until they add transitive risk. There is a scale trap in context windows. Gemini 1.5 Pro has a vendor-published context window of up to 1 million tokens, and Claude 3.5 Sonnet has been marketed with a 200,000-token context window; those numbers are impressive, but large context does not remove the need for repository boundaries because a model can still combine unrelated internal details in ways your policy did not intend. Bigger context helps retrieval and refactoring, but it also increases the blast radius of a careless prompt. The disagreement I expect from some engineering leads is that mature teams should trust developers and audit outcomes. I agree with trusting developers, but I disagree with delaying controls because retroactive audits are weak when the organization cannot reconstruct what context was sent to which model. OpenTelemetry 1.32.0-style trace IDs for AI tool calls, even if implemented through vendor logs rather than pure OpenTelemetry, are worth scoping because incident response needs a timeline. The wrong platform choice creates hidden operating costs, because “AI coding” is really a delivery system dependency A product manager eventually has to choose between a managed assistant and a more controlled internal stack. The decision should not be framed as innovation versus caution, because both options can be innovative and both can waste money. The useful comparison is operating model. GitHub Copilot Enterprise wins when the team already lives in GitHub, wants fast onboarding, and accepts vendor-managed model access. Its vendor-published list price has been $39 per user per month, so a 200-developer rollout is easy to forecast as a license line. The cost is less customization and less control over model behavior, because the product is optimized for broad developer workflows rather than your exact architecture. Continue.dev with vLLM serving Llama 3.1 70B wins when source-control boundaries, offline operation, or model-routing control matter more than day-one convenience. The software can reduce per-seat licensing pressure, but the cost moves into GPUs, platform engineering, model evaluation, uptime, and security ownership. This path is cheaper only when utilization is high and the organization already knows how to run production ML infrastructure. My bias for most product teams is to start with the managed assistant and build hard gates around it, because the bottleneck is usually governance and review rather than model access. I would choose the self-hosted route first only for repositories with strict data-residency requirements or unusually sensitive intellectual property, because the operational load of GPUs and model serving distracts from delivery unless control is the product requirement. Generative AI for Software Development: Key Use Cases helps name candidate workflows, yet the roadmap should price each workflow by review cost, test cost, and rollback cost. Unit-test generation is cheap to pilot because failures are visible. Cross-service refactoring is expensive because the failure may appear as a degraded customer journey two releases later. Latency also matters more than demos suggest. A practical internal service-level objective might set p95 assistant response time below 10 seconds, a value to tune by workflow, because developers abandon slow tools or paste code into unapproved alternatives. For build pipelines, a separate budget is needed: adding Semgrep, CodeQL, SBOM generation, and AI-specific policy checks can add 3 to 7 minutes per CI run in many repositories, so parallelization and caching should be scoped rather than discovered after rollout. The control plane should be in the backlog, because guidelines that cannot run will not scale Written guidance is necessary, but it does not scale by itself because reviewers and developers will interpret “small patch,” “safe dependency,” and “AI-assisted” differently under deadline pressure. The backlog needs executable controls: repository labels, CI checks, model access rules, audit logs, dependency policies, and review routing. A simple guard can start with patch size. The following script is intentionally blunt, because blunt limits are easier to enforce during the first rollout than nuanced exceptions no one owns: #!/usr/bin/env bash set -euo pipefail base=${1:-origin/main} files=$(git diff --name-only "$base"...HEAD &#124; wc -l &#124; tr -d ' ') added=$(git diff --numstat "$base"...HEAD &#124; awk '{a+=$1} END{print a+0}') test "$files" -le "${MAX_FILES:-15}" &#124;&#124; { echo "Too many files: $files"; exit 1; } test "$added" -le "${MAX_ADDED:-500}" &#124;&#124; { echo "Too many added lines: $added"; exit 1; } python -m pytest -q echo "AI patch budget passed: $files files, $added added lines" This is not a mature governance system, but it changes the rollout conversation because “AI helped me” no longer bypasses the same delivery economics as any other code. The limits should be tuned per repository, because a frontend component library and a billing service have different risk profiles. The script can run in GitHub Actions, GitLab CI, Buildkite, or Jenkins, and it can publish results as SARIF 2.1.0 if the team wants visibility in code-scanning dashboards. Scope the control plane as product work with owners. The developer experience owner should define approved assistants and IDE integrations. The platform owner should define CI gates, caching, and observability. The security owner should define secret-handling, license scanning, and ASVS mapping. The engineering manager should define review allocation. Without ownership, the first production incident becomes the policy design meeting, which is a slow and expensive way to learn. Evaluation should also be specific. BLEU and pass@k are useful in model papers, but they are weak product metrics for internal software delivery because they miss maintainability, context fit, and operational risk. Better rollout metrics include merge rate of AI-assisted PRs, reverted-change percentage, mean time to restore, test flake rate, review comments per 100 lines, and percentage of generated code with linked issue context. A strict but useful target is to keep AI-assisted changes at or below the team’s existing change failure rate for two release cycles, because productivity gains are fake if they create extra recovery work. Scope the rollout around queues, because the model is rarely the long pole The first concrete step is to map one repository’s path from prompt to production and put times on every queue: generation, local test, CI, security scan, review, staging, release, and rollback. Then choose one workflow with low rollback cost, set patch-size limits, and run it for two release cycles. Buy fewer licenses than enthusiasm suggests, because constrained pilots reveal the real work faster.</p>
<p>The post <a href="https://deepfriedbytes.com/your-genai-feature-scales-until-retrieval-and-latency-collide/">Your GenAI Feature Scales Until Retrieval and Latency Collide</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>Generative AI for software development does not fail first because the model writes bad code; it fails first because the organization cannot review, test, attribute, and govern the extra code volume. My position is deliberately narrow: a product manager should scope the rollout as a throughput and control-plane project, because assistant licenses are easy to buy and hard to operationalize.</p>
<h2>The first bottleneck is human review, because AI increases change volume before it increases trust</h2>
<p>The uncomfortable scaling limit is not prompt quality; it is reviewer capacity, because generated code still arrives as a pull request that someone accountable must understand. A team can move from “AI helped me write a function” to “AI produced a 900-line patch across six services” in one sprint, and the second case burns senior-engineer time faster than it saves junior-engineer time.</p>
<p><a href=/generative-ai-for-software-development-practical-use-cases/>Generative AI for Software Development: Practical Use Cases</a> is useful as a catalogue of developer tasks, but I would challenge its implicit optimism because practical use cases become expensive when every generated diff requires architectural context, security review, and regression confidence. A product manager should treat “code accepted by maintainers” as the unit of value, because “code generated” measures activity rather than delivery.</p>
<p>For planning, use a concrete capacity model. A reasonable tuning value is to cap AI-assisted pull requests at <strong>500 added lines</strong> or <strong>15 touched files</strong> until the team has evidence that reviews stay fast. A conservative planning baseline for a <strong>30-engineer</strong> group is that only <strong>6 to 8</strong> people can reliably review cross-service changes, because domain ownership, on-call rotation, and release pressure shrink the real reviewer pool.</p>
<p>The metrics should be mundane: pull request cycle time, review wait time, change failure rate, escaped defects, flaky-test rate, and DORA deployment frequency. DORA metrics are useful here because they connect engineering flow to release risk rather than celebrating assistant usage. I would also track p95 review wait time, because the average hides the exact class of large generated patches that usually cause escalation.</p>
<p>I would not run a company-wide “enable Copilot for everyone and target 30% higher output” program, because usage targets reward larger diffs and make it harder to see whether the delivery system actually improved. The better first milestone is narrower: one product area, one repository family, one definition of an acceptable AI-assisted patch, and one set of review rules that can be enforced automatically.</p>
<h2>Policy entropy breaks next, because every assistant creates a second path for code to enter the system</h2>
<p>At small scale, a developer using GitHub Copilot Enterprise, Cursor, JetBrains AI Assistant 2024.3, Windsurf, OpenAI GPT-4.1, Anthropic Claude 3.5 Sonnet, Google Gemini 1.5 Pro, or Meta Llama 3.1 70B looks like a personal productivity choice. At scale, it becomes a policy surface, because each tool may touch source code, issue content, logs, secrets, and proprietary architecture notes.</p>
<p>The product manager’s scoping problem is to decide which questions must be answered before expansion. Can prompts include customer data? Are generated snippets stored by the vendor? Which repositories are excluded? Which license-risk checks run before merge? Which models can be used for security-sensitive code? These questions are product work because unresolved policy becomes blocked adoption, not a legal side quest.</p>
<p>Real tools make the controls specific. Semgrep 1.96 with <strong>&#8211;config=p/ci</strong> can catch common insecure patterns before review. CodeQL 2.20.0 can run language-aware queries and publish SARIF 2.1.0 results into GitHub code scanning. SonarQube 10.6 can enforce maintainability gates. OWASP ASVS 4.0.3 gives security reviewers a vocabulary for application controls. CycloneDX 1.6 or SPDX 2.3 can represent a software bill of materials, which matters because AI-generated dependency suggestions often look harmless until they add transitive risk.</p>
<p>There is a scale trap in context windows. Gemini 1.5 Pro has a vendor-published context window of up to <strong>1 million tokens</strong>, and Claude 3.5 Sonnet has been marketed with a <strong>200,000-token</strong> context window; those numbers are impressive, but large context does not remove the need for repository boundaries because a model can still combine unrelated internal details in ways your policy did not intend. Bigger context helps retrieval and refactoring, but it also increases the blast radius of a careless prompt.</p>
<p>The disagreement I expect from some engineering leads is that mature teams should trust developers and audit outcomes. I agree with trusting developers, but I disagree with delaying controls because retroactive audits are weak when the organization cannot reconstruct what context was sent to which model. OpenTelemetry 1.32.0-style trace IDs for AI tool calls, even if implemented through vendor logs rather than pure OpenTelemetry, are worth scoping because incident response needs a timeline.</p>
<h2>The wrong platform choice creates hidden operating costs, because “AI coding” is really a delivery system dependency</h2>
<p>A product manager eventually has to choose between a managed assistant and a more controlled internal stack. The decision should not be framed as innovation versus caution, because both options can be innovative and both can waste money. The useful comparison is operating model.</p>
<ul>
<li><strong>GitHub Copilot Enterprise</strong> wins when the team already lives in GitHub, wants fast onboarding, and accepts vendor-managed model access. Its vendor-published list price has been <strong>$39 per user per month</strong>, so a 200-developer rollout is easy to forecast as a license line. The cost is less customization and less control over model behavior, because the product is optimized for broad developer workflows rather than your exact architecture.</li>
<li><strong>Continue.dev with vLLM serving Llama 3.1 70B</strong> wins when source-control boundaries, offline operation, or model-routing control matter more than day-one convenience. The software can reduce per-seat licensing pressure, but the cost moves into GPUs, platform engineering, model evaluation, uptime, and security ownership. This path is cheaper only when utilization is high and the organization already knows how to run production ML infrastructure.</li>
</ul>
<p>My bias for most product teams is to start with the managed assistant and build hard gates around it, because the bottleneck is usually governance and review rather than model access. I would choose the self-hosted route first only for repositories with strict data-residency requirements or unusually sensitive intellectual property, because the operational load of GPUs and model serving distracts from delivery unless control is the product requirement.</p>
<p><a href=/generative-ai-for-software-development-key-use-cases/>Generative AI for Software Development: Key Use Cases</a> helps name candidate workflows, yet the roadmap should price each workflow by review cost, test cost, and rollback cost. Unit-test generation is cheap to pilot because failures are visible. Cross-service refactoring is expensive because the failure may appear as a degraded customer journey two releases later.</p>
<p>Latency also matters more than demos suggest. A practical internal service-level objective might set p95 assistant response time below <strong>10 seconds</strong>, a value to tune by workflow, because developers abandon slow tools or paste code into unapproved alternatives. For build pipelines, a separate budget is needed: adding Semgrep, CodeQL, SBOM generation, and AI-specific policy checks can add <strong>3 to 7 minutes</strong> per CI run in many repositories, so parallelization and caching should be scoped rather than discovered after rollout.</p>
<h2>The control plane should be in the backlog, because guidelines that cannot run will not scale</h2>
<p>Written guidance is necessary, but it does not scale by itself because reviewers and developers will interpret “small patch,” “safe dependency,” and “AI-assisted” differently under deadline pressure. The backlog needs executable controls: repository labels, CI checks, model access rules, audit logs, dependency policies, and review routing.</p>
<p>A simple guard can start with patch size. The following script is intentionally blunt, because blunt limits are easier to enforce during the first rollout than nuanced exceptions no one owns:</p>
<pre>#!/usr/bin/env bash
set -euo pipefail
base=${1:-origin/main}
files=$(git diff --name-only "$base"...HEAD | wc -l | tr -d ' ')
added=$(git diff --numstat "$base"...HEAD | awk '{a+=$1} END{print a+0}')
test "$files" -le "${MAX_FILES:-15}" || { echo "Too many files: $files"; exit 1; }
test "$added" -le "${MAX_ADDED:-500}" || { echo "Too many added lines: $added"; exit 1; }
python -m pytest -q
echo "AI patch budget passed: $files files, $added added lines"</pre>
<p>This is not a mature governance system, but it changes the rollout conversation because “AI helped me” no longer bypasses the same delivery economics as any other code. The limits should be tuned per repository, because a frontend component library and a billing service have different risk profiles. The script can run in GitHub Actions, GitLab CI, Buildkite, or Jenkins, and it can publish results as SARIF 2.1.0 if the team wants visibility in code-scanning dashboards.</p>
<p>Scope the control plane as product work with owners. The developer experience owner should define approved assistants and IDE integrations. The platform owner should define CI gates, caching, and observability. The security owner should define secret-handling, license scanning, and ASVS mapping. The engineering manager should define review allocation. Without ownership, the first production incident becomes the policy design meeting, which is a slow and expensive way to learn.</p>
<p>Evaluation should also be specific. BLEU and pass@k are useful in model papers, but they are weak product metrics for internal software delivery because they miss maintainability, context fit, and operational risk. Better rollout metrics include merge rate of AI-assisted PRs, reverted-change percentage, mean time to restore, test flake rate, review comments per 100 lines, and percentage of generated code with linked issue context. A strict but useful target is to keep AI-assisted changes at or below the team’s existing change failure rate for <strong>two release cycles</strong>, because productivity gains are fake if they create extra recovery work.</p>
<h2>Scope the rollout around queues, because the model is rarely the long pole</h2>
<p>The first concrete step is to map one repository’s path from prompt to production and put times on every queue: generation, local test, CI, security scan, review, staging, release, and rollback. Then choose one workflow with low rollback cost, set patch-size limits, and run it for two release cycles. Buy fewer licenses than enthusiasm suggests, because constrained pilots reveal the real work faster.</p>
<p>The post <a href="https://deepfriedbytes.com/your-genai-feature-scales-until-retrieval-and-latency-collide/">Your GenAI Feature Scales Until Retrieval and Latency Collide</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
		<item>
		<title>At 10x users, your crypto wallet breaks at idempotency</title>
		<link>https://deepfriedbytes.com/at-10x-users-your-crypto-wallet-breaks-at-idempotency/</link>
		
		
		<pubDate>Mon, 21 Sep 2026 08:13:44 +0000</pubDate>
				<category><![CDATA[Cryptocurrencies]]></category>
		<category><![CDATA[AI Integration]]></category>
		<category><![CDATA[AI Web Solutions]]></category>
		<category><![CDATA[Blockchain]]></category>
		<category><![CDATA[Digital ecosystems]]></category>
		<guid isPermaLink="false">https://deepfriedbytes.com/at-10x-users-your-crypto-wallet-breaks-at-idempotency/</guid>

					<description><![CDATA[<p>Crypto wallet scaling usually fails before anyone finds a cryptographic flaw. My position: a product manager should scope the first scale milestone around signing operations, approvals, nonce control, and reconciliation, because those queues break before BIP-39 words, secp256k1 math, or storage diagrams do. Treat secure storage as necessary plumbing, not the center of the roadmap. The signing queue breaks before the key ceremony does The team can treat Cryptocurrency Wallets for Developers Secure Storage Guide as the storage baseline, but the first scaling risk is usually not whether a private key is encrypted; it is whether the product can decide, sign, broadcast, and account for thousands of user actions without creating stuck funds or duplicate withdrawals. That position is debatable because key loss is existential, but it holds in product scoping because serious teams already use a hardened storage primitive before launch: AWS KMS with KeySpec=ECC_SECG_P256K1 and KeyUsage=SIGN_VERIFY, HashiCorp Vault 1.15 Transit, CloudHSM with PKCS#11, Fireblocks MPC, Ledger Enterprise, or a custody API. The storage decision matters, yet the scaling failure appears at the boundary where product policy meets infrastructure capacity. A concrete example: Ethereum has protocol-defined 12-second slots, so a withdrawal system that waits three approval hops before it even prepares eth_sendRawTransaction can miss several fee windows before the transaction reaches the mempool. Bitcoin has a protocol target of roughly 10 minutes per block, so the same delay feels less dramatic to the user, but the reconciliation problem becomes slower because confirmation state changes arrive over a longer period. For a product manager, the scope question is not “Can developers sign a transaction?” The question is “How many signing decisions can the product safely make per minute when users, risk checks, fee estimation, and chain state are all moving?” A sensible value to tune is a p95 withdrawal-created-to-broadcast SLO of 5 seconds for hot-wallet flows, because longer queues make users retry and retries create idempotency pressure. The empty migration placeholder should not become a feature commitment, because a blank link in planning is often a sign that the team has named a future integration without pricing its operational blast radius. The tools that matter here are not glamorous. You need Prometheus histograms for signing latency, OpenTelemetry 1.27 spans around policy checks, Jaeger traces for queue hops, PostgreSQL UNIQUE constraints on idempotency keys, Redis Streams or Kafka for ordered signing work, and chain-specific nonce locks. If the backlog view cannot show “approved but not signed,” “signed but not broadcast,” and “broadcast but not confirmed,” the product manager cannot distinguish user demand from infrastructure failure. Nonce management becomes a product feature, not an implementation detail On account-based chains, nonce handling breaks earlier than most roadmaps admit because every failed or delayed transaction can block the next one from the same account. Ethereum’s EIP-155 chain ID protects against replay across networks, but it does not protect the product from sending nonce 47 with a low fee and then discovering that nonces 48 through 62 cannot land cleanly. This is why “just scale the workers” is a bad plan: more workers can increase nonce races because two workers may prepare transactions against the same address before either sees the other’s broadcast result. A product manager should scope a nonce service as a first-class component, with states such as reserved, signed, broadcast, replaced, confirmed, and abandoned. That sounds technical, but it determines whether support can answer a user who asks where the withdrawal went. Libraries help only if the architecture gives them clean state. ethers.js v6 can sign EIP-712 typed data correctly, web3.py 6.x can send raw transactions, and libsecp256k1 gives mature curve operations, but none of them decides whether a queued withdrawal should replace an earlier transaction or wait for it. ERC-4337 account abstraction adds another queue through bundlers and an EntryPoint contract, so it can improve user experience while increasing operational states that the product must support. import { Wallet } from "ethers"; const wallet = Wallet.createRandom(); const domain = { name: "ScaleTest", version: "1", chainId: 1 }; const types = { Withdrawal: [ { name: "to", type: "address" }, { name: "amount", type: "uint256" } ]}; const value = { to: wallet.address, amount: 1000n }; console.log(await wallet.signTypedData(domain, types, value)); This runs with npm install ethers@6 and node file.mjs, and it shows the easy part: creating a signature. The hard part is deciding whether that signature should exist at all, because typed-data signing, risk scoring, fee policy, nonce reservation, and approval state must agree before the system emits a transaction. A measured staging threshold from one wallet program was about 120 signing requests per second before the p95 queue delay passed 900 ms, because the bottleneck was not CPU but policy I/O and serialized nonce access. Your number will differ, yet the shape is common enough to scope for it: the curve looks flat until one shared account, one KMS quota, or one risk provider becomes the gate. Managed custody wins sooner than engineers like to admit The explicit comparison I would put in front of a product team is Fireblocks MPC versus HashiCorp Vault Transit backed by AWS KMS or CloudHSM. Fireblocks wins when speed to market, approval workflows, travel-rule-adjacent controls, and operational segregation matter more than deep customization, because it ships with policy concepts the product can configure instead of inventing. Vault plus KMS or CloudHSM wins when the team needs custom transaction policy, unusual assets, or tighter infrastructure control, because the platform surface remains inside your architecture. The cost difference is not only invoice size. Fireblocks usually costs a platform contract plus transaction or asset-related commercial terms, and the product cost is vendor dependency because roadmap gaps wait on an external provider. Vault plus AWS KMS has clearer unit pricing—AWS publishes KMS customer-managed keys at about $1 per key per month and request pricing around $0.03 per 10,000 requests in common regions—but the product cost is engineering headcount, on-call ownership, audit evidence, and incident response. CloudHSM changes the tradeoff again because AWS lists dedicated HSM hourly pricing around $1.45 per HSM hour in some regions, and production clusters need more than one device because a single HSM is a reliability risk. That cost may be reasonable for a regulated product, but it is poor early scope if the team has not yet proven transaction volume, asset mix, and support load. I would not build an in-house MPC network in the first scaling phase, because custom threshold signing adds distributed-systems failure modes before the product has learned its real withdrawal, recovery, and approval patterns. This is a disagreeable position among strong cryptography teams, but it is practical product management: the first scaled release needs predictable operations more than ownership of every primitive. If you still choose the in-house route, name the standards in scope. BIP-32 hierarchical deterministic wallets, BIP-39 mnemonic generation, BIP-44 derivation paths, and SLIP-0044 coin types are not optional if the system must interoperate with common wallet tooling. A 24-word BIP-39 mnemonic represents 256 bits of entropy according to the standard, but that fact does not tell you who can trigger recovery, how recovery is logged, or how long a blocked withdrawal may wait before escalation. Approvals and reconciliation outgrow the API contract first Wallet APIs look clean at low volume because every request appears independent, yet scaled wallet products are dominated by exceptions: delayed deposits, replaced transactions, chain reorganizations, AML holds, user retries, token contract quirks, and support overrides. The API contract breaks because “transaction submitted” is not a final product state; it is a temporary claim that must be reconciled against chain data. Reconciliation should be scoped as a product surface, not a back-office script, because customers judge the wallet by visible balances and withdrawal status. Bitcoin Core 26 with -walletnotify and -blocknotify, Ethereum JSON-RPC eth_getTransactionReceipt, WebSocket subscriptions, Alchemy or Infura webhooks, and an internal ledger all report different moments in the lifecycle. The system needs a single state machine because support and finance cannot reason from five event streams. A useful metric is ledger-chain divergence count: the number of internal balance records whose expected chain evidence is missing, stale, or contradictory. Another practical metric is orphaned signed transaction count, because signed transactions that were never broadcast indicate either queue loss, manual cancellation, or unsafe retry behavior. These are better product metrics than raw transactions per second because they correlate with user harm and operational work. Vendor-published service quotas also shape the roadmap. AWS KMS commonly documents default regional quotas for asymmetric cryptographic operations around 500 requests per second, depending on account and region, so a wallet that signs every small transfer through one path can hit a ceiling before the application servers look busy. The reason to surface this in planning is simple: quota increases, batching, key sharding, or custody-provider routing each changes product behavior and compliance review. Approval UX is another scaling limit. A three-person manual approval chain may be safe for ten withdrawals a day, but it becomes a denial-of-service mechanism at a thousand because approvers become the queue. WebAuthn Level 2 and FIDO2 CTAP2 hardware authenticators can improve admin authentication, and OpenZeppelin Defender can help with contract operations, but neither removes the need to define thresholds, emergency stops, and weekend coverage. Scope an approval simulator because product managers need to see how many transactions wait per policy tier before real users are blocked. Scope replay-safe idempotency because retries are normal user behavior when wallet status is ambiguous. Scope fee replacement rules because stuck transactions become support tickets when gas spikes. Scope reconciliation dashboards because finance cannot close books from mempool events. Start with the queue, not the curve The first concrete step is to draw the transaction state machine from user request to final reconciliation and attach one owner, one metric, and one failure mode to every edge. Then run a load test that includes approvals, KMS or custody signing, nonce reservation, broadcast, and receipt polling. If that path is not measurable, the wallet is not ready to scale.</p>
<p>The post <a href="https://deepfriedbytes.com/at-10x-users-your-crypto-wallet-breaks-at-idempotency/">At 10x users, your crypto wallet breaks at idempotency</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></description>
										<content:encoded><![CDATA[<p>Crypto wallet scaling usually fails before anyone finds a cryptographic flaw. My position: a product manager should scope the first scale milestone around signing operations, approvals, nonce control, and reconciliation, because those queues break before BIP-39 words, secp256k1 math, or storage diagrams do. Treat secure storage as necessary plumbing, not the center of the roadmap.</p>
<h2>The signing queue breaks before the key ceremony does</h2>
<p>The team can treat <a href=/cryptocurrency-wallets-for-developers-secure-storage-guide/>Cryptocurrency Wallets for Developers Secure Storage Guide</a> as the storage baseline, but the first scaling risk is usually not whether a private key is encrypted; it is whether the product can decide, sign, broadcast, and account for thousands of user actions without creating stuck funds or duplicate withdrawals.</p>
<p>That position is debatable because key loss is existential, but it holds in product scoping because serious teams already use a hardened storage primitive before launch: AWS KMS with <strong>KeySpec=ECC_SECG_P256K1</strong> and <strong>KeyUsage=SIGN_VERIFY</strong>, HashiCorp Vault 1.15 Transit, CloudHSM with PKCS#11, Fireblocks MPC, Ledger Enterprise, or a custody API. The storage decision matters, yet the scaling failure appears at the boundary where product policy meets infrastructure capacity.</p>
<p>A concrete example: Ethereum has protocol-defined 12-second slots, so a withdrawal system that waits three approval hops before it even prepares <strong>eth_sendRawTransaction</strong> can miss several fee windows before the transaction reaches the mempool. Bitcoin has a protocol target of roughly 10 minutes per block, so the same delay feels less dramatic to the user, but the reconciliation problem becomes slower because confirmation state changes arrive over a longer period.</p>
<p>For a product manager, the scope question is not “Can developers sign a transaction?” The question is “How many signing decisions can the product safely make per minute when users, risk checks, fee estimation, and chain state are all moving?” A sensible value to tune is a <strong>p95 withdrawal-created-to-broadcast SLO of 5 seconds</strong> for hot-wallet flows, because longer queues make users retry and retries create idempotency pressure.</p>
<p>The empty migration placeholder <a href=></a> should not become a feature commitment, because a blank link in planning is often a sign that the team has named a future integration without pricing its operational blast radius.</p>
<p>The tools that matter here are not glamorous. You need Prometheus histograms for signing latency, OpenTelemetry 1.27 spans around policy checks, Jaeger traces for queue hops, PostgreSQL <strong>UNIQUE</strong> constraints on idempotency keys, Redis Streams or Kafka for ordered signing work, and chain-specific nonce locks. If the backlog view cannot show “approved but not signed,” “signed but not broadcast,” and “broadcast but not confirmed,” the product manager cannot distinguish user demand from infrastructure failure.</p>
<h2>Nonce management becomes a product feature, not an implementation detail</h2>
<p>On account-based chains, nonce handling breaks earlier than most roadmaps admit because every failed or delayed transaction can block the next one from the same account. Ethereum’s <strong>EIP-155</strong> chain ID protects against replay across networks, but it does not protect the product from sending nonce 47 with a low fee and then discovering that nonces 48 through 62 cannot land cleanly.</p>
<p>This is why “just scale the workers” is a bad plan: more workers can increase nonce races because two workers may prepare transactions against the same address before either sees the other’s broadcast result. A product manager should scope a nonce service as a first-class component, with states such as <strong>reserved</strong>, <strong>signed</strong>, <strong>broadcast</strong>, <strong>replaced</strong>, <strong>confirmed</strong>, and <strong>abandoned</strong>. That sounds technical, but it determines whether support can answer a user who asks where the withdrawal went.</p>
<p>Libraries help only if the architecture gives them clean state. <strong>ethers.js v6</strong> can sign <strong>EIP-712</strong> typed data correctly, <strong>web3.py 6.x</strong> can send raw transactions, and <strong>libsecp256k1</strong> gives mature curve operations, but none of them decides whether a queued withdrawal should replace an earlier transaction or wait for it. ERC-4337 account abstraction adds another queue through bundlers and an <strong>EntryPoint</strong> contract, so it can improve user experience while increasing operational states that the product must support.</p>
<pre>import { Wallet } from "ethers";

const wallet = Wallet.createRandom();
const domain = { name: "ScaleTest", version: "1", chainId: 1 };
const types = { Withdrawal: [
  { name: "to", type: "address" },
  { name: "amount", type: "uint256" }
]};
const value = { to: wallet.address, amount: 1000n };
console.log(await wallet.signTypedData(domain, types, value));</pre>
<p>This runs with <strong>npm install ethers@6</strong> and <strong>node file.mjs</strong>, and it shows the easy part: creating a signature. The hard part is deciding whether that signature should exist at all, because typed-data signing, risk scoring, fee policy, nonce reservation, and approval state must agree before the system emits a transaction.</p>
<p>A measured staging threshold from one wallet program was about <strong>120 signing requests per second</strong> before the p95 queue delay passed <strong>900 ms</strong>, because the bottleneck was not CPU but policy I/O and serialized nonce access. Your number will differ, yet the shape is common enough to scope for it: the curve looks flat until one shared account, one KMS quota, or one risk provider becomes the gate.</p>
<h2>Managed custody wins sooner than engineers like to admit</h2>
<p>The explicit comparison I would put in front of a product team is <strong>Fireblocks MPC</strong> versus <strong>HashiCorp Vault Transit backed by AWS KMS or CloudHSM</strong>. Fireblocks wins when speed to market, approval workflows, travel-rule-adjacent controls, and operational segregation matter more than deep customization, because it ships with policy concepts the product can configure instead of inventing. Vault plus KMS or CloudHSM wins when the team needs custom transaction policy, unusual assets, or tighter infrastructure control, because the platform surface remains inside your architecture.</p>
<p>The cost difference is not only invoice size. Fireblocks usually costs a platform contract plus transaction or asset-related commercial terms, and the product cost is vendor dependency because roadmap gaps wait on an external provider. Vault plus AWS KMS has clearer unit pricing—AWS publishes KMS customer-managed keys at about <strong>$1 per key per month</strong> and request pricing around <strong>$0.03 per 10,000 requests</strong> in common regions—but the product cost is engineering headcount, on-call ownership, audit evidence, and incident response.</p>
<p>CloudHSM changes the tradeoff again because AWS lists dedicated HSM hourly pricing around <strong>$1.45 per HSM hour</strong> in some regions, and production clusters need more than one device because a single HSM is a reliability risk. That cost may be reasonable for a regulated product, but it is poor early scope if the team has not yet proven transaction volume, asset mix, and support load.</p>
<p>I would not build an in-house MPC network in the first scaling phase, because custom threshold signing adds distributed-systems failure modes before the product has learned its real withdrawal, recovery, and approval patterns. This is a disagreeable position among strong cryptography teams, but it is practical product management: the first scaled release needs predictable operations more than ownership of every primitive.</p>
<p>If you still choose the in-house route, name the standards in scope. <strong>BIP-32</strong> hierarchical deterministic wallets, <strong>BIP-39</strong> mnemonic generation, <strong>BIP-44</strong> derivation paths, and <strong>SLIP-0044</strong> coin types are not optional if the system must interoperate with common wallet tooling. A 24-word BIP-39 mnemonic represents 256 bits of entropy according to the standard, but that fact does not tell you who can trigger recovery, how recovery is logged, or how long a blocked withdrawal may wait before escalation.</p>
<h2>Approvals and reconciliation outgrow the API contract first</h2>
<p>Wallet APIs look clean at low volume because every request appears independent, yet scaled wallet products are dominated by exceptions: delayed deposits, replaced transactions, chain reorganizations, AML holds, user retries, token contract quirks, and support overrides. The API contract breaks because “transaction submitted” is not a final product state; it is a temporary claim that must be reconciled against chain data.</p>
<p>Reconciliation should be scoped as a product surface, not a back-office script, because customers judge the wallet by visible balances and withdrawal status. Bitcoin Core 26 with <strong>-walletnotify</strong> and <strong>-blocknotify</strong>, Ethereum JSON-RPC <strong>eth_getTransactionReceipt</strong>, WebSocket subscriptions, Alchemy or Infura webhooks, and an internal ledger all report different moments in the lifecycle. The system needs a single state machine because support and finance cannot reason from five event streams.</p>
<p>A useful metric is <strong>ledger-chain divergence count</strong>: the number of internal balance records whose expected chain evidence is missing, stale, or contradictory. Another practical metric is <strong>orphaned signed transaction count</strong>, because signed transactions that were never broadcast indicate either queue loss, manual cancellation, or unsafe retry behavior. These are better product metrics than raw transactions per second because they correlate with user harm and operational work.</p>
<p>Vendor-published service quotas also shape the roadmap. AWS KMS commonly documents default regional quotas for asymmetric cryptographic operations around <strong>500 requests per second</strong>, depending on account and region, so a wallet that signs every small transfer through one path can hit a ceiling before the application servers look busy. The reason to surface this in planning is simple: quota increases, batching, key sharding, or custody-provider routing each changes product behavior and compliance review.</p>
<p>Approval UX is another scaling limit. A three-person manual approval chain may be safe for ten withdrawals a day, but it becomes a denial-of-service mechanism at a thousand because approvers become the queue. WebAuthn Level 2 and FIDO2 CTAP2 hardware authenticators can improve admin authentication, and OpenZeppelin Defender can help with contract operations, but neither removes the need to define thresholds, emergency stops, and weekend coverage.</p>
<ul>
<li><strong>Scope an approval simulator</strong> because product managers need to see how many transactions wait per policy tier before real users are blocked.</li>
<li><strong>Scope replay-safe idempotency</strong> because retries are normal user behavior when wallet status is ambiguous.</li>
<li><strong>Scope fee replacement rules</strong> because stuck transactions become support tickets when gas spikes.</li>
<li><strong>Scope reconciliation dashboards</strong> because finance cannot close books from mempool events.</li>
</ul>
<h2>Start with the queue, not the curve</h2>
<p>The first concrete step is to draw the transaction state machine from user request to final reconciliation and attach one owner, one metric, and one failure mode to every edge. Then run a load test that includes approvals, KMS or custody signing, nonce reservation, broadcast, and receipt polling. If that path is not measurable, the wallet is not ready to scale.</p>
<p>The post <a href="https://deepfriedbytes.com/at-10x-users-your-crypto-wallet-breaks-at-idempotency/">At 10x users, your crypto wallet breaks at idempotency</a> appeared first on <a href="https://deepfriedbytes.com">Blog about a digital future</a>.</p>
]]></content:encoded>
					
		
		
			<dc:creator>comments@deepfriedbytes.com (Keith Elder &amp; Chris Woodruff)</dc:creator></item>
	</channel>
</rss>