Not because I don’t use them. I do. A lot.

At this point, they probably write more than 90% of the code I produce. If you measure output in lines of code per second, I’m obviously much faster than I was before. But if you measure the thing that actually matters - shipping something useful to production - the improvement is much smaller. Certainly nowhere near the 10-100x some people keep claiming.

If something really made a team 10x faster, that would not be subtle. Something that used to take a month would now take three days. Something that used to take a year would be done in five weeks. We would all notice.

I think part of the confusion is that we’re mistaking speed for velocity.

Speed is how quickly code appears on the screen. Velocity is whether the system is actually moving in the right direction. LLMs are extremely good at speed. But the direction problem is still mostly there, and in some ways more visible than before.

The easy part was never the hard part

Getting something that vaguely works on your laptop was never the whole job. The job is making it hold up inside a real codebase, with actual constraints, with other people touching it, using it, deploying it, debugging it and living with it afterwards.

People point at how quickly they can generate code now, and they are not lying. They probably are generating code much faster. But writing the code was never the only bottleneck, and in many cases not even the main one.

The hard parts are still the hard parts: deciding what should exist, agreeing on the shape of it, and dealing with the consequences afterwards.

More output, same throughput

LLMs absolutely increase output. They make it easier to start. Easier to explore. Easier to get to a rough first version. Easier to open ten threads of work instead of one.

But more output is not the same thing as more progress. Quite often it just means more unfinished work.

LLMs move quickly. But they do not know where to go in the deeper sense. They do not know which tradeoff matters, which future constraint will hurt, which messy edge case is actually the product, or which feature should simply not exist.

Why we keep pressing the button

Now there is always something that could be running. Another prompt. Another refactor. Another tiny tool. Another MVP. Another experiment that is suddenly cheap enough to start, but not cheap enough to finish.

And of course we keep pressing the button.

Why wouldn’t we?

You describe something in plain English, wait a few seconds, and there it is: a first iteration, a quick hack, something working. Even when the result is mediocre, it still gives you that little reward of momentum. Something moved. Something appeared. The machine answered.

The loop is so short, and the payoff so immediate, that it becomes difficult not to reach for it again. Not because the result is always good, but because the experience itself is rewarding.

Output does not equal value

If you spend enough time close to the actual work, it becomes very hard to get excited by demos of prototypes, huge line counts, or vague claims about replacing entire functions of a company.

Because there currently is an enormous amount of noise.

And one of the easiest ways to generate noise right now is to confuse “I can produce more artifacts” with “I can create more value”.

Those are not the same thing.

This is also why some of the current hype feels shallow. Stories about how many lines got written, endless demos of things that vaguely work on first pass. None of that really addresses whether anything of value got created, other than the act of making more software-shaped output.

Friction was doing us a favour

LLMs remove friction from building. But friction was sometimes doing us a favour.

When building something took longer, you had to choose more carefully. You had to kill more ideas earlier. You had to be more selective about which direction deserved a week of your time.

Now the cost of indulging every passing idea is much lower, so you can follow all of them a little bit. The result is not necessarily more clarity. Sometimes it is the opposite.

You end up with more code, more branches, more prototypes, more motion.

But some things just take time.

Understanding a domain takes time. Building trust in a product takes time. Seeing how real users respond takes time. Learning whether a project deserves years of maintenance takes time. No amount of generated code removes that.

Maybe that is why so much of this can feel vaguely hopeless even while the tooling keeps improving. LLMs get better and better at helping us do work that was never the true bottleneck.

And yes, I use LLMs heavily. I am not writing this from the outside. They are useful. Sometimes extremely useful. But I also think they make it easier to confuse acceleration with direction.

And if you accelerate without improving direction, you do not necessarily get where you want to go faster.

You just get lost at higher speed.

The whole thing is about 800 lines across five packages. It takes markdown files, runs them through an asset pipeline and template renderer, and outputs static HTML that gets deployed to Cloudflare Pages. Nothing revolutionary - but every piece is mine, and I understand exactly what happens when I run go run cmd/web/main.go -build-site.

The build pipeline

The entry point wires everything together in sequence:

 1// 1. Bundle and minify assets, get back a manifest
 2assetBundler, err := assets.NewBundler(cfg.Assets, "public", staticDir)
 3manifest, err := assetBundler.Build()
 4
 5// 2. Pass manifest to builder so templates can resolve cache-busted URLs
 6b, err := builder.New(builder.Options{
 7    TemplateDir:   "templates",
 8    OutputDir:     staticDir,
 9    AssetManifest: manifest,
10    Target:        target,
11})
12
13// 3. Parse markdown, render HTML, generate sitemap and RSS
14b.Build()

Assets must be bundled first because the builder needs the manifest to render templates correctly. Templates reference assets like {{ asset "app.css" }}, and that function needs to know that app.css maps to /css/app.a1b2c3d4.css. Without the manifest, the links would be wrong.

That’s the whole build. Three steps.

Parsing markdown

The parser is the simplest piece. Markdown files have YAML frontmatter between --- delimiters, followed by content:

1---
2title: "Some post"
3description: "About something"
4date: 2026-02-21
5published: true
6view: blog
7---
8
9The actual content here...

Parsing this is a bytes.SplitN call:

 1func (p *Parser) Parse(path string, data []byte) (*Content, error) {
 2    parts := bytes.SplitN(data, []byte("---\n"), 3)
 3    if len(parts) != 3 {
 4        return nil, fmt.Errorf("invalid markdown format: expected frontmatter between --- delimiters")
 5    }
 6
 7    var meta Metadata
 8    if err := yaml.Unmarshal(parts[1], &meta); err != nil {
 9        return nil, fmt.Errorf("failed to parse frontmatter: %w", err)
10    }
11
12    if meta.Slug == "" {
13        slug := strings.TrimSuffix(path, filepath.Ext(path))
14        slug = strings.TrimPrefix(slug, "./")
15        meta.Slug = filepath.ToSlash(strings.Trim(slug, "/"))
16    }
17
18    return &Content{
19        Metadata: meta,
20        Body:     bytes.TrimSpace(parts[2]),
21    }, nil
22}

Split on ---\n, unmarshal the middle part as YAML, derive the slug from the filename. That’s it. No external frontmatter library, no special parser - just SplitN into three parts and take the pieces.

The ParseDir method walks a directory and collects all .md files, sorted newest first. One thing I like about it - it takes an fs.FS instead of a raw directory path. In the build pipeline it gets os.DirFS("content"), but it means the parser is easy to test with fstest.MapFS without touching the filesystem.

 1func (p *Parser) ParseDir(fsys fs.FS, includeUnpublished bool) ([]Content, error) {
 2    var contents []Content
 3
 4    err := fs.WalkDir(fsys, ".", func(path string, d fs.DirEntry, err error) error {
 5        if d.IsDir() || filepath.Ext(path) != ".md" {
 6            return nil
 7        }
 8
 9        data, err := fs.ReadFile(fsys, path)
10        // ...parse and filter unpublished...
11        contents = append(contents, *content)
12        return nil
13    })
14
15    sort.Slice(contents, func(i, j int) bool {
16        return contents[i].Metadata.Date.After(contents[j].Metadata.Date)
17    })
18
19    return contents, nil
20}

Asset bundling and cache busting

The asset pipeline takes source CSS and JS from /public, bundles them, minifies them, and generates cache-busted filenames. It’s configured in config.yaml:

 1assets:
 2  css:
 3    bundles:
 4      - name: app
 5        files:
 6          - reset.css
 7          - style.css
 8  js:
 9    bundles:
10      - name: app
11        files:
12          - main.js

The bundler concatenates files in order, minifies the result, then hashes it:

 1func (b *Bundler) bundleCSS(bundle Bundle) error {
 2    var combined strings.Builder
 3
 4    for _, file := range bundle.Files {
 5        data, err := os.ReadFile(filepath.Join(b.publicDir, "css", file))
 6        if err != nil {
 7            return fmt.Errorf("failed to read %s: %w", file, err)
 8        }
 9        combined.Write(data)
10        combined.WriteString("\n")
11    }
12
13    minified, err := b.minifier.Bytes("text/css", []byte(combined.String()))
14    if err != nil {
15        slog.Warn("failed to minify CSS, using original", "bundle", bundle.Name, "error", err)
16        minified = []byte(combined.String())
17    }
18
19    hash := b.hash(minified)
20    filename := fmt.Sprintf("%s.%s.css", bundle.Name, hash[:8])
21
22    os.WriteFile(filepath.Join(b.outputDir, "css", filename), minified, 0o644)
23
24    b.manifest[bundle.Name+".css"] = "/css/" + filename
25    return nil
26}

So app.css becomes /css/app.a1b2c3d4.css. The manifest maps "app.css" to "/css/app.a1b2c3d4.css", and that manifest gets passed to the template renderer.

The nice thing about hashing the output rather than the input is that the filename only changes when the actual delivered bytes change. Reformat your source CSS, add comments, reorganize - if the minified output is identical, the hash stays the same and browsers keep their cached version.

Templates and the asset function

Templates are split into three directories: common/ (shared layout, header, footer), views/ (content type templates like blog, page), and pages/ (static pages like index, 404).

The renderer concatenates common templates with each view template, so every view inherits the shared layout:

 1func (r *Renderer) registerTemplates(dir, prefix, common string) error {
 2    return filepath.WalkDir(dir, func(path string, d fs.DirEntry, walkErr error) error {
 3        data, err := os.ReadFile(path)
 4
 5        var templateContent strings.Builder
 6        templateContent.WriteString(common)
 7        templateContent.Write(data)
 8
 9        r.templates[renderName] = template.Must(
10            template.New(renderName).
11                Funcs(r.templateFuncs()).
12                Parse(templateContent.String()),
13        )
14        return nil
15    })
16}

The asset template function is where the manifest pays off:

1"asset": func(name string) string {
2    if url, ok := r.manifest[name]; ok {
3        return url
4    }
5    return "/" + name
6},

In a template, {{ asset "app.css" }} resolves to /css/app.a1b2c3d4.css. Simple lookup, no magic.

This pairs with the cache config:

1cache:
2  html: max-age=0, must-revalidate
3  assets: public, max-age=31536000, immutable

HTML is never cached. Assets are cached forever and marked immutable. When the CSS changes, the hash changes, the URL changes, and browsers fetch the new version. Old cached versions just expire naturally.

Markdown rendering

For converting markdown to HTML, I use goldmark.

 1md := goldmark.New(
 2    goldmark.WithParserOptions(
 3        parser.WithAutoHeadingID(),
 4        parser.WithASTTransformers(
 5            util.Prioritized(&imagePathTransformer{}, 100),
 6        ),
 7    ),
 8    goldmark.WithExtensions(
 9        extension.GFM,
10        extension.Footnote,
11        extension.Typographer,
12        highlighting.NewHighlighting(
13            highlighting.WithStyle("catppuccin-mocha"),
14            highlighting.WithFormatOptions(
15                chromahtml.WithLineNumbers(true),
16                chromahtml.WithClasses(true),
17            ),
18        ),
19    ),
20)

A few choices here:

Syntax highlighting uses Chroma with the Catppuccin Mocha theme, rendered as CSS classes rather than inline styles. This means the color scheme lives in a stylesheet instead of being baked into every code block’s HTML. Change the theme in one place, every code block updates.

The image path transformer is a custom AST transformer that rewrites relative image paths to absolute ones:

 1func (t *imagePathTransformer) Transform(node *ast.Document, reader text.Reader, pc parser.Context) {
 2    _ = ast.Walk(node, func(n ast.Node, entering bool) (ast.WalkStatus, error) {
 3        if !entering {
 4            return ast.WalkContinue, nil
 5        }
 6
 7        img, ok := n.(*ast.Image)
 8        if !ok {
 9            return ast.WalkContinue, nil
10        }
11
12        dest := string(img.Destination)
13
14        // Skip already-absolute URLs
15        if strings.HasPrefix(dest, "http") || strings.HasPrefix(dest, "/") {
16            return ast.WalkContinue, nil
17        }
18
19        dest = strings.TrimPrefix(dest, "./")
20        img.Destination = []byte("/" + dest)
21
22        return ast.WalkContinue, nil
23    })
24}

This means I can write ![alt](images/photo.png) in markdown and it becomes /images/photo.png in the HTML. Content images live next to the markdown in /content/images/ and get copied to /static/images/ during the build.

A nice side effect is that this plays well with Obsidian as a markdown editor. I can drag and drop images directly into a post and they just work - Obsidian writes the relative path, the transformer rewrites it at build time. No manual path fiddling.

Goldmark’s AST transformer API makes this surprisingly clean - you walk the tree, find image nodes, rewrite their destinations.

Building a page

With all the pieces in place, building a single page is straightforward:

 1func (b *Builder) buildPage(content internalparser.Content, contents []internalparser.Content) error {
 2    // Convert markdown to HTML
 3    var buf bytes.Buffer
 4    b.markdown.Convert(content.Body, &buf)
 5
 6    // Render through the template
 7    html, err := b.renderer.Render(content.Metadata.View, map[string]any{
 8        "title":       content.Metadata.Title,
 9        "description": content.Metadata.Description,
10        "body":        buf.String(),
11        "date":        content.Metadata.Date,
12        "site":        b.fullSiteContext(contents),
13    })
14
15    // Minify the final HTML
16    minified, _ := b.minifier.Bytes("text/html", html)
17
18    // Write to disk
19    outFile := filepath.Join(b.outputDir, content.Metadata.Slug+".html")
20    return os.WriteFile(outFile, minified, 0o644)
21}

Markdown in, HTML out, minified, written to disk. The view field in the frontmatter determines which template renders it - blog uses templates/views/blog.html, page uses templates/views/page.html. Each template inherits the common layout, so there’s no duplication.

Deployment targets

The builder has a concept of deployment targets. Right now there are two: local and cloudflare. The local target just generates HTML files. The Cloudflare target also generates _headers and _redirects files that Cloudflare Pages reads at deploy time:

 1func (b *Builder) generateTargetArtifacts() error {
 2    switch b.target {
 3    case TargetCloudflare:
 4        if err := b.writeHeaders(); err != nil {
 5            return fmt.Errorf("failed to write headers file: %w", err)
 6        }
 7        if err := b.writeRedirects(); err != nil {
 8            return fmt.Errorf("failed to write redirects file: %w", err)
 9        }
10    }
11    return nil
12}

The headers file maps path patterns to cache-control values from the config. The redirects file handles two things: custom redirects from config.yaml (like old blog URLs that moved) and canonical redirects so /blog/some-post/index.html redirects to /blog/some-post.

Adding a new deployment target - say, Netlify or S3 - would just be another case in the switch.

OG images

Every post needs a social sharing image for Twitter/LinkedIn previews. I didn’t want to browse Unsplash for a vaguely related stock photo every time I published something, so I just didn’t.

Instead, the build generates them. If a post doesn’t have a custom image in its frontmatter, the builder creates one using the title and description:

 1func (b *Builder) generateOGImages(contents []internalparser.Content) {
 2    for i := range contents {
 3        if contents[i].Metadata.Image != "" {
 4            continue
 5        }
 6        slug := contents[i].Metadata.Slug
 7        safeSlug := strings.ReplaceAll(slug, "/", "-")
 8        webPath, err := ogimage.Generate(
 9            b.outputDir,
10            safeSlug,
11            contents[i].Metadata.Title,
12            contents[i].Metadata.Description,
13        )
14        if err != nil {
15            slog.Warn("failed to generate og image", "slug", slug, "error", err)
16            continue
17        }
18        contents[i].Metadata.Image = webPath
19    }
20}

The generator itself uses gg (a 2D graphics library) with embedded JetBrains Mono fonts. It renders at 2x resolution and downscales for sharp text:

 1//go:embed fonts/JetBrainsMono-Bold.ttf
 2var boldFont []byte
 3
 4const (
 5    width   = 1200
 6    height  = 630
 7    scale   = 2
 8)
 9
10func Generate(outputDir, slug, title, description string) (string, error) {
11    dc := gg.NewContext(width*scale, height*scale)
12
13    // Dark background
14    dc.SetColor(color.NRGBA{R: 0x0A, G: 0x0A, B: 0x0A, A: 0xFF})
15    dc.Clear()
16
17    // Theme-colored square in the corner
18    dc.SetColor(color.NRGBA{R: 0x24, G: 0x33, B: 0xf3, A: 0xFF})
19    dc.DrawRectangle(padding*s, padding*s, boxSize, boxSize)
20    dc.Fill()
21
22    // Draw title, site name, description...
23
24    // Downscale 2x → 1x for crisp output
25    dst := image.NewRGBA(image.Rect(0, 0, width, height))
26    draw.CatmullRom.Scale(dst, dst.Bounds(), dc.Image(), dc.Image().Bounds(), draw.Over, nil)
27
28    // Write PNG
29    return "/images/og/" + slug + ".png", png.Encode(f, dst)
30}

The 2x render trick is the same idea as Retina displays - draw everything at double resolution, then downscale with a good interpolation algorithm (CatmullRom). The result is noticeably sharper than rendering at 1x, especially for small text like the description.

Here’s what it generated for this post:

No Figma, no Canva, no manual step. Write a post, run the build, get an image.

Was it worth it?

Objectively? No. Hugo would do everything I need and more. But building this taught me things I wouldn’t have learned otherwise:

• How cache busting actually works end-to-end, from hashing bytes to setting headers
• How goldmark’s AST transformer API lets you rewrite content at the tree level
• How little code a static site generator actually needs

The whole thing compiles in under a second and builds the site in about 50ms. There are no node_modules, no config files for the config files, no plugins. Just Go, some markdown, and a config.yaml.

Sometimes the best tool is the one you understand completely.

It sounds like a paradigm shift until you look at what people actually do. They write FAQ sections. They add structured data. They tweak content to show up in AI Overviews.

In other words, they do SEO and call it AEO. But WebMCP might actually be the first thing that changes that.

WebMCP

WebMCP is a proposal from Google and Microsoft and that recently opened up for early preview in Chrome. It’s a JavaScript API that lets websites expose structured tools to AI agents running in the browser. Not content. Not markup hints. Actual callable functions.

Think of it like this: instead of hoping an AI agent can figure out how to navigate your e-commerce site by reading your DOM, you hand it a function called addToCart(productId, quantity) with a schema and a description. The agent calls it, your JavaScript runs, and the UI updates. No guessing, no brittle screen-scraping.

The proposal defines two types of APIs:

• Declarative: Standard actions defined directly in HTML forms
• Imperative: More dynamic interactions that require JavaScript

Here’s a simplified example from the spec:

 1if ("modelContext" in window.navigator) {
 2    window.navigator.modelContext.provideContext({
 3        tools: [
 4            {
 5                name: "search-products",
 6                description: "Search for products matching a query",
 7                inputSchema: {
 8                    type: "object",
 9                    properties: {
10                        query: { type: "string", description: "Search terms" },
11                        maxPrice: { type: "number", description: "Maximum price" }
12                    },
13                    required: ["query"]
14                },
15                execute({ query, maxPrice }) {
16                    // Your existing search logic
17                    return { content: [{ type: "text", text: JSON.stringify(results) }] };
18                }
19            }
20        ]
21    });
22}

If you’ve worked with MCP before, this will look familiar. That’s intentional - WebMCP is designed to align with MCP, but instead of running tools on a backend server, the tools run as client-side JavaScript in the browser. The website is the MCP server.

Why this feels different

The “A” in AEO has always stood for “Answer”. Optimize your content so AI models cite you. But that’s still just visibility - you’re optimizing to be quoted.

WebMCP isn’t about being quoted. It’s about being operated. You’re not hoping an agent reads your page. You’re giving it a searchProducts function and letting it call it.

That’s not Answer Engine Optimization. That’s Agent Engine Optimization - or whatever you want to call it. The work is fundamentally different. You’re thinking about your site as an API, not a document. What tools should you expose? What inputs does an agent need? What should it get back?

For e-commerce, it’s product search, cart management, checkout. For SaaS, it might be account actions or configuration. Content sites may not expose a checkout, but they might offer scoped search, filtered retrieval, or citation bundles. Different tools, but still tools.

Either way, writing WebMCP tools is the first time “optimizing for AI” means doing something other than SEO.

Early days

WebMCP is still in early preview. But the direction feels very interesting. We’ve been adding machine-readable layers to the web for years - meta tags, structured data, OpenGraph, web manifests. Each one helped machines understand what a page is. WebMCP is different because you’re telling agents what your page can do.

And when you’re writing those tools - deciding what to expose, how to describe it, what schema to use - that’s AEO. It just happened to be so that that the A was never about answers.

At Precis, I built an internal service that runs Screaming Frog crawls at scale using Cloud Run Jobs. This post walks through the core setup - a simplified version you can deploy in about 30 minutes.

For this proof-of-concept, we’re going to use Cloud Run Jobs, Cloud Storage, and a simple Dockerfile combined with a bash file used as its entrypoint.

Why Cloud Run Jobs

Assuming 4 CPU / 16GB - a size I’ve found works well for a wide variety of crawls (more on that later) - here’s how Cloud Run Jobs compares to an always-on VM of the same size:

You’d need over 6 hours of daily crawl time before Cloud Run Jobs even matches the cost of a single VM.

It also scales horizontally. Need to crawl 10 sites at the same time? Just fire off 10 jobs - each gets its own container with dedicated resources. With a VM, you’d have to run crawls sequentially - and if all crawls need to be ready by the time you get to work in the morning, that single VM becomes a bottleneck fast. Now you’re provisioning multiple VMs, and the cost comparison shifts even further.

What we’re building

The setup is straightforward:

1. Dockerfile that installs Screaming Frog
2. Entrypoint script that runs the crawl
3. Cloud Run Job that executes the container
4. GCS bucket to store exports

Prerequisites

You’ll need:

• Google Cloud Project with billing enabled
• gcloud CLI installed and configured
• Screaming Frog license
• Basic Docker knowledge

Setting up GCP resources

These next steps assume you have gcloud sdk installed and that you are somewhat familiar with GCP.

Note that many of the steps below require you to have these two ENVs set.

1PROJECT_ID="your-gcp-project"
2REGION="your-preferred-region"

Enable the required APIs:

1gcloud services enable artifactregistry.googleapis.com run.googleapis.com

Create a storage bucket for the CSV exports that Screaming Frog will generate. We’ll later mount this bucket to the Cloud Run Job instance.

1gsutil mb -l ${REGION} gs://${PROJECT_ID}-crawl-output

Create a service account for the job and give it permission to read, write and delete files:

1gcloud iam service-accounts create screaming-frog-runner \
2    --display-name="ScreamingFrog Runner"
3
4gcloud projects add-iam-policy-binding ${PROJECT_ID} \
5    --member="serviceAccount:screaming-frog-runner@${PROJECT_ID}.iam.gserviceaccount.com" \
6    --role="roles/storage.objectAdmin"

Store your Screaming Frog license and a persistent machine ID in Secret Manager:

 1# Enable Secret Manager API
 2gcloud services enable secretmanager.googleapis.com
 3
 4# Create license secret
 5echo -n "YOUR-LICENSE-KEY" | gcloud secrets create screaming-frog-license \
 6    --data-file=-
 7
 8# Create a persistent machine ID
 9uuidgen | gcloud secrets create screaming-frog-machine-id \
10    --data-file=-
11
12# Grant access to service account
13gcloud secrets add-iam-policy-binding screaming-frog-license \
14    --member="serviceAccount:screaming-frog-runner@${PROJECT_ID}.iam.gserviceaccount.com" \
15    --role="roles/secretmanager.secretAccessor"
16
17gcloud secrets add-iam-policy-binding screaming-frog-machine-id \
18    --member="serviceAccount:screaming-frog-runner@${PROJECT_ID}.iam.gserviceaccount.com" \
19    --role="roles/secretmanager.secretAccessor"

Building the Docker image

Create a Dockerfile:

 1FROM ubuntu:22.04
 2
 3# Install dependencies
 4RUN apt-get update && apt-get install -y \
 5    openjdk-21-jre \
 6    xvfb \
 7    wget \
 8    ca-certificates \
 9    && rm -rf /var/lib/apt/lists/*
10
11# Download and install Screaming Frog
12RUN wget -O /tmp/screamingfrog.deb \
13    https://download.screamingfrog.co.uk/products/seo-spider/screamingfrogseospider_23.2_all.deb && \
14    apt-get update && \
15    apt-get install -y /tmp/screamingfrog.deb && \
16    rm /tmp/screamingfrog.deb
17
18# Copy entrypoint script
19COPY entrypoint.sh /entrypoint.sh
20RUN chmod +x /entrypoint.sh
21
22ENTRYPOINT ["/entrypoint.sh"]

A few things to note:

• Ubuntu 22.04: Screaming Frog distributes as a .deb package, so any Debian-based image should work fine
• xvfb: Screaming Frog requires a display even in CLI mode, xvfb provides a virtual one so it can run fully headless

Now create entrypoint.sh:

 1#!/bin/bash
 2set -e
 3
 4URL="${CRAWL_URL}"
 5OUTPUT_BASE="${OUTPUT_DIR:-/mnt/crawl-output}"
 6
 7# Create a per-run subfolder: domain/YYYY-MM-DD_HHMMSS
 8DOMAIN=$(echo "$URL" | sed -E 's|https?://([^/]+).*|\1|')
 9TIMESTAMP=$(date -u +"%Y-%m-%d_%H%M%S")
10OUTPUT_DIR="${OUTPUT_BASE}/${DOMAIN}/${TIMESTAMP}"
11mkdir -p "$OUTPUT_DIR"
12
13# Setup license, machine ID, and EULA
14mkdir -p /root/.ScreamingFrogSEOSpider
15echo "${SF_LICENSE}" > /root/.ScreamingFrogSEOSpider/licence.txt
16echo "${SF_MACHINE_ID}" > /root/.ScreamingFrogSEOSpider/machine-id.txt
17cat > /root/.ScreamingFrogSEOSpider/spider.config << 'EOF'
18eula.accepted=15
19EOF
20
21# Run the crawl
22xvfb-run screamingfrogseospider \
23    --headless \
24    --crawl "$URL" \
25    --output-folder "$OUTPUT_DIR" \
26    --export-tabs "Internal:All,External:All,Response Codes:All,Page Titles:All,Meta Description:All,H1:All,Images:All" \
27    --overwrite \
28    --save-crawl
29
30echo "Crawl completed successfully"

The script does four things:

1. License setup: Writes the license key we stored in Secret Manager to the path Screaming Frog expects.
2. Machine ID: Writes the persistent UUID so each container run identifies as the same machine.
3. EULA acceptance: Required for it to run.
4. Runs the crawl: xvfb-run provides the virtual display, and we export a few selected tabs - feel free to edit.

So now in your directory you should have:

.
├── Dockerfile
└── entrypoint.sh

Deploying the Cloud Run Job

Deploy the job directly from source (this builds the image using Cloud Build and deploys in one command):

 1gcloud run jobs deploy screaming-frog-crawler \
 2    --source . \
 3    --region=${REGION} \
 4    --service-account=screaming-frog-runner@${PROJECT_ID}.iam.gserviceaccount.com \
 5    --cpu=4 \
 6    --memory=16Gi \
 7    --max-retries=0 \
 8    --task-timeout=3600 \
 9    --set-env-vars=OUTPUT_DIR=/mnt/crawl-output \
10    --set-secrets=SF_LICENSE=screaming-frog-license:latest,SF_MACHINE_ID=screaming-frog-machine-id:latest \
11    --add-volume name=crawl-storage,type=cloud-storage,bucket=${PROJECT_ID}-crawl-output \
12    --add-volume-mount volume=crawl-storage,mount-path=/mnt/crawl-output

This command will:

• Build your Docker image using Cloud Build
• Push it to Artifact Registry automatically
• Create (or update) the Cloud Run Job
• Mount the bucket we created as a volume

Resource sizing

The 4 CPU, 16GB RAM I found to be a good starting point for most crawls. Scale up to 8 CPU / 32GB for large sites (100K+ URLs).

Important: Screaming Frog periodically checks available disk space and stops the crawl if it detects 5GB or less remaining. On Cloud Run, available memory serves as disk space - there’s no separate disk allocation, even with the GCS mount. So while 2 CPU / 8GB technically works, you’ll be cutting it close with Screaming Frog’s 5GB threshold.

Running crawls

Manual execution:

1gcloud run jobs execute screaming-frog-crawler \
2    --region=${REGION} \
3    --update-env-vars=CRAWL_URL=https://example.com

Check execution status:

1gcloud run jobs executions list \
2    --job=screaming-frog-crawler \
3    --region=${REGION}

View logs:

1gcloud logging read \
2    "resource.type=cloud_run_job AND resource.labels.job_name=screaming-frog-crawler" \
3    --limit=50 \
4    --format=json

Accessing crawl results

You can browse and download files directly from the GCP Console by navigating to your bucket. Or use the CLI:

First create a folder where you want to store the files.

1mkdir -p ./crawl-output/

List crawl outputs:

1gsutil ls gs://${PROJECT_ID}-crawl-output/

Download the latest crawl for a domain:

1LATEST=$(gsutil ls gs://${PROJECT_ID}-crawl-output/domain.tld/ | sort | tail -1)
2gsutil -m cp -r "${LATEST}*" ./crawl-output/

Or download all crawls:

1gsutil -m cp -r gs://${PROJECT_ID}-crawl-output/* ./crawl-output/

The output directory includes:

• internal_all.csv - All internal URLs discovered
• external_all.csv - External links
• response_codes_all.csv - HTTP status codes
• page_titles_all.csv - Page titles
• meta_description_all.csv - Meta descriptions
• h1_all.csv - H1 tags
• images_all.csv - Image inventory
• crawl.seospider - Full crawl file (open in Screaming Frog GUI)

If all went well, you should now have something that looks close to this:

What’s next

This gives you a working setup for running Screaming Frog crawls serverless. From here, you could:

• Add configuration files: Use Screaming Frog’s config files to standardize crawl settings across executions
• Implement progress tracking: Parse log output to report crawl progress in real-time
• Build a web UI: Create a simple interface for managing crawls and viewing results (hint: this is what we built)
• Add notifications: Send alerts when crawls complete or fail
• Track changes: Compare crawls over time to detect new issues

Once deployed, crawls run unattended and you only pay for what you use.

The problem: tight coupling

Let’s say you’re building a user service. You have a database package that handles persistence:

 1package db
 2
 3import "database/sql"
 4
 5type DB struct {
 6    conn *sql.DB
 7}
 8
 9func NewDB(conn *sql.DB) *DB {
10    return &DB{conn: conn}
11}
12
13type User struct {
14    ID       string
15    Username string
16    Email    string
17}
18
19func (db *DB) CreateUser(ctx context.Context, user *User) error {
20    // database implementation
21}
22
23func (db *DB) GetUserByID(ctx context.Context, userID string) (*User, error) {
24    // database implementation
25}

And a user service that uses it:

 1package user
 2
 3import "yourapp/db"
 4
 5type Service struct {
 6    db *db.DB
 7}
 8
 9func NewService(database *db.DB) *Service {
10    return &Service{db: database}
11}
12
13func (s *Service) RegisterUser(ctx context.Context, username, email string) (*db.User, error) {
14    user := &db.User{
15        ID:       generateID(),
16        Username: username,
17        Email:    email,
18    }
19
20    if err := s.db.CreateUser(ctx, user); err != nil {
21        return nil, err
22    }
23
24    return user, nil
25}

This looks reasonable at first, but there are problems:

1. Misplaced domain model: User is in the db package, but it’s a domain concept, not a database concern. It feels wrong there.
2. Can’t test without a database: To test user.Service, you need a real *db.DB. No mocks, no in-memory stores.
3. Tight coupling: The service is tied to the exact database implementation.

The typical workaround is moving User to a shared models package. But now you have a grab-bag package that everyone imports, and you still can’t test without the database.

What does “accept interfaces, return structs” actually mean?

The idea is straightforward:

• When your function or method needs a dependency, accept an interface describing only the behavior you need.
• When your function creates something, return a concrete type, not an interface.

Here’s the same example, redesigned. The user service defines what it needs:

 1package user
 2
 3type User struct {
 4    ID       string
 5    Username string
 6    Email    string
 7}
 8
 9type userStore interface {
10    CreateUser(ctx context.Context, user *User) error
11    GetUserByID(ctx context.Context, userID string) (*User, error)
12    UpdateUser(ctx context.Context, user *User) error
13    DeleteUser(ctx context.Context, userID string) error
14}
15
16type Service struct {
17    userStore userStore
18}
19
20func NewService(userStore userStore) *Service {
21    return &Service{userStore: userStore}
22}
23
24func (s *Service) RegisterUser(ctx context.Context, username, email string) (*User, error) {
25    user := &User{
26        ID:       generateID(),
27        Username: username,
28        Email:    email,
29    }
30
31    if err := s.userStore.CreateUser(ctx, user); err != nil {
32        return nil, err
33    }
34
35    return user, nil
36}

Notice what changed:

• The User type now lives in the user package where it belongs
• The Service accepts a userStore interface, not *db.DB
• The interface defines only what the service needs - four methods
• NewService still returns a concrete *Service, not an interface

No circular dependencies, no shared models package.

Why accept interfaces?

When you define an interface at the point of use, you’re being explicit about your actual requirements. The user.Service above doesn’t need a full database connection. It needs four methods. That’s it.

Since your code depends on an interface, you can swap in a mock for tests:

 1package user
 2
 3type mockUserStore struct {
 4    user *User
 5}
 6
 7func (m *mockUserStore) CreateUser(ctx context.Context, user *User) error {
 8    return nil
 9}
10
11func (m *mockUserStore) GetUserByID(ctx context.Context, id string) (*User, error) {
12    return m.user, nil
13}
14
15func (m *mockUserStore) UpdateUser(ctx context.Context, user *User) error {
16    return nil
17}
18
19func (m *mockUserStore) DeleteUser(ctx context.Context, id string) error {
20    return nil
21}

Now you can test your service logic without touching a database:

 1func TestService_GetUser(t *testing.T) {
 2    expectedUser := &User{ID: "user-123", Username: "testuser"}
 3
 4    store := &mockUserStore{
 5        user: expectedUser,
 6    }
 7
 8    service := NewService(store)
 9
10    user, err := service.GetUser(ctx, "user-123")
11
12    assert.NoError(t, err)
13    assert.Equal(t, expectedUser, user)
14}

No database setup, no cleanup. Just fast, deterministic unit tests.

The same interface can be satisfied by completely different implementations:

• A PostgresUserStore for production
• A MockUserStore for unit tests
• An InMemoryUserStore for integration tests
• A CachedUserStore that wraps another store

Your service doesn’t know or care which one it gets.

Why return structs?

When you return a concrete type, the caller knows exactly what they’re getting. They can see all the methods, all the fields (if exported), and the compiler can help them use it correctly.

1package db
2
3func NewUserStore(pool *pgxpool.Pool) *UserStore {
4    return &UserStore{
5        pool: pool,
6    }
7}

Anyone calling db.NewUserStore knows they’re getting a *db.UserStore. No guessing, no type assertions needed.

In Go, interfaces are satisfied implicitly - there’s no implements keyword. This means the consumer of a package should define what interface they need, not the producer.

Your db.UserStore doesn’t need to know it implements user.userStore. It just has methods. The user.Service defines the interface it requires, and Go’s type system figures out the rest.

If you return an interface from your constructor, you’re forcing all callers to use that interface. But different callers might need different subsets of functionality. By returning a struct, each consumer can define their own interface with just the methods they actually use.

Putting it together

Here’s how this looks in practice. You have a database package with a concrete store:

 1package db
 2
 3import "yourapp/user"
 4
 5type UserStore struct {
 6    pool *pgxpool.Pool
 7}
 8
 9func NewUserStore(pool *pgxpool.Pool) *UserStore {
10    return &UserStore{pool: pool}
11}
12
13func (s *UserStore) CreateUser(ctx context.Context, u *user.User) error {
14    // actual database implementation
15}
16
17func (s *UserStore) GetUserByID(ctx context.Context, userID string) (*user.User, error) {
18    // actual database implementation
19}
20
21// ... more methods

And a service package that defines what it needs:

 1package user
 2
 3type userStore interface {
 4    CreateUser(ctx context.Context, user *User) error
 5    GetUserByID(ctx context.Context, userID string) (*User, error)
 6    UpdateUser(ctx context.Context, user *User) error
 7    DeleteUser(ctx context.Context, userID string) error
 8}
 9
10type Service struct {
11    userStore userStore
12}
13
14func NewService(userStore userStore) *Service {
15    return &Service{userStore: userStore}
16}

In your main package, you wire them together:

1pool := connectToDatabase()
2userStore := db.NewUserStore(pool)
3userService := user.NewService(userStore)

The db.UserStore satisfies the user.userStore. The service is testable, the store is reusable, and the dependency flows in one direction.

To summarize

This pattern gives you three things that are hard to get any other way:

1. Your domain logic lives where it belongs - User is in the user package, not scattered across db or models
2. Testing becomes trivial - swap implementations without mocking frameworks or test databases
3. Dependencies flow one direction - high-level packages define interfaces, low-level packages implement them

It’s not about being dogmatic. Sometimes you need to return an interface - particularly when you’re building a library and need to hide implementation details. But as a default? Accept interfaces, return structs. Your future self will thank you.

Because, the same way some people pick up a hammer or a pencil, I open a text editor. It’s how I build things, fix what’s broken, or follow a half-formed idea just to see if it works.

However, if we rewind the tape a bit, where did it all start?

The Microsoft FrontPage era

We’d have to travel back to somewhere between 2000 and 2002. One could argue that HTML and CSS aren’t real programming - and I’d probably agree. But to ten-year-old-me, sitting in front of a big beige CRT monitor in the early 2000s, HTML was programming.

Back then, the internet - at least the part I was building - was mostly held together by tables. Not the database kind - the layout kind: <table>.

Technically <div> and <span> did exist, but they didn’t do anything. I’d drop one into FrontPage - nothing. No borders, no padding - just emptiness. Tables, on the other hand, actually made something appear on the screen.

What I ended up with was endless rows and columns, nested so deeply you’d think a recursive loop had written them. Somehow, though, it worked. I’d drag things around in FrontPage until everything looked sort of right.

Then came deployment:

1. Save my files.
2. Fire up an FTP client¹.
3. Upload my changes.
4. Send the link to my friends on MSN Messenger.

No build step, no deploy pipeline, no twelve layers of abstraction. Just HTML, an FTP client, and the thrill of seeing it work.

PHP: my first “real” programming language

A little after figuring out basic HTML, I stumbled onto PHP. Suddenly, I could create one file for my header and footer and include them on every page instead of copying and pasting. Mind blown. It also meant I could start building things like guestbooks and visitor counters.

My first database? Ever heard of .txt?

• Atomic? No.
• Did it work? Yes.

I stuck with PHP for many years whenever I built something for the web. I was there in the early days of WordPress, drifted away for a while, then came back around the time Laravel appeared - and used it as my main framework for a couple of years.

Sockets, Servers and Java

Around the same time, I started writing IRC bots - the kind that hung out in a channel, asked quiz questions, or replied with jokes. I remember using a framework called PircBot.

Somewhere during my IRC years, I also started running servers. I hosted Ventrilo and Counter-Strike servers for friends and eventually had a bot that could spin one up on command.

Later, it even accepted text messages - so friends (and friends of friends) could pay a few kronor via SMS to rent a CS server for a couple of hours.

Looking back, I guess that was my first taste of automation, even some entrepreneurship, though I didn’t think of it as such back then. To me, it was just another problem to solve.

Still the same feeling

Because I wasn’t trying to build a career or make a living out of it. I was just trying to make things work, to automate some small problem, or just to see if I actually could build it.

That same curiosity started bleeding into everything else.

• I built small websites that needed visitors, so I learned about SEO.
• I wanted to measure how people were using my websites, so I set up urchin.js².

So while programming was my way into tech, it also quietly opened the door to the world I’ve now worked in for the past fifteen years.

Over time, the projects changed, but the feeling has always stayed the same.

So of course I built my own blog engine

I could’ve used Astro, Hugo, or heck, even WordPress. But I like building things for the sake of building things.

So this site runs on a Go-based setup I wrote for myself. It takes Markdown content, turns it into HTML, bundles and minifies CSS and JS, fingerprints assets for caching, generates RSS feeds and sitemaps, and even spits out Cloudflare Pages-compatible _headers and _redirects files.

It’s not meant to be the simplest solution - it’s meant to be my solution. I want to understand every moving part, from how Markdown turns into a page to how cache headers get set. The end result is something that works exactly the way I want it to.

If it’s overkill for a personal site? Yes. And that’s exactly the point. No frameworks, no plugins - just the joy of wiring the pieces together.

Programming started as a way to tinker, and it ended up shaping most of what I’ve done since. Two decades later, not much has changed. The code looks different, the tools are fancier, but the feeling is the same - that moment when something I built appears on the screen.

That’s still the best part.