Stuart Grassie: TemporalCohesion

What I'm building: a football data platform for one

Mon, 03 Aug 2026 00:00:00 +0000

For the last while I’ve been building a football data platform. It collects match results and statistics, stores bookmaker odds, fits a statistical model to predict match outcomes, and then compares what the model thinks against what the market thinks. This post is the start of a long series about how it works.

The obvious first question is why.

Partly because football is the sport I actually watch, and I wanted better answers to questions like “is this team actually good or just lucky?”. Partly because the problem has a lovely shape for a side project: real-world data that’s messy in interesting ways, some genuinely fun maths, and a clear way to keep score. If your model says a result is 55% likely and the bookmaker says 45%, one of you is wrong, and the fixture list will eventually tell you which.

I also wanted a proper excuse to use .NET 10, PostgreSQL and Entity Framework Core on something with real data volume, rather than another todo app.

What it actually does

The pipeline, end to end:

Import historical match results and statistics.
Ingest bookmaker odds from The Odds API into a canonical form.
Fit a team strength model (a simplified Dixon-Coles, more on that in a later post) that turns past results into attack and defence ratings.
Convert those ratings into expected goals, then into probabilities for match outcomes: home/draw/away, over/under goals, both teams to score.
Compare the model’s probabilities against the bookmaker’s prices and flag where they disagree.
Evaluate everything ruthlessly: Brier scores, log loss, calibration curves, simulated staking with drawdown tracking.

One thing I want to state plainly at the outset, because it shapes a surprising amount of the design: this system never places a bet. There is no code path that stakes money, and there never will be. All the staking and bankroll machinery is simulation, there to answer the question “would this model have made money?” rather than to make any. That constraint turns out to be liberating. When nothing real is at stake you can be honest about your results, and honest evaluation is the whole point.

The stack

.NET 10 / C#, because it’s what I write all day and the newer language versions are pleasant.
PostgreSQL with Npgsql, because match data is relational to its bones and Postgres does everything I need, including materialized views for the heavier analytics.
EF Core 10 with a single DbContext covering multiple schemas. That decision has some interesting consequences I’ll cover in its own post.
NodaTime, because a fixture happens on a date and an odds observation happens at an instant, and DateTime is a bad model for both.
Blazor Server with Radzen components for the UI, which is called Jeff. Every project needs a Jeff.
TickerQ for background jobs, and a CLI for the batch work.

What this series will cover

Over the coming months I’ll write about the architecture (there’s a one-way dependency rule I’m quite pleased with), the betting maths from first principles (what an overround is, why raw bookmaker odds lie to you, Shin’s method for un-lying them), the Poisson goals model, and the evaluation machinery. There will be a lot of C#.

If you know .NET but nothing about betting markets, you’re the reader I have in mind. I knew nothing about them either when I started; most of this series is me writing up what I had to learn.

Next up: how the solution is laid out, and the dependency rule that keeps it honest.

Traefik in the homelab: one proxy for everything

Mon, 27 Jul 2026 00:00:00 +0000

Everything self-hosted on my network now sits behind a single Traefik instance. This post kicks off a series about how that happened, what the setup looks like, and the assorted yaks that got shaved along the way (there is a home-made certificate authority in this series; you have been warned).

The problem

The homelab grew the way homelabs do. One service becomes four, four becomes a dozen, and suddenly nothing is memorable: this thing is on port 8096, that one is on 5002, a third is on 3000 but only over plain HTTP. Bookmarks help until they don’t. What I wanted was names, chesskit.cybertron.lan rather than an IP and a port, and I wanted HTTPS everywhere, even though nothing here is reachable from the internet.

Why Traefik

The short version: everything I run is already in Docker, and Traefik’s Docker provider means a service declares its own routing in the compose file that deploys it. A handful of labels, restart the container, and the proxy knows about it. No central nginx config to edit, no reload step to forget.

The labels for a typical service look like this:

labels:
  - traefik.enable=true
  - traefik.http.routers.chesskit.rule=Host(`chesskit.cybertron.lan`)
  - traefik.http.routers.chesskit.entrypoints=websecure
  - traefik.http.routers.chesskit.tls=true
  - traefik.http.services.chesskit.loadbalancer.server.port=80

That is the whole integration. The service and the proxy never share a compose file; they share a Docker network and a naming convention.

The shape of the thing

The pieces, each of which gets its own post:

One Traefik container, with the Docker socket mounted read-only, watching for labelled containers.
An external Docker network called proxy that Traefik and every fronted service join.
Wildcard DNS so *.cybertron.lan resolves to the box Traefik runs on.
An internal certificate authority issuing a wildcard cert for *.cybertron.lan, because HTTPS on a LAN is worth having and Let’s Encrypt doesn’t hand out certificates for .lan.
A private registry on the NAS that built images get pushed to.

None of this is exotic. Most of it is the standard homelab Traefik arrangement you’ll find in a hundred blog posts, and this series will not pretend otherwise. What I’ll try to add is the reasoning, the mistakes, and the specific sharp edges I hit, because the hundred blog posts mostly skip those.

What this series is not

It is not a Kubernetes series. It is not a “zero trust homelab” series. It is one proxy, on one network, run by one person, and deliberately so. The whole point of the exercise was to make the lab nicer to live with, not to give myself a second job operating it.

Next post: What you need to actually understand Traefik: routers, services and entrypoints.

Using TimeWarp.Nuru as a command-line parser

Tue, 21 Jul 2026 00:00:00 +0000

The CLI is the oldest frontend in BlogHelper9000, and its command-line parsing is done by TimeWarp.Nuru, a route-based CLI framework that brings web-style routing and the mediator pattern to console apps. What I get out of it is that the CLI behaves like any other .NET application: commands are classes, logic lives in handlers, and dependencies arrive through Microsoft.Extensions.DependencyInjection. What I pay for it is riding a 3.0 beta with some genuinely sharp edges, and I’ll show both.

At the time of writing BlogHelper9000 is on TimeWarp.Nuru version 3.0.0-beta.71.

Setting up the application

The entry point looks like a minimal ASP.NET Core app, with one important difference I’ll get to:

var builder = NuruApp.CreateBuilder()
    .UseMicrosoftDependencyInjection()
    .AddConfiguration(args);

builder.ConfigureServices(services =>
{
    services.AddSingleton<IConfiguration>(_ => new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json", optional: true)
        .AddEnvironmentVariables()
        .Build());
    services.AddLogging(logging => logging.AddConsole());
    services.AddOptions<BlogHelperOptions>().BindConfiguration("BlogHelperOptions");
    services.AddSingleton<IFileSystem, FileSystem>();
    services.AddSingleton<PostManager>();
    // ... more service registrations
});

NuruApp app = builder
    .DiscoverEndpoints()
    .Build();

return await app.RunAsync(args);

DiscoverEndpoints() scans the assembly for command classes, and RunAsync(args) parses the arguments and dispatches to the right handler. Routing is source-generated, so the project file needs the Nuru interceptors namespace or none of it works:

<InterceptorsNamespaces>$(InterceptorsNamespaces);TimeWarp.Nuru.Generated</InterceptorsNamespaces>

The important difference is that ConfigureServices block. The obvious thing, registering services directly on builder.Services the way ASP.NET Core has trained everyone to, compiles cleanly and then throws InvalidOperationException at startup for every single command. The source generator only recognises registrations made inside ConfigureServices. It also inlines that lambda into a static method, so the lambda can’t capture anything from Program; that’s why the IConfiguration is rebuilt inside it from the same sources the host uses, and why logging is registered by hand. The generator registers neither for you.

Defining a command

Each command is a plain C# class that implements ICommand<TResult>. For commands that don’t return a value, TResult is Unit. Routing is declared with the [NuruRoute] attribute:

[NuruRoute("add", Description = "Add a new post")]
public sealed class AddCommand : ICommand<Unit>
{
    [Parameter(Description = "The title of the post to add.")]
    public string Title { get; set; }

    [Parameter(Description = "Comma-separated list of tags for the post.")]
    public string Tags { get; set; }

    [Option("is-draft", "d", Description = "Indicates whether the post is a draft.")]
    public bool IsDraft { get; set; }

    [Option("featured-image", "i", Description = "The path to the featured image for the post.")]
    public string FeaturedImage { get; set; }

    // ...
}

[Parameter] maps to positional arguments and [Option(longName, shortName)] maps to named flags, so this is bloghelper add "My new post" csharp,dotnet --is-draft.

The attribute is not optional, and this is the sharpest edge in the current beta. A command class without [NuruRoute] isn’t an error or a warning: DiscoverEndpoints() silently skips it, it disappears from --help, and no guessed route reaches it. Related, [NuruRouteGroup] is silently ignored by the generator in beta.71, which is why the schedule commands are flat hyphenated routes (schedule-list, schedule-import, schedule-stats) rather than proper schedule list subcommands. There’s a comment in the source saying exactly that, so future me knows the flat names are a workaround and not a design choice.

The handler pattern

The logic lives in a nested Handler class implementing ICommandHandler<TCommand, TResult>, with dependencies injected through the constructor exactly like any other service:

public sealed class Handler(ILogger<Handler> logger, IBlogService blogService, IScheduleService scheduleService)
    : ICommandHandler<AddCommand, Unit>
{
    public ValueTask<Unit> Handle(AddCommand request, CancellationToken cancellationToken)
    {
        var tags = string.IsNullOrWhiteSpace(request.Tags)
            ? null
            : request.Tags.Split(',', StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries);

        var filePath = blogService.AddPost(request.Title, request.IsDraft, request.IsFeatured,
            request.IsHidden, request.FeaturedImage, tags);

        if (filePath is null)
        {
            logger.LogError("Could not add post '{Title}' — a post already exists at the target path", request.Title);
            return default;
        }

        logger.LogInformation("Added new post at {File}", filePath);
        // ... add the post to a schedule series, if one was supplied
        return default;
    }
}

An asynchronous command just changes the signature to async ValueTask<Unit> and returns default at the end as before; the image handler awaits the Unsplash download and the ImageSharp processing that way.

Because commands are plain classes and handlers receive everything through the constructor, the tests don’t need any framework support: IFileSystem is swapped for MockFileSystem from System.IO.Abstractions.TestingHelpers, TimeProvider is pinned to a fixed time, and the remaining services are substituted with NSubstitute. The command and handler code doesn’t change at all to accommodate this.

Packaging as a .NET global tool

None of this interferes with packaging the CLI as a .NET tool. Three lines in the .csproj:

<PackAsTool>true</PackAsTool>
<ToolCommandName>bloghelper</ToolCommandName>
<PackageOutputPath>./releases</PackageOutputPath>

and installing locally from source is:

> dotnet tool install --global --add-source ./releases bloghelper

which makes bloghelper add "My new post" --is-draft available everywhere.

Would I recommend a beta command-line parser for a work project? No. For a personal tool, the trade is fine: the mediator pattern keeps each command self-contained and testable, the DI container is the same one the rest of the solution uses, and the sharp edges are the kind you hit once, write a comment about, and move on.

Next up: From Oakton to TimeWarp.Nuru, or how this CLI has now survived three command-line parsers.

One solution, seven projects: how BlogHelper9000 is structured

Mon, 13 Jul 2026 00:00:00 +0000

BlogHelper9000 began life as a single console project. It’s now seven non-test projects plus four test projects. One of those seven, BlogHelper9000.TestHelpers, only exists to support the tests, but it’s useful enough to count in its own right.

The number is less interesting than how it got there. I didn’t start with a diagram and divide the code into boxes. Each split followed a second consumer, an awkward dependency, or something that had become needlessly difficult to test. This post is the map for the rest of the series.

BlogHelper9000.sln
  BlogHelper9000.Core/       domain logic, YAML parsing, post management
  BlogHelper9000.Imaging/    ImageSharp, Unsplash, featured image generation
  BlogHelper9000/            the CLI (exe), references Core + Imaging
  BlogHelper9000.Nvim/       embedded Neovim client via MsgPack-RPC
  BlogHelper9000.Tui/        Terminal.Gui workspace (exe), references Core + Imaging + Nvim
  BlogHelper9000.Mcp/        MCP server (exe), references Core + Imaging
  BlogHelper9000.TestHelpers/ shared test infrastructure

Why each cut exists

Core holds the operations that make BlogHelper9000 more than a collection of frontends. PostManager knows the Jekyll directory layout, MarkdownHandler reads and rewrites post files, YamlConvert parses front matter, and BlogService composes them into operations such as publishing a post. The scheduling database also lives here, which is why Core now references Microsoft.Data.Sqlite as well as System.IO.Abstractions and a few Microsoft.Extensions packages. It has no UI dependencies.

Imaging keeps ImageSharp, Unsplash and the featured-image pipeline out of Core. ImageSharp requires a licence key to build this project, which is a good enough reason not to make it part of the library that parses every post. The CLI, TUI and MCP server all use Imaging now, but Core and Nvim don’t need to know it exists.

Nvim is the embedded Neovim client: process management, MsgPack-RPC and the screen grid. It’s separate from the TUI project for one reason: testability. BlogHelper9000.Nvim.Tests exercises RPC framing and grid logic without Terminal.Gui anywhere in sight, and without starting a real nvim process.

The three executable projects are the CLI, TUI and MCP server. They contain the interaction each frontend needs, but shared blog operations stay in Core. When I added the MCP server, most of the project was tool definitions and a Program.cs; the operations behind those tools already existed in Core and Imaging. The DI registrations are much the same in all three:

builder.Services.AddSingleton<IFileSystem, FileSystem>();
builder.Services.AddSingleton<MarkdownHandler>();
builder.Services.AddSingleton<PostManager>();
builder.Services.AddSingleton<TimeProvider>(TimeProvider.System);
builder.Services.AddSingleton<IBlogService, BlogService>();

That block appears, with minor variations, in all three entry points. I could extract it. I haven’t, because three copies of fifteen lines are cheaper than the wrong abstraction. The registrations already differ: the TUI needs NvimClient, while the MCP server does not.

Where the boundary actually is

The rule is not that frontends never touch the file system. The TUI’s file browser enumerates drafts and posts, its editor reads and writes the current document, and its copy, move and delete commands are implemented in the TUI. Moving those operations into BlogService would make Core aware of behaviour that only one frontend has.

The boundary is about meaning. Publishing a post, parsing its front matter and finding author branding are blog operations, so frontends ask PostManager or BlogService to do them. Selecting a file in a Terminal.Gui tree is TUI behaviour, so it stays there. Both sides use IFileSystem rather than static File and Directory calls, which lets the tests use MockFileSystem instead of a real disk.

That is a less tidy rule than “frontends never touch the file system”, but it matches the code. Not every file operation deserves a domain service. The ones that express how the blog works do.

What I’d tell 2022 me

Split Core out on day one. It happened around the time the metadata fixing got serious, and the refactor was tedious. Do not split anything else until a second consumer shows up or the existing boundary gets in the way of testing.

I would also name the projects properly from the start. Renaming a .NET project means updating the solution, its test project and any InternalsVisibleTo attributes that point at it. I’ve done enough of that for one tool.

Next up: using TimeWarp.Nuru as the command-line parser behind the CLI.

BlogHelper9000, four years on

Mon, 06 Jul 2026 00:00:00 +0000

In March 2022 I wrote a post called Automating jekyll with dotnet, laid out seven requirements for a little CLI tool, ran dotnet new console --name BlogHelper9000, and promised a series of follow-up posts.

You may have noticed the follow-up posts didn’t entirely materialise. The tool did, though, and it kept growing in the background while the blog sat quiet — the info command informs me the gap since the last post reached 609 days, which is a number I’d rather not dwell on. The best way I can think of to break that streak is to finally write the series I promised, about the tool that’s been quietly accumulating features the whole time.

The 2022 scorecard

The original requirements, and how they went:

Cross-platform — yes. It’s all .NET (now on .NET 10), developed on macOS, run on Linux in CI, and nothing in it cares which.
Add posts from the command line — yes, bloghelper add "My new post" has worked since the early days.
A blog status overview — yes. The info command reports post counts, drafts, and the aforementioned days-since-last-post shame counter.
Publish from the command line — yes, including the filename dance Jekyll requires.
Generated featured images with branding — yes, via ImageSharp and Unsplash. This one was the most fun.
Fix years of inconsistent metadata — done, and it’s the subject of its own post.
Notify Twitter of new posts — no. Twitter stopped being Twitter, the API stopped being free, and I stopped caring. The only requirement I’m formally abandoning.

Not a bad hit rate, if you ignore the four-year delivery window.

What it actually became

The 2022 plan was a console app. What exists now is a small ecosystem that I didn’t plan so much as it naturally grew:

A CLI (bloghelper), rewritten from Oakton onto TimeWarp.Nuru, installable as a dotnet tool.
A TUI workspace built on Terminal.Gui v2 — file browser, command palette, status bar — with a full Neovim instance embedded inside it as the editor, speaking MsgPack-RPC to a nvim --embed child process.
An MCP server, so an LLM can drive the same blog operations by talking to the exact domain services the CLI uses. The first draft of this post was created by it, which is either delightful or unsettling depending on your disposition.
A core library underneath all three, with the YAML front matter parser I wrote back in the 2022 posts still doing the actual work.

The interesting part, looking back, isn’t any single feature. It’s that the 2022 decision to keep domain logic separate from the console frontend — made at the time as a result of my experience — is the only reason the TUI and the MCP server were cheap to add. Each new frontend has been a thin adapter over the same PostManager and BlogService classes.

The plan, such as it is

Over the next year I’m going to write the whole thing up properly: the solution structure, the command-line framework migration, the YAML parsing, the testing approach, the image pipeline, the build, the MCP server, and a long stretch on the Neovim embedding, which turned out to be the deepest rabbit hole of the lot.

One or two posts a week. The tooling that generates the posts will be dogfooding itself the entire way, and if the days-since-last-post counter climbs back into triple digits, you have my permission to laugh at me.

build dotnet 8 in an offline build server

Mon, 04 Nov 2024 00:00:00 +0000

In a corporate environment, the build server you have access to, and might even be responsible for, is almost usually ‘offline’ - which is to say that for annoyingly sensible security reasons, it is not reachable by the outside internet, and it does not have access to the outside internet.

Building dotnet 8 applications in such an environment is fairly straightforward, but does mean that you need to run dotnet restore and dotnet build separately.

You must configure a package source which is reachable internally, either on a drive, network share or internal feed - as long as it is accessible via a URI.

The source can be configured in a Nuget.config, or can be passed by the --source option.

Typically, I’ve been running:

dotnet restore --nologo --source <the-source-uri>

As we have a single source where we keep all packages. If we had multiple source locations, I would use the Nuget.confg option and specify the sources in there.

The build step becomes:

dotnet build --no-restore --nologo --configuration Release

We make sure to disable the restore on the build, because we’ve already done it.

Nice and easy.

Simple YAML Parser Refactoring

Tue, 15 Nov 2022 00:00:00 +0000

After reviewing the code for the simple YAML parser I wrote, I decided it was getting a little messy, so before continuing, I decided to refactor it a little bit.

The simples thing to do was to separate the serialisation and the deserialisation into separate classes, and simple call those from within the YamlConvert class in the existing methods. This approach tends to be what other JSON and YAML libraries do, with added functionality such as being able to control aspects of the serialisation/deserialisation process for specific types.

I currently don’t need, or want, to do that, as I’m taking a much more brute force approach - however it is something to consider for a future refactor. Maybe.

I ended up with the following for the YamlConvert:

public static class YamlConvert
{
    private static YamlSerialiser Serialiser;
    private static YamlDeserialiser Deserialiser;
    
    static YamlConvert()
    {
        Serialiser = new YamlSerialiser();
        Deserialiser = new YamlDeserialiser();
    }
    
    public static string Serialise(YamlHeader header)
    {
        return Serialiser.Serialise(header);
    }

    public static YamlHeader Deserialise(string filePath)
    {
        if (!File.Exists(filePath)) throw new FileNotFoundException("Unable to find specified file", filePath);

        var content = File.ReadAllLines(filePath);

        return Deserialise(content);
    }

    public static YamlHeader Deserialise(string[] rawHeader)
    {
        return Deserialiser.Deserialise(rawHeader);
    }
}

It works quite well, as it did before, and looks a lot better. There is no dependency configuration to worry about, as I mentioned above I’m not worried about swapping out the serialisation/deserialisation process at any time.

Simple YAML parser - parsing the header

Sat, 30 Jul 2022 00:00:00 +0000

Previously we left off with a method which could parse the YAML header in one of our markdown files, and it was collecting each line between the --- header marker, for further processing.

One of the main requirements for the overall BlogHelper9000 utility is to be able to standardise the YAML headers in each source markdown file for a post. Some of the posts had a mix of different tags, that were essentially doing the same thing, so one of the aims is to be able to collect those, and transform the values into the correct tags.

In order to achieve this, we can specify a collection of the valid header properties up front, and also a collection of the ‘other’ properties that we find, which we can hold for further in the process when we’ve written the code to handle those properties. The YamlHeader class has already been defined, and we can use a little reflection to load that class up and pick the properties out.

private static Dictionary<string, object?> GetYamlHeaderProperties(YamlHeader? header = null)
{
    var yamlHeader = header ?? new YamlHeader();
    return yamlHeader.GetType()
        .GetProperties(BindingFlags.DeclaredOnly | BindingFlags.Public | BindingFlags.Instance)
        .Where(p => p.GetCustomAttribute<YamlIgnoreAttribute>() is null)
        .ToDictionary(p =>
        {
            var attr = p.GetCustomAttribute<YamlNameAttribute>();

            return attr is not null ? attr.Name.ToLower() : p.Name.ToLower();
        }, p => p.GetValue(yamlHeader, null));
}

We need to be careful to ignore collecting properties that are not part of the YAML header in markdown files, but that we use in the YamlHeader that we can use when doing further processing - such as holding the ‘extra’ properties that we’ll need to match up with their valid counterparts in a further step. Thus we have the custom YamlIgnoreAttribute that we can use to ensure we drop properties that we don’t care about. We also need to ensure that we can match up C# property names with the actual YAML header name, so we also have the YamlNameAttribute to handle this.

Then we just need a way of parsing the individual lines and pulling the header name and the value out.

(string property, string value) ParseHeaderTag(string tag)
{
    tag = tag.Trim();
    var index = tag.IndexOf(':');
    var property = tag.Substring(0, index);
    var value = tag.Substring(index+1).Trim();
    return (property, value);
}

Here we just return a simple tuple after doing some simple substring manipulation, which is greatly helped by the header and its value always being seperated by ‘:’.

Then if we put all that together we can start to parse the header properties.

private static YamlHeader ParseYamlHeader(IEnumerable<string> yamlHeader)
{
    var parsedHeaderProperties = new Dictionary<string, object>();
    var extraHeaderProperties = new Dictionary<string, string>();
    var headerProperties = GetYamlHeaderProperties();

    foreach (var line in yamlHeader)
    {
        var propertyValue = ParseHeaderTag(line);

        if (headerProperties.ContainsKey(propertyValue.property))
        {
            parsedHeaderProperties.Add(propertyValue.property, propertyValue.value);
        }
        else
        {
            extraHeaderProperties.Add(propertyValue.property, propertyValue.value);
        }
    }

    return ToYamlHeader(parsedHeaderProperties, extraHeaderProperties);

All we need to do is, to setup up some dictionaries to hold the header properties, get the dictionary of valid header properties, and then loop through each line, parsing the header tag and verifying whether the property is a ‘valid’ one that we definitely know we want to keep, and or one we need to hold for further processing. You’ll noticed in the above code, that it’s missing an end brace: this is deliberate, because the ParseHeaderTag method and ToYamlHeader method are both nested methods.

Reading through the code to write this post has made me realise that we can do some refactoring to make this look a little nicer.

So we’ll look at that next.

Writing a simple YAML parser

Fri, 22 Jul 2022 00:00:00 +0000

The next thing to do to get BlogHelper9000 functional is to write a command which provides some information about the posts in the blog. I want to know:

How many published posts there are
How many drafts there are
A short list of recent posts
How long it’s been since a post was published

I also know that I want to introduce a command which will allow me to fix the metadata in the posts, which is a little messy. I’ve been inconsistently blogging since 2007, originally starting off on a self-hosted python blog I’ve forgot the name of before migrating to Wordpress, and then migrating to a short lived .net static site generator before switching over to Jekyll.

Obviously, Markdown powered blogs like Jekyll have to provide non-markdown metadata in each post, and for Jekyll (and most markdown powered blogs) that means: YAML.

Parse that YAML

There are a couple of options when it comes to parsing YAML. One would be to use YamlDotNet which is a stable library which conforms with V1.1 and v1.2 of the YAML specifications.

But where is the fun in that?

I’ve defined a POCO called YamlHeader which I’m going to use to use as the in-memory object to represent the YAML metadata header at the top of a markdown file.

If we take a leaf from different JSON converters, we can define a YamlConvert class like this:

public static class YamlConvert
{
    public static string Serialise(YamlHeader header)
    {
    }

    public static YamlHeader Deserialise(string filePath)
    {
    }
}

With this, we can easily serialise a YamlHeader into a string, and deserialise a file into a YamlHeader.

Deserialise

Deserialising is the slight more complicated of the two, so lets start with that.

Our first unit test looks like this:

    [Fact]
    public void Should_Deserialise_YamlHeader()
    {
        var yaml = @"---
layout: post
title: 'Dynamic port assignment in Octopus Deploy'
tags: ['build tools', 'octopus deploy']
featured_image: /assets/images/posts/2020/artem-sapegin-b18TRXc8UPQ-unsplash.jpg
featured: false
hidden: false
---
post content that's not parsed";
        
        var yamlObject = YamlConvert.Deserialise(yaml.Split(Environment.NewLine));

        yamlObject.Layout.Should().Be("post");
        yamlObject.Tags.Should().NotBeEmpty();
    }

This immediately requires us to add an overload for Deserialise to the YamlConvert class, which takes a string[]. This means our implementation for the first Deserialise method is simply:

public static YamlHeader Deserialise(string filePath)
{
    if (!File.Exists(filePath)) throw new FileNotFoundException("Unable to find specified file", filePath);

    var content = File.ReadAllLines(filePath);

    return Deserialise(content);
}

Now we get into the fun part. And a big caveat: I’m not sure if this is the best way of doing this, but it works for me and that’s all I care about.

Anyway. A YAML header block is identified by a single line of only --- followd by n lines of YAML which is signified to have ended by another single line of only ---. You can see this in the unit test above.

The algorithm I came up with goes like this:

For each line in lines:
  if line is '---' then
    if header start marker not found then
      header start marker found
      continue
     break loop
    store line
  parse each line of found header

So in a nutshell, it loops through each line in the file, look for the first --- to identify the start of the header, and then until it hits another ---, it gathers the lines for further processing.

Translated into C#, the code looks like this:

public static YamlHeader Deserialise(string[] fileContent)
{
    var headerStartMarkerFound = false;
    var yamlBlock = new List<string>();

    foreach (var line in fileContent)
    {
        if (line.Trim() == "---")
        {
            if (!headerStartMarkerFound)
            {
                headerStartMarkerFound = true;
                continue;
            }

            break;
        }

        yamlBlock.Add(line);
    }
        
    return ParseYamlHeader(yamlBlock);
}

This is fairly straightforward, and isn’t where I think some of the problems with the way it works actually are - all that is hidden behind ParseYamlHeader, and is worth a post on its own.

Introduction to command line parsing with Oakton

Thu, 14 Jul 2022 00:00:00 +0000

In the introductory post to this series, I ended with issuing a command to initialise a new console project, BlogHelper9000. It doesn’t matter how you create your project, be it from Visual Studio, Rider or the terminal, the end result is the same, as the templates are all the same.

With the new .net 6 templates, the resulting Program.cs is somewhat sparse, if you discount the single comment then all you get in the file is a comment and a Console.WriteLine("Hello, World!");, thanks to all the new wizardry in the latest versions of the language and the framework.

Thanks to this new fangled sorcery, the app still has a static main method, you just don’t need to see it, and as such, the args string array is still there. For very simple applications, this is all you really need to do. However, once you get past a few commands, with a few optional flags, things can get complicated, fast. This can into a maintenance headache.

In the past I’ve written my own command line parsing abstractions, I’ve used Mono.Options and other libraries, and I think I’ve finally settled on Oakton as my go to library for quickly and easily adding command line parsing to a console application. It’s intuitive, easy to use and easy to maintain. This means you can easily introduce it into a team environment and have everyone understand it immediately.

Setup Command loading

After following Oakton’s getting started documentation, you can see how easy it is to get going with a basic implementation. I recommended introducing the ability to have both synchronous and asynchronous commands able to be executed, and you achieve this by a small tweak to the Program.cs and taking into consideration the top-level statements in .net 6, like this:

using System.Reflection;

var executor = CommandExecutor.For(_ =>{
    _.RegisterCommands(typeof(Program).GetTypeInfo().Assembly);
});

var result = await executor.ExecuteAsync(args);
return result;

In .net 5, or if you don’t like top-level statements and have a static int Main you can make it static Task<int> Main instead and return the executor.ExecuteAsync instead of awaiting it.

Base classes

In some console applications, different commands can have the same optional flags, and I like to put mine in a class called BaseInput. Because I know I’m going to have several commands in this application, I’m going to add some base classes so that the different commands can share some of the same functionality. I’ve also used this in the past to, for example, create a database instance in the base class, which is then passed into each inheriting command. It’s also a good place to add some common argument/flag validation.

What I like to do is have an abstract base class, which inherits from the Oakton command, and add an abstract Run method to it, and usually a virtual bool ValidateInput too; these can then be overriden in our actual Command implementations and have a lot of nice functionality automated for us in a way that can be used across all Commands.

Some of the detail of these classes are elided, to stop this from being a super long post, you can see all the details in the Github repo.

public abstract class BaseCommand<TInput> : OaktonCommand<TInput>
    where TInput : BaseInput
{
    public override bool Execute(TInput input)
    {
        return ValidateInput(input) && Run(input);
    }

    protected abstract bool Run(TInput input);

    protected virtual bool ValidateInput(TInput input)
    {
        /* ... */
    }
}

This ensures that all the Commands we implement can optionally decide to validate the inputs that they take in, simply by overriding ValidateInput.

The async version is exactly the same… except async:

public abstract class AsyncBaseCommand<TInput> : OaktonAsyncCommand<TInput>
    where TInput : BaseInput
{
    public override Task<bool> Execute(TInput input)
    {
        return ValidateInput(input) && Run(input);
    }

    protected abstract Task<bool> Run(TInput input);

    protected virtual Task<bool> ValidateInput(TInput input)
    {
        /* ... */
    }
}

There is an additional class I’ve not yet shown, which adds some further reusable functionality between each base class, and that’s the BaseHelper class. I’ve got a pretty good idea that any commands I write for the app are going to operate on posts or post drafts, which in jekyll are stored in _posts and _drafts respectively. Consequently, the commands need an easy way of having these paths to hand, so a little internal helper class is a good place to put this shared logic.

internal class BaseHelper<TInput> where TInput : BaseInput
{
    public string DraftsPath { get; }

    public string PostsPath { get;  }

    private BaseHelper(TInput input)
    {
        DraftsPath = Path.Combine(input.BaseDirectoryFlag, "_drafts");
        PostsPath = Path.Combine(input.BaseDirectoryFlag, "_posts");
    }

    public static BaseHelper<TInput> Initialise(TInput input)
    {
        return new BaseHelper<TInput>(input);
    }

    public bool ValidateInput(TInput input)
    {
        if (!Directory.Exists(DraftsPath))
        {
            ConsoleWriter.Write(ConsoleColor.Red, "Unable to find blog _drafts folder");
            return false;
        }

        if (!Directory.Exists(PostsPath))
        {
            ConsoleWriter.Write(ConsoleColor.Red, "Unable to find blog _posts folder");
            return false;
        }

        return true;
    }
}

This means that our base class implementations can now become:

private BaseHelper<TInput> _baseHelper = null!;
protected string DraftsPath => _baseHelper.DraftsPath;
protected string PostsPath => _baseHelper.PostsPath;

public override bool Execute(TInput input)
{
    _baseHelper = BaseHelper<TInput>.Initialise(input);
    return ValidateInput(input) && Run(input);
}

protected virtual bool ValidateInput(TInput input)
{
    return _baseHelper.ValidateInput(input);
}

Note the null!, where I am telling the compiler to ignore the fact that _baseHelper is being initialised to null, as I know better.

This allows each command implementation to hook into this method and validate itself automatically.

First Command

Now that we have some base classes to work with, we can start to write our first command. If you check the history in the repo, you’ll see this wasn’t the first command I actually wrote… but it probably should have been. In any case, it only serves to illustrate our first real command implementation.

public class InfoCommand : BaseCommand<BaseInput>
{
    public InfoCommand()
    {
        Usage("Info");
    }

    protected override bool Run(BaseInput input)
    {
        var posts = LoadsPosts();
        var blogDetails = new Details();

        DeterminePostCount(posts, blogDetails);
        DetermineDraftsInfo(posts, blogDetails);
        DetermineRecentPosts(posts, blogDetails);
        DetermineDaysSinceLastPost(blogDetails);

        RenderDetails(blogDetails);

        return true;
    }

    /**...*/
}

LoadPosts is a method in the base class which is responsible for loading the posts into memory, so that we can process them and extract meaningful details about the posts. We put store this information in a Details class, which is what we ultimately use to render the details to the console. You can see the details of these methods in the github repository, however they all boil down to simple Linq queries.

Summary

In this post we’ve seen how to setup Oakton and configure a base class to extend the functionality and give us more flexibility, and an initial command. In subsequent posts, we’ll cover more commands and I’ll start to use the utility to tidy up metadata across all the posts in the blog and fix things like images for posts.