csMACnz's Blog

Vibe Coding up a Vibe Blogging Machine

Tue, 07 Apr 2026 01:11:00 +0000

Introducing VibeBlog, my playground for exploring GitHub’s agentic workflows (that’s the whole AI-powered automation thing everyone’s talking about). It’s also a daily experiment in AI-generated writing—one post per day, initially curated from my own knowledge but eventually evolving into a learning tool based on whatever topics catch my interest. The goal? Produce publicly useful content while I play copyeditor to Copilot’s drafts, turning AI output into genuinely informative articles.

Seriously, as a learning exercise in various Github Copilot features, it has been great fun to build out a daily generated blog site with minimal human-in-the-loop vibes.

What follows is a bit of a diary of the development and how I got here.

The Setup

I didn’t start by “writing posts.” I started by building a machine that could publish them reliably.

When creating a repository in GitHub, you can start with an initial prompt, So I did.

A static site generated blog. Articles will be written and stored as markdown and rendered as a static site. Uses GitHub Actions to build and publish. Agwntic instruction files will support copilot agent tasks to write the posts.

C# is the preferred language for any code used in this repository

Yeah, I went with what I know, C#. I could have left it up to the AI but I did want to at least feel confident that what it produced was valid, safe and maintainable code.

While I did review what it created, I left this as the first prototype and commited without change. As I went on, reviews often fell into more of a YOLO camp, but I did try to stick to a couple of guardrails:

Read everything before it gets merged
Don’t produce any commits directly

(I only had a couple of late-stage exceptions to that second rule, due to trying to save a few premium session tokens on one-lineer changes.)

This PR was complete enough that will a few extra clicks to enable GitHub Pages, I had a live blog with a working generator and deploy pipeline within minutes of creating the repo. From there, I just iterated on the codebase and the agentic workflow, with the goal of making it more robust and more automated over time.

From this point on, every PR started as an agentic session prompt, and proceeded as a Pull Request collaboration workflow.

I went straight into a CICD mindset and added automated tests, especially leveraging playwright, and some GitHub Actions for doing more robust build and deploy. The first prompt out of the gate did manage to provide a build and deploy which was great.

It looks like we gave no tests. We should add some xunit and playwright .net tests for both the generator app and the generated site. These tests should be part of the github actions, and verified against pull requests too.

Fortunately for me, I was able to give a very vague prompt for CI and it ran with it, producing more and better that I probably would have even thought to ask for.

Review the github actions for build and test and offer improvements

I went through a short manual testing and feedback phase at this point, where I identified some bugs. Since I was already using a custom domain for GitHub Pages, it did require a little bit of change to make the expected relative paths work, but that was a quick fix. From this, I layered on some extra integrity checking of the site as well.

I managed to even get it to scrape my styles from the existing blog for consistency. This was a lesson on the Github copilot Firewall configuration. It wasn’t perfect, but it got the job done with a bit of iteration.

Updating site design to align with parent blog

The style of the site needs updated. Id like to follow the style of the existing parent site blog.csmac.nz Ensure the pull request has a screenshot of the design attached as a comment. Please make sure you also download background images referenced in the css for reuse in this site as well

As I went, I tried to add additional improvement to both avoid regression, and add automation to catch the same or similar issues from resurfacing. Each time it did something wrong, I would correct it, but ask it to “remember” essentially.

Please fix this articles title length.

Please also update the agent prompts to always enforce this, even if the title provided is too long, enforce a shorter alternative title to the suggested one. This restriction is higher priority than keeping the suggested title.

Becoming a publishing platform

The foundations were set, now the writing could begin.

This went through a couple of refining phases:

Updating Copilot agent instructions toward the actual writing style

Take a look at my site https://github.com/csMACnzBlog/csMACnzBlog.github.io and improve the writing style guidance to match my writing style in github copilot markdown instructions

Enhance blog-post-writer style guide with patterns from csMACnzBlog

I would like a review of the writing style from the articles on blog.csmac.nz compared with the writing style used in this VibeBlog so far. I would like to enhance the style guides in this repository with any missing refinements or details for producing future posts.

This improved the voice of the articles. These two prompts together did a great job of refining the style and tone of the writing, and I was pretty happy with the results. It was able to pick up on some of the nuances of my writing style that I hadn’t explicitly called out in the instructions, which was great. The final blog-post-writer.md agent is now pretty detailed and specific, and I think it will do a good job of maintaining a consistent voice across future posts.

Robustness phase

I followed up by adding some more robustness to the publishing process, with a focus on making sure that the quality gates were in place and that the agent instructions were well-defined and versioned. This included:

Featured image support: PR #20 (merged 2026-02-27)
https://github.com/csMACnzBlog/Vibeblogging/pull/20
Adding baseline validation steps to that setup flow: PR #102 (merged 2026-03-28)
https://github.com/csMACnzBlog/Vibeblogging/pull/102
Pagination (15 posts/page): PR #62 (merged 2026-03-18)
https://github.com/csMACnzBlog/Vibeblogging/pull/62
Search page via lazy-loaded modal iframe: PR #64 (merged 2026-03-19)
https://github.com/csMACnzBlog/Vibeblogging/pull/64
Search UX fixes + tag focus/order: PR #66 (merged 2026-03-19)
https://github.com/csMACnzBlog/Vibeblogging/pull/66
Clicking a tag opens search pre-filtered: PR #103 (merged 2026-03-29)
https://github.com/csMACnzBlog/Vibeblogging/pull/103

Image Generation

This took a few iterations to get working but I was able to leverage some python and the Hugging Face API to get some blog images being generated for me as part of the build process. This was a fun one to iterate on, and I had to learn a bit about how to use the Hugging Face API and how to handle credentials in GitHub Actions. I tried out the familiarity of Powershell to start with but in the end went with what the models know best - Python. I also had to iterate on the prompt a bit to get it to generate images that were actually relevant and looked good. I ended up with a pretty good prompt that I can reuse for future image generation tasks as well. Maybe as a reduction exercise I’ll see if I can migrate what I have back to Powershell, but for now I’m happy with the Python solution.

Image generation pipeline moved from PowerShell → Python + HuggingFace client: PR #21 (merged 2026-02-27)
https://github.com/csMACnzBlog/Vibeblogging/pull/21

It did take a bunch of trial-and-error to get the credentials and API calls working correctly, but I got there in the end. The images are now being generated and uploaded to the site as part of the PR generation lifecycle process, which is pretty cool. Most of my drama came from fighting different environment sandboxes and differences between Copilot, and Workflow.

I even did a bit of experimenting with the new https://github.github.com/gh-aw/. It is pretty cool, but wasn’t the right tool for this particular job, since the repository sits public in the open. But I have some ideas for how I might be able to leverage it in the future for some of my more private repositories, so it was worth playing around with it a bit.

Finishing the automation

I had a pretty good workflow going at this point, but I wanted to get to the point where I could just merge a PR and have it go live without any manual steps. This involved making sure that all the quality gates were in place and that the agent instructions were well-defined.

I was still triggering and reviewing the daily post generation manually, but I wanted to get to the point where it was fully automated. So I tried a few different strategies and landed on the following approach:

In another repository, I trigger the creation of a new Issue into the repository, and assign it to Copilot to work on
Copilot then creates a new branch, and starts working on the post in that branch, using the agent instructions and style guide to produce the content
Once the post is ready, it creates a PR from that branch into main, and assigns it to me for review
Another automation checks over the PR for quality gates (links, accessibility, etc) and if it passes, it merges the PR automatically, which triggers the build and deploy process to publish the new post

I read every post daily. So far I haven’t really had to make any changes to the articles, and find them both accurate and interesting to read.

I even followed up with an automation that can investigate a topic suggestion and raise a more detailed Issue. If one of these is present, it uses that to assign to copilot instead of the generic “write a post about X” issue. This allows me to have a bit more control over the topic and the direction of the post, while still leveraging the automation to do the heavy lifting of writing and publishing.

A working system

By early April 2026, the engineering arc looks like this:

A C# generator that deploys cleanly to Pages (PR #1)
CI + testing from day one (PR #2, PR #4)
Deployment correctness handled early (subpath link fix) (PR #9)
Publishing quality gates baked in (PR #10, PR #105)
Agent instructions + setup treated as versioned system components (PR #5, PR #6, PR #24, PR #102)
Media pipeline with real-world credential iteration (PR #20, PR #21, PR #88, PR #92)
Scale features (pagination + search) (PR #62, PR #64, PR #66, PR #103)
A daily agentic publishing workflow that’s been battle-tested through break/fix cycles (PR #68, PR #79–#86, PR #77/#87)

Towards this later couple of weeks I’ve been focusing more on the main site and blog, trying to tie this new part into the larger site more seamlessly. This has identified many opportunities to apply similar learnings back to the way I opperate in the rest of the site’s development.

What’s Next?

Well I’m pretty well set up for the moment. At some point topic choice might get interesting, and I don’t know what the LLM is going to do. I do have my mechanism to generate topics and topic notes, so perhaps that feeds it for a while. It is one I am very interested to watch.

Otherwise, I just need to keep an eye on the daily workflow and make sure it doesn’t break. I have a good sense of what the failure modes are, so hopefully I can catch and fix them quickly if they come up. And since it runs on GitHub workflows, I get notifications when it fails, so that should help me stay on top of it.

Security is always a concern, but since this is a public repo with no sensitive data, the main risk is someone trying to break the workflow or inject bad content. I’ll need to monitor for that and be ready to revert if necessary. Hopefully I’ve locked down enough of the accesses that it is resilient to obvious attack vectors.

Appendix

Some of the PRs are referenced directly for your interest and further reading.

.NET Conf 2023 Review

Fri, 01 Dec 2023 11:18:13 +1300

This year I watched a bunch of the sessions from .NET Conf, both live streams and in the days following. I’ve collated my top recommendations and my reviews and thoughts on others that may be of interest to some Developers in the Microsoft ecosystem.

Without any further ado, we’ll crack into the must-watch list for 2023.

Must Watch

Why not kick into it with the main keynote? This talk gives a great overview of all the great new advancements released as part of .Net 8 so you can learn more about what you want to dive deeper into.

Follow this up with a deeper dive into all the enhanced cloud-native capabilities delivered to us this year as well, including your first look at the new .Net Aspire stack.

Pivoting to C# 12 which drops with .Net 8, check out the newest features to learn what’s been added to the language.

And if you are still on .Net 6 or a few versions behind on what has been developing, here is a wider look across the last few releases to catch up on what you’ve missed.

But to round that off with the question “What should I do with all this new info?” I recommend Bill Wagner’s pragmatic view of Everyday C# to guide you in how to pick and choose the best bits for you.

A focus on ASP.Net Core

We now switch focus to the API hosting application space with ASP.Net Core. This part is probably not relevant to all Developers so I’ve kept it out of must-watch. However, if you are a web app or MVC Developer then these are probably still on your “must-watch” list according to me.

If you want a really good high-level overview, look no further than ASP.NET Basics for experts with Layla - covering minimal APIs, open telemetry and the new Aspire tooling.

For a more targeted look at Open Telemetry, I highly recommend this video, especially if you haven’t started your journey in this space as it has become easier to enable than ever.

Similarly, whether you haven’t looked at Polly, or are already on the Polly train, this resiliency talk is a must-see.

Dive deeper into specifics

Here are some great deep dives into areas you may have some familiarity with but want to get a deep look into.

Starting with a configuration Deep dive, to catch you up on the power of the IConfiguration patterns built into .Net.

And if you are working with containers at all you’ll watch to check out the latest advancements here

Architecture

There were some great architecture talks, especially for organising your code in large systems.

Starting with a great Vertical Slice Architecture talk that opens with a brilliant overview of spaghetti, layered, and clean architecture to set the scene first.

(Here is an earlier presentation to help fill the gaps where the live demo gods failed us on the day - Pick a Side - Clean vs Vertical Slice Architecture | Luke Parker)

And if you are still sold on Clean Architecture after those videos, you will want to check out this great talk from Steve Smith.

Blazor

Blazor is going to be a small subset of Developers still, so whether you are new to it and want to learn more, or use it and want to see what is new, this video will satisfy you.

Niche pro-user tools

For those already more familiar with most of what .Net has to offer, there were some great deep-dives into some technologies that won’t be for every day, but have some very specific use cases you may cross paths with.

Starting with the new source generators:

And a great reverse-proxy solution called YARP (Yet Another Reverse Proxy)

And for the Swagger/OpenAPI users, we get an introduction to Kiota - an opinionated cross-platform client code generator for open API specifications.

If you want to build and distribute dotnet new templates of your own, take a look at Making templates with Rob Conery:

And more

This is by no means the total list of all the sessions. If you haven’t seen what you were looking for here, there is a Session Finder to search for more videos to watch and learn.

The conference website as always is found at dotnetconf.net

See you all for the next one in 2024! Happy viewing.

My idea of an ideal CICD pipeline

Tue, 01 Aug 2023 05:00:00 +0000

This article represents my current thoughts around building and deploying software (circa 2023). It is a general straw man. As usual in software development, the caveats apply that there is no one size fits all silver bullet pattern. This is where I start as a “given no further constraints” approach. It is highly likely to be too complex for a single-dev piece of software and is equally as likely not to be suitable for large monolithic applications either. However, for software somewhere in between those two it may fit your needs as it does mine.

Why CICD?

My definition of CICD is continuous static analysis, build, test, publish, deploy, verification and releasing to customers, for every change. While CI stands for Continuous Integration, CD ambiguously stands for either Continuous Delivery or Continuous Deployment, where Delivery and Deployment are equally ambiguous and have debatable definitions. So, while I will use the acronym here, see my definition above for what that means to me.

The crux of CICD is: always shipping, always automatically.

This unintuitively implies having automated gates that can stop and halt these deployments, and that somewhere right at the beginning the automation is kicked off by an initial interaction or contribution (such as a file change in a code repository). Also, do have a rollback plan handy.

The benefits we are trying to achieve are faster feedback loops, and changing mindset to help find and fix issues sooner, much closer to where the associated change was thought through and implemented. This is often referred to as Shift Left. In other cases, it allows us to reduce the time between a production issue being discovered, and the last time that component was being worked on by a human (which could be the difference between months in between, and hours/minutes in between thanks to CICD).

A brief digression into testing

Testing could be its own series of blog posts, so we will gloss over it slightly here. But it is necessary to say that automated testing is a large part of doing CICD well. And the ability to perform enough manual testing when required needs to be accounted for. This is how we achieve our safety and trust to enable straight-shot pipelines all the way to being in front of customers. We can look at a high-level view of these briefly.

Automated Testing

Automated testing should cover as much of the application as possible. This can range across unit tests, property-based tests, composition tests, mutation tests, boundary tests, snapshot tests, integration tests, acceptance tests, specification tests, regression tests, end-to-end tests, UI tests, smoke tests and any other kinds of tests you can think of to have run automated before code can go to production.

A new project might aim for a minimal subset of the above (such as unit, integration, and smoke) while a more established piece of software may have most or all of these in place.

The main goal in terms of CICD is to ensure most of your testing runs before merging in changes to a release candidate, and enough validation testing is run during and after deployment to ensure fidelity.

Manual Testing

Manual testing requires enough real dependencies to be meaningful, or else you may as well just automate any task that you are testing using fakes or mocks. Largely this is exploratory testing or deep testing of something complex requiring a human eye for validation.

The two tools I would lean on here are having a feature flagging and dark release pattern in your development process and having a secondary manual test environment to deploy pre-release changes to.

Feature flagging allows you to deploy to preproduction and production with inert functionality (sometimes called a shadow release). You can then enable this functionality either isolated to a canary server instance, per environment (e.g. preproduction only) or per user in a way that enables testers or beta users to interact with the flagged changes before a wider release.

Having an additional pre-production environment you can deploy changes to will enable Pull Requests or feature branches to be deployed there and manually tested before they are merged into a production candidate release.

Reliability Engineering

One aspect that makes this all come together and work is the developer thinking through production, and how they will know and trust it is working. Ongoing monitoring and alerting in place is a key part of this strategy. Without monitoring and alerting detecting and escalating issues you are effectively in chaos.

Automated tests catch the known failure scenarios, making sure they do not occur, or regress. But there are unknown failure scenarios, as well. We can detect these using signals such as error log count, 400 or 500 HTTP status codes, or request/response time spikes. These don’t tell us what the problem is but are great at signalling we do have a problem.

The feedback loop of monitoring tools alerting the developer who wrote the code (or their on-call proxy who can reach out to them) is what gives the confidence to release changes straight to production, complementing the other testing guards in place that catch the known issues before they reach customers so we can also catch these unknown issues after they reach customers.

I’ve left automated rollback out of this pipeline currently, as it adds additional challenges in understanding what is safe and when it isn’t. However, this is the next step you would consider as you delved into the CICD space. Having a manual plan, though, is essential. (I often use roll-forward to roll back. Either with a re-deploy of an older version, or with a patch+merge+release - often both together.)

A side note on the shape of software

What is it that we are testing and deploying? My philosophy is to include everything that makes up this component. By which I mean the application, any infrastructure (using configuration-as-code) and any side-car utilities (such as queues, caches, stores etc) not shared with other components.

It follows that all our static verification and testing also covers these infra and sidecar deployments as well.

If you do have queues, stores, or caches that are shared, you will want to extract these as their own components and deployments separately. But I’ve found where these are coupled and self-contained to a particular application, then having them co-located in a repository and deployed in the same pipeline is a simplification worth having.

Also note that your configuration-as-code for anything in the permanent storage capacity has appropriate backups, as well as the right checks and balances to restrict the pipeline from doing anything destructive automatically.

If feasible, you might also leverage configuration as code for your monitoring and alerting tools in the same way through this pipeline.

The pipeline

CICD is a pipeline, and these are the key phases and steps in that pipeline in my ideal model:

We will break this into a few chunks for a deeper discussion.

Pre merge

Ideally, I want trunk (e.g. main branch) to always be production capable. Achieving this means ensuring the quality and fidelity of any changes before they are merged in. This may look like a Pull Request with automated pre-build checks (e.g. GitHub Actions) and code reviews. It may also look like git commit hooks that are run and confirmed ahead of a merge. Exactly what you implement here is dependent on the software in question, and the engineers working on it.

For me, this encompasses linting and other static analysis, security scanning, compiling, and executing all tests, and producing candidate artifacts that can also be verified and validated here too.

Ideally, for the fasted feedback, you want this to take as little time as possible. If this takes more than 10 minutes (ideally 5 or less) you will want to consider reducing scope, performance tuning, or potentially splitting something monolithic up somehow.

Trunk Build

At all times, you want to be able to validate trunk against the same lint, build and test as you would for any candidate commit. Ensuring you can run the same checks in both places maintains the validity of your trunk.

In addition, we would also take the generated artifacts and make them available to the further deployment processes coming up next in the pipeline. So our trunk build also publishes artifacts somewhere.

Preproduction deploy and verify

It is important to have an environment that is production-like but isn’t serving customers to first test your artifacts in. Despite your best efforts of testing, some bugs and misconfigurations can’t be spotted until you run a real deployment, and watch the software interact with real dependencies (or at least their preproduction equivalents).

Before deploying to production proper you will want to run through deployment and post-deployment verification testing in such an environment to provide extra automated confidence before reaching real customers.

This post-deployment verification is also your last chance to have automated checks catch issues before they get released out to real customers.

In some situations, this may be a canary or an early-adopters group, though ideally you want something without real customers to do this testing on, and instead use feature flags for beta user test groups, and any A/B testing.

Production Deploy and Verify

It is now time to deploy to production. You’ve satisfied your tests and verified that it can work in a deployed environment after deploying, mitigating any issues in the production deployment process (which assumes your pre-prod and prod deployment processes are near-identical).

We repeat our deployment and verify steps once more only this time against our customer-facing production environment. Our change has now reached production.

If you do find production issues, you will want to go back and add regression checks to the testing at the appropriate point in the pipeline to make sure the same issue halts deployments in the future. This is one of the feedback cycles you need to adhere to when taking on CICD in this way, to strengthen the reliability and trust in the process.

Notification

Once you have a deployed and verified production deployment, you’ll want to make sure anyone (or any system) that wants to know about it can do so.

Specifically, you want to have release notes publish, to have any APM systems notified of the release, Internal Comms published on something like Slack or Discord, as well as any documentation related to the changes updated and published.

Ideally, all of this is automated and automatic. For documentation and release notes, having a way to derive these from the source itself, or from the Source Control revision history or commit messages, or Pull Request titles etc. This saves a bunch of manual busy work which would otherwise distract a human on every deployment.

The same goes for using APIs to post to Slack or other chat systems to notify other internal or external stakeholders of the change. A great way to identify the cause of a system outage can be to know which of your dependencies have made releases around the time an outage was introduced. Be nice to your API consumers.

Wrap Up

The above largely outlines a strawman of my MVP. Ideally, I would go through and set up at least the skeleton of this for any new service where I expect it to live in production for any period of time. (If I don’t think this code will be productionised and is merely a toy or demo, such overhead isn’t worth investing in). This would largely be even before any functionality is implemented (e.g. deploying the new empty project).

Having the Pipeline in place, we continue to extend the testing to improve confidence in the deployment process, and this goes hand-in-hand with building out the functionality.

Iteratively and continuously build and ship the software to meet the requirements of the project of work.

An almost seamless transition from Ghost to Hugo

Tue, 10 Aug 2021 20:30:00 +0000

Well, I finally did it. Ghost has been sitting on an older version for a long time, my certs were expiring, and I did something about it.

TL;DR;

My blog engine and hosting were deteriorating fast, so I moved to GitHub Pages, Hugo, and Static Site Generation, (mostly) without losing or breaking anything.

Synopsys

Years ago I moved from WordPress to Ghost. This let me get on with writing and not worrying about the hosting. I moved it to Azure and even added a dynamic SSL certificate rotator using Lets Encrypt.

Eventually, the cert rotator broke, and the blog engine got enough major versions behind to make the upgrade path non-trivial to fix. Not to mention Google Ads decided my site wasn’t good enough for them and so disabled any ads that could have shown up. Hosting costs on Azure, while small, were a factor to think about too.

Around the time I decided to do something about it, I also started paying attention to other people using static site generation to host their blogs. It didn’t take long to find a few really great articles that showed how simple this was going to be. The simplest of which seemed to be Hugo on GitHub Pages. So that is the path I went down.

What went well

My initial research was well-founded, and I very easily found the tool jbarone/ghostToHugo designed for the job.

Initially, everything was super simple, and I had my content in Hugo in under an hour. Mainly I was following youngkin’s article: Create a Free Blog Site Using GitHub Pages and Hugo, and then using ghostToHugo for my content.

Install Hugo CLI locally
create basis repository setup with the suggested theme youngkin
export my Ghost backup
install ghostToHugo tool
run ghostToHugo to migrate posts into the site
run locally to test

(I’m not repeating details here, because youngkin has it nailed.)

Getting this running on GitHub Pages was easy following that article too. Even setting a CNAME and DNS (a temporary subdomain while testing of course) was nice and easy to get working. (I use DNSimple.) Job done. Right?

What went wrong

RSS. My RSS feed has been a bit of an issue really. I knew it would probably break, and it wasn’t even available out of the box on Hugo either. So I fevered away adding RSS generation to the site generation and getting a fair approximation of my old feed. I knew it wouldn’t be perfect, but I learned a bit about how the templating engine worked along the way. And when that hot cutover day came, it certainly failed. I had a few bugs to fix and all 50 people who have added this blog to their feed probably got a bit of a noisy mess. But hopefully, as they now read this article they now understand why. Also, there are now sub-categories with sub-category RSS feed URLs linked to from the right-hand sidebar now.

Custom Theming. I built my custom theme way back when I moved to Ghost. And I didn’t really feel like losing it. So I painstakingly mapped it from the Ghost templating engine to the one Hugo uses. This sucked. and it took ages (weeks) to get it right. Along the way, I probably did make some improvements that will be valuable later, like fixing the sidebar and better image handling possibilities. But this was probably the most time-consuming part of the whole process. In the end, retrologs’ Creating a Hugo Theme From Scratch was super useful at figuring out what went where, but I don’t actually understand some parts of the collection lookup code I wrote in some places yet. But it worked. At the very least, my CSS appears to have held the test of time (surprisingly!).

Permalinks. Turns out Ghost had a few permalink redirect capabilities. And also the default URLs in Ghost were /slug-for-page/ while Hugo is /post/slug-for-page/. Luckily the Aliases feature in Hugo allowed me to add both the draft/preview link (/p/<GUID>/) as well as the old URL so that hopefully, no one loses my content, and instead get bounced to the right place.

Disqus. See the comments at the bottom? I’ve been using Disqus to make them available. Turns out this was quite useful in moving to a static hosted solution. For the most part, this went well, because Disqus actually have a migration tools feature available to use to move the URL associated with a page. The what went wrong part was that I prepared all the pages that looked like the normal post URL (/slug-for-page/) before I cut over my DNS, but didn’t map any of the preview URLs (/p/<GUID>/) across. I think what happened is that Ghost provided an id to Disqus as well as a URL, and it uses the first URL it sees as the identifying one. When I preview drafts, that preview URL is the one it saves. Since Hugo is just using the actual URL and not the id anymore, I lost my mapping over a few pages which I neglected to migrate. (My incorrect assumption was the draft ids didn’t have comments 😢.) I was without comments on some pages while I remedied that across a couple of evenings. It wasn’t too bad to put together a new CSV file for that though in the end.

Google Ads. Turns out they are still not happy with my website (and neither am I to be fair) so there is a bit more work to get them reenabled. It barely makes dollars a month, to be honest, but with it not working it makes a lot less of course.

Content. I spent far too much time looking at my content. While Ghost had tags, Hugo has both Tags and Categories. I went through every historic post (193 over 13 and a half years if you can believe it) and updated many tags (deleting a bunch, adding a few) and creating categories for my content. I could have spent days (and still might) tidying up some of the content that never had any attention since coming out of WordPress and getting auto-converted to markdown all those years ago. But I held back, since I wanted to actually get this process done and back to writing. There was a little bit of work migrating some of the image functionality over too, but for the most part that was fine, too.

Summary

Would I choose to do it again? Yes. I am certainly in a better place. So far, I think I’ve backed the right horse in Hugo. The workflow seems about as good as it was before, and I am aware of a few tools such as CloudCannon for better CMS management workflows if I desire. I might miss the Ghost app for drafts, but I mostly write on my desktop anyway. And since it is in GitHub, anything that can commit markdown to there is a way to draft.

Now that I have that done, the next step is likely to repeat the whole process with csmac.nz as well and fix its certs, ads, and content. (That last one has been bugging me since I moved to Ghost to be fair… ¯\_(ツ)_/¯)

Failed to query available provider packages - Terraform NewRelic Provider :facepalm:

Tue, 13 Jul 2021 19:18:43 +0000

Recently I tried upgrading from Terraform 0.12 to 1.0.1. I think the key here is leaving 0.12.

Anyway, there was a breaking change in 0.13 around how providers worked, and we got this cryptic error.

(Caveat: I don’t actually know how much of this has to do with a newrelic provider update as well or instead, but was what we got the week of 2021/07/12.)

Initializing provider plugins...
- Reusing previous version of newrelic/newrelic from the dependency lock file
- Finding latest version of hashicorp/newrelic...
- Using previously-installed newrelic/newrelic v2.23.0
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider hashicorp/newrelic: provider registry registry.terraform.io does not have a provider 
│ named registry.terraform.io/hashicorp/newrelic
│
│ Did you intend to use newrelic/newrelic? If so, you must specify that source address in each module which requires that provider. To see which      
│ modules are currently depending on hashicorp/newrelic, run the following command:
│     terraform providers
|

So I’d already added this at my root module but this error still occurred.

terraform {
  required_providers {
    newrelic = {
      source  = "newrelic/newrelic"
      version = "2.23.0"
    }
  }
}

provider "newrelic" {
  api_key    = "12345"
  account_id = "12345"
}

It took a while to find the right docs to understand this one.

The key message from above is the You must specify that source address in each module which requires that provider. message. We are using modules, and the breaking change to 0.13 and above is that modules can specify required_providers.

What they don’t tell you clearly, is that when trying to resolve “newrelic” provider by default, it will look for “hashicorp/newrelic”. However the actual provider source is “newrelic/newrelic”. So to make this work the optional required_providers is actually mandatory, and actually needs the source = "newrelic/newrelic" property.

terraform {
  required_providers {
    newrelic = {
      source = "newrelic/newrelic"
    }
  }
}

Adding this to every module that uses newrelic solved the problem.

If you are doing this, you might also need to migrate your state over as well. Do this after running terraform init from inside your config folder. This migrates the data in your state file to correctly map to the new provider.

terraform state replace-provider newrelic newrelic/newrelic

More details here: https://www.terraform.io/docs/cli/commands/state/replace-provider.html

If you do migrate 0.12 to 0.13 first, instead of stupidly jumping straight to 1.0.1, you can follow a more useful migration guide and the 0.13upgrade command instead. 🤦‍♂️

https://www.terraform.io/docs/cli/commands/0.13upgrade.html https://www.terraform.io/upgrade-guides/0-13.html

Kotlin, TeamCity and reflection

Tue, 11 May 2021 17:42:06 +0000

I’ve been getting stuck into upskilling in Kotlin. Mostly for TeamCity build automation and trying to produce quality tested code in a space where automation was retrofitted in.

Today’s fun little problem was the error message:

Runtime error RootProjectId: kotlin.reflect.jvm.internal.impl.builtins.KotlinBuiltIns$3[92]: java.lang.AssertionError: Built-in class kotlin.Any is not found

The canonical stack overflow question that (at the time of writing this) had no answer: https://stackoverflow.com/questions/67199546/why-am-i-getting-java-lang-assertionerror-built-in-class-kotlin-any-is-not-fou

The repro was the simplest piece of code in the end. I was even able to point it at a teamcity class over my own (Project):

import kotlin.reflect.full.declaredMemberProperties

val props = Project::class.declaredMemberProperties

(For reference, i tried a few version configurations, but they were mainly kotlin version 1.3.31 teamcity version 2019.2 and kotlin version 1.4.31 and teamcity version 202.2.2)

At this point in time and with some further testing, I’ve come to the conclusion that reflection just isn’t supported on teamcity with Kotlin. I’ve found one or two Issue links to back that up too: https://youtrack.jetbrains.com/issue/TW-59901

(¬_¬ )

If anyone else managed to find a solution that works, or if any future version adds the missing puzzle pieces that make this work, I would be greatful to hear.

For now, I have replaced the reflection with good ol’ verbose fragile code.

Luckily, the tests allow reflection no problem so at least the fragility of this case is covered. If someone adds a property then at least a test fails until the verbose code had been amended.

Bluetooth Devices keep disconnecting

Mon, 09 Nov 2020 07:11:07 +0000

It all started when I installed Windows Update 20H2 in October 2020.

I’ve had a stable machine (HP ZBook laptop for reference) for months and months with the same Bluetooth hardware, no issues. But that fateful day I installed the 20H2 update from Microsoft, everything changed.

Suddenly I was having disconnect issues. I have a Bluetooth keyboard and Bluetooth mouse connected (I think they are both low energy?) And I found that one of them would reconnect, and the other wouldn’t. If I turned one-off, the other would reconnect, but the first one wouldn’t be able to when turned back on.

As a temporary fix, I was able to connect the first device, then delete and re-add the second device. After somewhere between 10 minutes and half an hour, one device would disconnect again.

I tried all the solutions from googling this issue, but the only one that worked was to update the Intel Bluetooth drivers.

In Device Manager on Windows, under Bluetooth, the device is listed as “Intel(R) Wireless Bluetooth(R) and no amount of Windows Update to resolve new drivers helped. But then I found the official driver support from intel: Intel Driver Support Assistant.

I installed the tool and let it run. Lucky for me, there was a newer Bluetooth Driver for Windows 10 that I could download and install (amongst a few other Intel drivers as well).

After installing the latest intel Bluetooth driver, I haven’t noticed any other issues since. It is back to business as usual now.

Nullable and notnull Serialisation Properties

Sun, 26 Jul 2020 02:00:00 +0000

The C# 8 feature Nullable has been well received by myself and others into our workflows and has improved code bases immensely. However, there is one niggly workaround that I’m not a fan of, and that is = default!;

problem definition

// This class is used to serialise/deserialise a payload from a server
public class MyContractDTO
{
    public string Value { get; set; } = default!;
}

My codebase is now littered with this code hack to get it to compile because nullable says it can’t guarantee that this property is not null.

Let’s take a closer look at what we are saying.


// With the Nullable feature enabled, I need to ensure Value is not null:
// - either in the constructor from a notnull argument
// - or giving it a default non-null value.
public string Value { get; set; }

// Initialise the property to its default value (which for reference types like string is null)
public string Value { get; set; } = default;

// The '!' says to treat the value as if it is notnull, the developer knows better
public string Value { get; set; } = default!;

What does this achieve? Well, the compiler ignores the fact that it could be null, and we initialise it to be null. This is a contradiction if ever I saw one, and I don’t like it.

Normal solutions

There are two solution paths to take here:

Acknowledge that it might be null and make it nullable.
Initialise in the constructor.

These work great with normal codebases and in some cases, one is better than the other. If you know it certainly can be null, use the first option. If you know you never expect or want it to be null, use the second.

Bonus points once you have the constructor is to make the type immutable. For all domain and application logic, This is what I have done successfully so far and will continue doing.

But serialisation

Here is the problem. Serialisation.

When we have a type that is going to be used to deserialise transport models we again have the same choices as above, but a few more points to consider.

The JSON may or may not include the field
The JSON may have the field set to null
The Deserialiser might not support constructors
The Deserialiser doesn’t know about Nullable and can’t ensure the safety is upheld.

Again we can make the call to acknowledge that it might be null and make it nullable. But there are drawbacks here. If you do this, you have to add all the error-handling for dealing with nullable checks. And if you control both server and client in this situation, then you might be writing and testing code for something you never plan to, nor may never need to ever support. Being null is an exceptional/fatal situation you don’t want to have to constantly guard against.

Which leads us back to the de-facto solution currently being advocated and used:

public string Value { get; set; } = default!;

I’m still not happy. So why don’t we work on that?

The JSON may or may not include the field or it might be null

Luckily for us, this concern is fairly easy to address. Say we are using Newtonsoft.Json and want this extra piece of reassurance.

[JsonProperty(Required = Required.Always)]
public string Value { get; set; } = default!;

The Required attribute annotation is designed for exactly this situation. When we use Newtonsoft.Json as our deserialiser, we can get a JsonSerializationException for free.

This ensures we:

have minimal code doing the null checking
don’t have to guard every access to a nullable property
Treat null or missing as fatal errors as part of an existing serialisation error handling process (which we should always have anyway).

This still doesn’t stop any other piece of code from creating an invalid object state, though. But this may be the easiest solution to add those missing guarantees alongside using default!.

Use Constructors

Newtonsoft.Json helps once again by supporting constructors. Make sure all the mandatory non-nullable properties are in the constructor. And as long as there is no default constructor (which when doing nullable right you can’t anyway) and the constructor parameters have names matching the properties, this just works as expected.

public class MyContractDTO
{
    public MyContractDTO(string value)
    {
        if(value is null) throw new ArgumentNullException(nameof(value));
        Value = value;
    }
    
    public string Value { get; set; };
}

You do however have to write your null-guard into the constructor to ensure it fails with an appropriate error message. Without this, null might still sneak through, even if you annotate the property.

Once more, Newtonsoft.Json with constructors also means support for read-only (immutable) objects.

What about not using Newtonsoft.Json?

There is a new kid on the block - System.Text.Json. However, this is one area where it doesn’t shine so bright compared to Newtonsoft.Json.

Pretty much none of the above works. Constructors are not supported. Required annotation is not supported. Along with many other things.

More on what does and does not translate across from Newtonsoft.Json is documented here.

Instead, you will need to write your own converter and manually deserialise your object with your own explicit null checks (which could be done inside the constructor).

For example, given our simple type above, we might do the following:

[System.Text.Json.Serialization.JsonConverter(typeof(MyContractDTOConverter))]
public class MyContractDTO
{
    public MyContractDTO(string value)
    {
        if (value is null) throw new ArgumentNullException(nameof(value));
        Value = value;
    }
    
    public string Value { get; }
}

// Made using the examples given at https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-migrate-from-newtonsoft-how-to#required-properties
// Your decisions may vary.
// e.g. this only supports {"value": "..."} format, no extra properties allowed.
// For a more complex object, you would probably be more flexible.
// Better guides here: https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-converters-how-to
public class MyContractDTOConverter : JsonConverter<MyContractDTO>
{
    private readonly JsonEncodedText ValueName = JsonEncodedText.Encode("value");

    public override Implementation Read(
        ref Utf8JsonReader reader,
        Type typeToConvert,
        JsonSerializerOptions options)
    {
        if (reader.TokenType != JsonTokenType.StartObject)
        {
            throw new JsonException();
        };

        string? value = default;

        reader.Read();
        // One property must exist
        if (reader.TokenType != JsonTokenType.PropertyName)
        {
            throw new JsonException();
        }

        // That property must have the right name
        if (reader.ValueTextEquals(ValueName.EncodedUtf8Bytes))
        {
            value = ReadProperty(ref reader, options);
        }
        else
        {
            throw new JsonException();
        }

        reader.Read();
        // There must be no other properties
        if (reader.TokenType != JsonTokenType.EndObject)
        {
            throw new JsonException();
        }

        return new MyContractDTO(value);
    }

    private string ReadProperty(ref Utf8JsonReader reader, JsonSerializerOptions options)
    {
        Debug.Assert(reader.TokenType == JsonTokenType.PropertyName);

        reader.Read();

        return reader.GetString();
    }

    private void WriteProperty(Utf8JsonWriter writer, JsonEncodedText name, string stringValue, JsonSerializerOptions options)
    {
        writer.WritePropertyName(name);
        writer.WriteStringValue(stringValue);
    }

    public override void Write(
        Utf8JsonWriter writer,
        Implementation implementation,
        JsonSerializerOptions options)
    {
        writer.WriteStartObject();
        WriteProperty(writer, ValueName, implementation.Value, options);
        writer.WriteEndObject();
    }
}

At which point you now have a bunch more code and logic to look after, but can achieve the same/similar results to what Newtonsoft.Json could do. Depending on your situation, maintaining this code may be more effort than declaring it nullable and maintaining checks around that in consuming code. Up to you.

Decisions, Decisions

So what would I recommend?

Firstly, don’t just use default! on your serialised types.

For maximum effect, make your classes have constructors for notnull values, and maybe even make your properties Immutable, if that makes sense. Have your constructors guard against nulls so that your compile-time assurances have runtime verifications. Especially is this is a client library you provide to others.

If you are using Newtonsoft.Json you should at least apply the [JsonProperty(Required = Required.Always)] to all your notnull properties. But add the constructors as well because you can. If you can stick with Newtonsoft.Json, your life will be very easy.

If you are using (or have to use) System.Text.Json, write custom converters for your types so that you can have those constructors mentioned above. Make sure you keep them flexible enough to ignore any extra properties you might add in the future to avoid breaking backwards compatibility.

This experiment has a companion GitHub repo of tests (a mixture of proof they work, and proof they fail tests) available here.

Happy Null-Hunting.

We are all 10x engineers, but I don't think it means what you think it means

Mon, 15 Jul 2019 09:50:55 +0000

If your reading this blog, and used twitter in the past few days, you have probably already seen this tweet, or some of the replies coming through from the community:

10x engineers

Founders if you ever come across this rare breed of engineers, grab them. If you have a 10x engineer as part of your first few engineers, you increase the odds of your startup success significantly.

OK, here is a tough question.

How do you spot a 10x engineer?
— Shekhar Kirani (@skirani) July 11, 2019

As a developer who tries to be good at their job, I have thoughts on this. As a Human, I like to be heard. (Note that being heard is not the same thing as being agreed with, or being right.) So here I am giving my opinion on this idea of a 10x developer, and specifically these “10 signs” put up online.

I like the idea that there are no 10x people, instead, you can strive to produce 10x teams.

1. Meetings

A lot of people don’t like meetings. But unfortunately, communication is a big part of effective teams and businesses. While it is true that some people love meetings, and some meetings don’t hold productivity, the key here is communication. If you can find an effective means of communication within a team that works, that is most important. Having a team full of people that don’t have meetings doesn’t make a 10x team. If meetings are causing you or your team problems, it is work looking at other strategies for effective communication. There are alternatives out there.

2. Office Hours

“Nine to Five” is such an old fashioned idea these days. Not only do some countries have alternative ideas like 4 day weeks, long lunch hours, “flexitime”, but more and more business work across locations and across timezones. We are asking people to work outside these hours for various reasons already, so why not some give and take here?

Most people would love flexible work hours. It turns out that if people can work around their own schedule, they put their best work forward when they are present. I would much rather have someone contribute 100% at 9 pm at night because their kids are in bed and they can concentrate rather than pay them to sit at their desk for several hours texting and calling their family. Just a thought.

We are so fortunate that in our industry is so capable and adjustable to different working hours that it is a shame not to support this. It is not that “10x developers are the people who like this”, it is actually that most employees want this, and it can be done.

3. Screen colour and worn keyboards

If I saw a sports star on the field/pitch/court wearing badly worn shoes, I wouldn’t think “This is a superstar”, I would probably think “he isn’t paid enough”.

Why are you letting your developers use worn-out equipment? Buying the required hardware and software is such an important thing, just do it! In all seriousness, though, giving staff the right tools to get their job done is always a good idea.

There is an interesting dichotomy with custom technical tools as well. If you have one developer that goes against the grain of the team with customisation, then it both makes it harder for them to use other machines (e.g. during pairing/mobbing) and for others to use theirs. There is some efficiency to using the out of the box defaults too. You can get up and running faster, and waste less time doing customisations every time you refresh your machine or change jobs and get new hardware.

Also, if you have to have everything customised your own way, you won’t be able to acclimatise to new situations, new tools, new teams. Not someone you want to add to your team.

4. Good memory

One my big comparisons between my generation and the one before me (that is the one with the access to Google, but not access to iPads) was that at school we were no longer taught to memorise and recall information. Instead, we were taught, and learned, the importance of finding and referencing information. There is too much information for us all to hold in our heads. knowing where to look and what to look for is far more important that memorised information for a technology that will be obsolete in 3 years (if you are lucky), let alone memorising a dozen of them over time.

Yes, it is great to get people who know the domain really well, and for some people, it really is a skill. But people can’t know what they haven’t seen yet, and new hires always start as a blank slate. Not to mention tomorrow will bring a new set of libraries, applications and services that need to be learned and understood.

Scott Hanselman talks of the experience-groundhog-day scenario.

Do you have 10 years’ experience, or the same year 10 times? https://t.co/LhIeWiWl3H
— Scott Hanselman (@shanselman) May 29, 2018

5. Full Stack

Don’t you love buzz words? FullStack. A full-stack would be from the HTML/CSS/js to the backend code, to the database. But it is also understanding how the TCP/HTTP layer works with packets and routing. How does your platform infrastructure run, do you manage and patch the servers? Database backups and data security, not to mention how the hardware scales, can you write the assembly to work with arm processors too?

Jack of all trades, master of none. That is a saying that acknowledges you are spread too thin. You can’t be an expert in everything, so instead, you can hire an expert in each thing. Yes, you want everyone to be a generalist, but make sure you have coverage with all of the specialists, too.

I can’t say I’ve met too many people who can design and build an amazing user experience, and also optimise a large scale database, and run a datacentre. Oh and also works well with others…

6. Wait, WAT?

This one I just have to quote it in two parts:

10x engineers can convert “thought” into “code” in their mind and write it in an iterative fashion.

Just for a second, imagine with me, that you could replace the words “10x engineer” with “a developer”. Crazy idea, but it might actually work.

Given a product feature, they can write that entire feature in one or two sittings of 4 to 6 hours with a caffeinated drink without distraction.

We all have good days where we fire on all cylinders. Or a task or feature fits into just the right hole in a system. Cherish the days when it happens, but realise its more about the stars aligning and less about the ability of your developer. We are all capable of this. But the rest of the time it’s just a Monday.

7. Good Memory. Part Deux

We talked about this. Enough said.

8. Learning

This is another one of those things with a dichotomy. If you are always learning something new, you never get to master anything. There is a benefit to having people who are masters, and people who can learn, and people who can teach and share. You need balance both as a developer and person, but also in and within a team. If you don’t have this yet, look at hiring the capabilty, but don’t use it to rule in or out talented people.

9. Terrible leaders

Yes. I think exactly what you want to do is hire someone who cannot teach how the system works to new staff. That will help you grow. No, not really. The team is only as strong as its weakest point. And the ability to both cover for, and teach and grow newer talent is key to growth and success.

Also, while it is true that some people will be amazing on your team, but will interview poorly, I can’t say there is a direct reverse correlation here, either. Sorry.

10.

10x engineers rarely job hunt or move out of the company. They move out because you make their life miserable with the process, meetings, training, and other non-value-added activities. If you come across them, hold on to them. Celebrate them.

So close. This is the first point I almost agree with. Just go ahead and replace “10x engineers” again with “developers”, Or even just “employees” will do.

0.1x developers

While there are seeds in here that make for an interesting debate, they have nothing to with “10” or “X”. The best summary of the whole situation is this: The scale is off. I think the truth of the matter is that there are developers who might contribute to failings in a team (and some warning signs appear above, under the wrong inversion, though). We can call them the 0.1x developers. They are the ones to truly watch out for.

As founders and employers, you will want to be doing all the right things to attract and keep good talent, and that is more important than going after a mythical creature known as the 10x developer.

What someone needs to do is follow the footsteps of Dylan Beattie’s Rockstar programming language and make a language or piece of hardware board and call it “10x”. Then we can all be Rockstar Programmers and 10x Engineers.

Snuffleupagus-Oriented Programming

Mon, 08 Jul 2019 06:00:00 +0000

An impromptu introduction to Snuffleupagus-Oriented Programming (SOP): https://t.co/jPkjPRgqQr
— Casey Muratori (@cmuratori) July 21, 2015

Sometimes naming something is the best way to start talking about a topic that lots of people do but don’t realise or haven’t pulled together as a concept. This is one of those ideas that finally has a good (?) name.

How do you build out a new API? Here we are meaning programming contracts effectively. You can hack away until you have something functional, and then start consuming it. Better yet, build it out as code to do a job and then ship it to someone else to try and use.

Most developers get to a point where they see the value in having contract design up front across team boundaries, layer boundaries, reusable components, HTTP/Soap boundaries etc. An evolved design for these will never be as good as an intentional one. And on the flip side, we have all used clunky library APIs in various languages, BCL or otherwise.

What is Mr Snuffleupagus

The name comes from Sesame Street. Muppets basically. Big Bird and Bert and Ernie, Elmo etc. There was a storyline decades ago where Big Bird kept seeing Mr Snuffleupagus and talking about him, but no one else ever saw him. They concluded it was an imaginary creature for a long time. In the show, it was very ambiguous what the truth was. Eventually, they actually saw the Snuffleupagus too and concluded it wasn’t imaginary after all. More here.

Enter SOP

The ideas from this story match an approach to building your API.

First, we are going to imagine that what we want to use exists. Then we are going to write code against the API, crafting our usage case, without implementing anything. Once we are happy that our API is usable, we implement it and make it actually exist, make it real. There may be a few tweaks once the realities and limitations of the implementation are taken into account, but the idea is to try and keep it as usable as the original usage cases as possible, usability intact.

via GIFER

TDD

This is a great candidate approach to work with TDD as well, just with more implementation code along the way to produce passing tests. Like Mr Snuffleupagus, this implementation code is imaginary until you have enough test cases to make it real, and refactor.

Thanks

Go forth and start sharing SOP, Snuffleupagus-Oriented Programming. Imagine, Build Usage, Make it Real. Thanks Casey Muratori for coining the term and making it stick. See more of his programming on HandMade Hero (YouTube), and his company Molly Rocket for games and more.

Looking Back on C# 7: Pattern matching

Mon, 01 Jul 2019 06:00:00 +0000

With C# 8 on our doorstep, I figure it is a good time to reflect on recent additions to the language that have come before. There are some great improvements you may have missed, some that I really enjoy using, and some I consider have reached canonical usage status that I think are all worth some reflection.

Pattern matching is a powerful feature that has been unlocked against various language constructs in C#. The idea is to take existing features like case from switches and is and extract their capabilities into a “pattern matching” concept over types and values. This can then be applied back to these language features, and other places in the future.

Before the new concept of patterns, case accepted a constant as an argument, and is would accept a type.

// Check the type of the variable "is assignable to" Shape
if(widget is Shape)
{
    switch ((widget as Shape).Type)
    {
        // When value matches the constant "Square"
        case "Square":
            // ...
            break;
        default:
            break;
    }
}

With the introduction of C# 7.0, both of these language features have been enhanced to use the new Pattern Matching syntax. This is a backwards-compatible change, meaning that the pattern type can be a const or a type. As well as these existing cases, it also includes when clauses and var patterns as well.

is Expression

Originally, is was able to check a type. This could create more readable code but often left casting or as operator usage in the aftermath.

if (widget is Shape)
{
    var myShape = widget as Shape;
    // var myShape = (Shape)widget;
}

This has been improved by the pattern matching. First, we have constant checking:

if (widget.Type is null)
{
    // ...
}

if (widget.Type is "FOO")
{
    // ...
}

As well as this, the Type checking Pattern now includes support to create a scoped variable of the correct type, similar to the new out parameter functionality. This language feature is collectively known as “expression variables”.

if (widget is Shape shape)
{
    // ... use `shape`
}

Switch Statement

A switch used to only match on constants, but with the new pattern matching, we can do so much more.

void SwitchIt(object value)
{
    var result = 0;
    switch (value)
    {
        // We can still switch on constants, even when the types don't match
        case 0:
            break;
        // We can switch on type, and even create a scoped variable (like with `is`)
        case IEnumerable<int> childSequence:
        {
            foreach(var item in childSequence)
                result += (item > 0) ? item : 0;
            break;
        }
        // using `when`, we can do range or bounds checking
        case int n when n > 0:
            result += n;
            break;
        // We can constant check against null
        case null:
            throw new NullReferenceException("Null found in sequence");
        default:
            throw new InvalidOperationException("Unrecognized type");
    }

    return result;
}

There is another new pattern that is useful with switch, which is the var pattern. The “var pattern” is similar to the type pattern, except that it always matches, but creates a variable with the assigned value.

switch(shape)
{
    case Circle c:
        WriteLine($"circle with radius {c.Radius}");
        break;
    case Rectangle s when (s.Length == s.Height):
        WriteLine($"{s.Length} x {s.Height} square");
        break;
    case Rectangle r:
        WriteLine($"{r.Length} x {r.Height} rectangle");
        break;
    case var s:
        // Always matches (similar to default) but gives access to the value as `s`
        WriteLine($"This is some kind of {s.Name} shape");
        break;
}

It is worth pointing out that the order of your case arguments now matters. while not a logical breaking change, since it never mattered what order constants where declared, it would always match the right answer, this is a conceptual change you need to be aware of. Mixing and matching patterns in a switch mean that the order does matter, and the first pattern that matches will get executed. To repeat, when all cases are constants this wouldn’t make any difference to the outcome but more advanced checks will.

In C# 7.1, the patterns were extended to work correctly with generic variables as well. In C# 7.0, you could use these type patterns as expected by first casting the T foo value to an Object (which could cause Boxing) and then the type checking would all work. In C# 7.1, this cast is no longer necessary, and also avoids any boxing and unboxing along the way. The docs go over this is more detail with an example comparing the two implementations, as there are a few subtle differences, especially around null/default cases.

Pattern Matching coming soon

This feature is an interesting one in the sense that it is both newer and probably under-used and less known. You can program away happily in C# never needing to use it and not coming across it.

But with C# 8, there is a bunch of new features that will be using this existing pattern matching coming, so you might want to get on board with this in preparation, because soon you will need to understand code using it, and probably see a lot more of it around.

Bring on the Switch Expression!

Looking Back on C#: async and await

Mon, 24 Jun 2019 06:00:00 +0000

Multithreaded programming has always been a difficult thing to get your head around, and there are many pitfalls easily stumbled into. To help combat this, Microsoft gave us async/await in C#.

Async/Await is a language feature that has been around since Visual Studio 2012 and C# 5 and hides a bunch of the boilerplate state machine code required to safely park a logical thread of execution while it waits for some work to complete or to respond from another thread, IO, or network device. This allows code to be more logically procedural and linear, therefore easier to read and comprehend.

Since it first came out there has been a raft of improvements across different versions of C#.

The basics

The foundation of how it all works rests on the shoulders of a Library, the Task Parallel Library, or TPL. Tasks have been around since 2010 and were part of the .Net 4 Framework. Similar to what Promises provide in javascript, this library allowed a logical chain of execution across waiting for blocking or longrunning execution while releasing the UI thread from being blocked. This was the introduction of TaskFactory and Task, in a fairly similar form to what we have today. This was a huge improvement from the days of callback chaining because it reduced heavy nesting of lambdas into more of a linear pipeline and a clear place for error handling to take place.

When C# 5 introduced async/await as a first-class language feature, it was able to leverage the library and extend on it to give us more readable code. It even handled Exceptions by throwing them in the place where await appeared, making try/catch blocks useful in asynchronous code.

What does async/await look like in C# 5?

// A classic synchronous method
public string MakeAWebRequest(Uri uri)
{
    // WebClient has synchronous methods, but it is recommended to use HttpClient for newer apps
    var client = new WebClient();
    return client.DownloadString(uri);
}

// It's async younger brother
public async Task<string> MakeAWebRequestAsync(Uri uri)
{
    // I use WebClient again for better comparing. Use HttpClient!
    var client = new WebClient();
    return await client.DownloadStringAsync(uri);
}

There are two distinctive features in this comparison. The return type is wrapped in a Task<T>, and there is a keyword async on the method signature with await beside method calls that return Task<T> results. Otherwise, the linear execution flow is largely unchanged.

You can easily use the return type without the keywords. In this case, the code works and operates as normal, passing object references around without any async state. The result object captures the state required for the caller to do the asynchronous work in the future, or respond to its completion. However, if you do use async and await, then you should always return either Task (where usually returns void) or Task<T> (where usually returns T). (In my opinion, there are no reasons left to ever do async without Task - there used to be but not anymore.)

Now we have that out of the way, let’s move forward to C# 6 and beyond!

Async and Exceptions

Exception handling was a big part of this feature on day one. You could simply wrap your async calls in a try{}catch{} and it would work as you would expect it to. The task you are awaiting throws an Exception, your catch triggers.

But initially, this did not work inside the catch or finally blocks in any expected way. In fact, it caused a compiler error. In C# 6 await in catch/finally blocks were given proper compiler support to do the right thing.

Using the example from The new language feature docs directly:

Resource res = null;
try
{
    res = await Resource.OpenAsync(…);       // You could do this.
    …
} 
catch(ResourceException e)
{
    await Resource.LogAsync(res, e);         // Now you can do this …
}
finally
{
    if (res != null) await res.CloseAsync(); // … and this.
}

Async and Console Apps

The language version was C# 7. We had async everywhere, and the NetStandards and Frameworks were full of async API calls and interfaces. It was a contagious thing, and you really had to jump through hoops to try to call a red function from a blue one.

And then there was main. That pesky little entry point into your application. The one the compiler generates for you that kicks off all of the application execution. The one that had to call into your top-most async method to RunAsync. And it had to be public static void Main(string[] args) or public static int Main(). That is not an async method.

Luckily, with the first ever minor language update, C# 7.1, we were given the mighty and powerful async main!

// You can finaly use Task<int>!
public static async Task<int> Main()
{
...
}

// Also available in no return value flavour!
public static async Task Main()
{
...
}

Async and Tests

We have a tonne of async methods in our system. But we should also be testing that code. And our testing frameworks were synchronous.

Luckily our testing frameworks have finally caught up, and with XUnit we can write tests that return Task and are async, and we also have the IAsyncLifetime` interface.

// This is what we had to do to test async
public class MyTestCase
{
    private readonly MyClass _systemUnderTest;

    public MyTestCase()
    {
        _systemUnderTest = MyClass();
        
        Task.Run(async () => await _systemUnderTest.Init()).GetAwaiter().GetResult();
    }
    
    public void CanRunSuccessfully()
    {
    
        var result = Task.Run(async () => await _systemUnderTest.Run()).GetAwaiter().GetResult();
        
        Assert.True(result);
    }
}

Using Task.Run and GetAwaiter or other synchronising methods can be very error-prone, and prone to deadlocks. Avoiding these is the best approach always (though still not always avoidable).

The best way to tet with async
public class MyTestCase : IAsyncLifetime
{
    private readonly MyClass _systemUnderTest;

    public MyTestCase()
    {
        _systemUnderTest = MyClass();
    }
    
    public async Task InitializeAsync()
    {
        await _systemUnderTest.Init();
    }


    public Task DisposeAsync()
    {
        return Task.CompletedTask;
    }

    public async Task CanRunSuccessfully()
    {
   
        var result = await _systemUnderTest.Run();
        
        Assert.True(result);
    }
}

I’m not sure if other test runners are able to handle async as well as XUnit does, so I always just use XUnit.

ValueTask

The original implementation of the async/await language feature was strongly tied to the Task and `Task types.

In C# 7, the language feature was enhanced, similar to other features, to use a pattern based on method signatures (like the Add for the initialiser syntax). Specifically, the GetAwaiter method must be available on the type used with the await keyword.

Along with this change was the introduction of a new type to leverage this pattern, ValueTask. A ValueTask is a value type (struct) that will be stack-allocated and copied by value. If your method uses caching, and most of the time returns a simple value instead of an awaited execution, the ValueTask may be more efficient than the Task type.

This is because the overheads of Heap-allocation of the reference type Task can have an impact on performance. If you detect this as an issue, you can use the new ValueTask instead. This will be a stack-allocated value type containing the response value and copied around.

Guidance for this: if you mostly return a value, but occasionally call an actual asynchronous IO execution, ValueTask will probably add value (e.g. heavy result caching). If you actually await most of the time, Task should be fine. As usual, measure and test before making the change arbitrarily.

Conclusion

Async programming is becoming the canonical way of building apps with IO in most languages. And in practice, most of the apps I write are IO-bound apps. C# and dotnet make this simple with async/await and the language keeps improving our experience using this successfully. This is a must-use feature that is unavoidable, but knowing the limitations and extensions available to use it well is still very important to do. Use it, but make sure you know enough about how it works to use it well.

Looking Back on C# 7: refs enhancements

Mon, 17 Jun 2019 06:00:00 +0000

We talked about the out variables in the previous post in the series, but there are a few other enhancements related to ref as well.

Hopefully, the concepts of values, pointers, stacks and heaps make sense to you at a conceptual level. These are crucial concepts to using and understanding ref and out.

As a quick recap, stack memory is local to a function and the function call stack. This memory is semantically “pushed” to for each method call, and “popped” from on each return (the call stack, stack overflow etc). The heap is a shared memory space where objects are allocated and stored, which is acted on by the Garbage Collector.

Values are mostly stack-allocated. They are copied around so that setting a value variable does not affect any other variable. Primitive types and structs are Value types. Reference types are the classes in C#. Classes are always stored on the heap, and all variables of class types are pointers or references. Assigning a reference to a new variable will point to the same value on the heap. Changing a field value on a reference type will be reflected in both variables. (Boxing values onto the heap is another thing, too.)

We already have out that allows the caller to declare a variable on the current stack that it passes by address (pointer, reference, or ref) that the method is contracted to set for you. This essentially gives us some of the power of reference types from a stack-allocated value. When we use ref, we gain all of the power of passing by reference that we have from heap allocated class types. We are essentially saying that the caller can use the existing value of the reference, and also set a new value to the stack variable if it wants to as well. Basically, all the restrictions that out imposes are taken off. A ref may not just be a stack variable, but could also be a field on a class that you want to access directly by reference, instead of having to constantly dereference it.

Pass by ref has been in C# since the beginning, but like the out parameters, only works in method signatures. However, from the beginning, you could never declare a ref variable.

In C# 7, ref has been extended to work with return types, and with variables. You can return a value by reference. And so that it the returned result can be assigned to something in a useful way, we also now have ref variable typing. Like most C# language features, there is safety built in.

You must add the ref keyword to the method signature and to all return statements in a method.
A ref return may be assigned to a value variable (by copy) or a ref variable.
You can’t assign a standard method return value to a ref local variable.
You can’t return a ref to a variable whose lifetime doesn’t extend beyond the execution of the method.

These rules ensure the safety of your code and ensure readability, that it is clear about what is happening.

I’m going to blatantly steal the Microsoft examples for this because I don’t want to invent an example and get it wrong.

// We declare the method as returning by reference (rather than copy)
public ref int Find(int number, int[] numbers)
{
    for (int i = 0; i < numbers.Length; i++)
    {
        if (numbers[i] == number)
        {
            // All returns must use the `ref` keyword
            return ref numbers[i]; // return the storage location, not the value
        }
    }
    // We can still throw an exception if necessary
    throw new IndexOutOfRangeException($"{nameof(number)} not found");
}

// arrays are already allocated to the heap and passed by reference rather than by value
// (see the `stackalloc` keyword for stack allocating arrays, though)
int[] array = { 1, 15, -39, 0, 7, 14, -12 };

// We have to use `ref` to call the method
// We choose to declare `place` as a reference
ref int place = ref Find(7, array); // aliases 7's place in the array
place = 9; // replaces 7 with 9 in the array
WriteLine(array[4]); // prints 9

In the above example, we could have chosen to declare place without the ref keyword and the value returned would be copied instead. However, in this case, the assignment of place = 9; would be overriding the local copy, and not modifying the original array.

Why would you use these pass by reference additions? Huge performance enhancements can be achieved by avoiding stack and heap copying or dereferencing of values in certain algorithms. Performance is the name of the game here.

In C# 7.2, the conditional operator (isTrue ? x : y syntax) can now evaluate to a reference result when both operands (x and y above) are also references.

ref var r = ref (arr != null ? ref arr[0] : ref otherArr[0]);

In 7.2 we also got ref readonly, which allows returned references to disallow modification enforced by the compiler. This may save time constantly dereferencing a child field in a scenario that you need to get the latest value, for instance in a loop. Again, performance is the target use-case.

The first version of ref variables was immutable only. Whatever you declared them to point to was what they always referred to for their lifetime. In C# 7.3, they were updated so you can reuse a variable to point to a different reference instead.

To complement the safety of the out restrictions compared to ref we also get the in keyword.

Declaring a method parameter with in essentially makes it a read-only reference. the method is not allowed to modify the value passed in but gets all the benefits of being passed by reference rather than by value (copied). The in keyword will make the compiler ensure that the method is not allowed to modify the original passed in value. If necessary it will create a defensive shadow-copy to ensure that is true.

This is well paired with another new feature, readonly struct. Declaring a struct as read-only means the compiler will ensure you are indeed read-only. (It will disallow public int Foo { get; private set; } for example.) You can use the in keyword for any methods that you want to take a ref to one of these structs again to ensure clarity when reading the code, but also enforced by the compiler.

I mentioned the defensive shadow copy above. The language and runtime do not guarantee that the internal implementation detail of a Property or Method is non-mutable from the contracts, so the compiler will get defensive, and make copies before calling anything that might cause a mutation. This way, the language guarantees the expectations of passing a read-only reference, but maybe doesn’t match performance expectations in the process. As a developer, by making the type a readonly struct instead, the compiler can rely on the guarantees and won’t make any copies. The struct won’t compile if it mutates any of its internal state, so we have stronger guarantees at compile-time and run-time.

These features are certainly power features, and when you need them they will be useful. But like most advanced features, you may be sacrificing readability for performance and optimisation. Use sparingly, but maybe measure first and then sprinkle in and measure again.

Looking Back on C# 7: Out variables

Mon, 10 Jun 2019 06:00:00 +0000

What are out parameters?

Sometimes you want a method to pass back a value by reference. In C and C++ this is done by passing the address in as an argument. In C#, we use the out keyword.

public bool HasCount(string str, out int length)
{
    length = 0;
    if (str.Length > 0)
    {
        length = str.Length;
        return true;
    }
}

string myString = "A String";
int length;
if (HasCount(myString, out length))
{
    // Do some stuff here.
}

A method must initialise the out parameter value, but the caller doesn’t have to initialise it first. Callers must use the out keyword when calling a method with an out parameter. This helps with readability making it unambiguous how it works.

More details of this for those unfamiliar with the concept can be found here.

In the libraries, there are a few methods that use this, for instance Int32.TryParse.

int number;
if (Int32.TryParse(numberAsString, out number))
{
    // use `number` as an integer value. 
}

It also comes in useful when you are building C-style interops. But not something you see very much these days.

New out variables

This syntax always felt a bit clunky. Modern APIs would suggest using a return value over an out parameter, and the early use cases tended to be either cross-compatibility with C libraries or parse scenarios. Primarily it was discouraged for readability reasons, though there are benefits from using it in certain places.

However, with C# 7, a new syntax was introduced, allowing inline declaration of out parameters.

if (Int32.TryParse(numberAsString, out int number))
{
    // use `number` as an integer value. 
}

You can now declare the parameter inline, without needing the empty declaration line that was essentially wasting line-count. The other benefit is you can now use var instead of an explicit type.

if (Int32.TryParse(numberAsString, out var number))
{
    // use `number` as an integer value. 
}

In C# 7.3, this out variable syntax was extended to include field initializers, property initializers, constructor initializers, and query clauses. Basically, you have the power to use this everywhere that you can use out parameters.

With the new changes, readability has hugely increased, the risk of using the value before initialisation has been removed, and it is probably something worth encouraging using again in those methods that could benefit with having out parameters for clean APIs, and performance.

A small but simple improvement to the language that I would replace in every place you use out parameters.

Looking Back on C# 7: Local functions

Mon, 03 Jun 2019 10:00:00 +0000

Lambdas - a recap

All the way back in C# 3 we were given lambdas. These are anonymous functions that can be passed around as what are essentially function pointers.

“Anonymous” refers to the fact that the function has no name, and is not tied to an actual class or instance. The term comes from functional programming.

In terms of implementation details, there are two categories of lambdas, those which stand alone and are pure functions, and those that have a captured scope, known as a closure. Closure, again, is a functional programming term. We capture variables from the scope of the parent and encapsulate them into this anonymous function instance.

Pure functions can easily be refactored into a public static function in a static class very easily, and the compilation is very similar.


static Action<int> GetRandomValueFunction()
{
    Console.WriteLine("Use static value for counter: {0}", counter);
    return () => 5;
}

// These two methods are the same as the above
static Action<int> GetRandomValueFunction()
{
    Console.WriteLine("Use static value for counter: {0}", counter);
    return StaticValueFunction;
}

static int StaticValueFunction()
{
    return 5;
}

There is no simple construct equivalent for closures. Hopefully, by example, we can see how these transform into simple static methods anyway. (In an example stolen from SO)

 static Action CreateShowAndIncrementAction()
{
    Random rng = new Random();
    int counter = rng.Next(10);
    Console.WriteLine("Initial value for counter: {0}", counter);
    return () =>
    {
        Console.WriteLine(counter);
        counter++;
    };
}

Given the above C# code, the compiled code will more closely resemble the below:


static Action CreateShowAndIncrementAction()
{
    ActionHelper helper = new ActionHelper();        
    Random rng = new Random();
    helper.counter = rng.Next(10);
    Console.WriteLine("Initial value for counter: {0}", helper.counter);

    // 
    return helper.DoAction;
}

class ActionHelper
{
    public int counter;

    public void DoAction()
    {
        Console.WriteLine(counter);
        counter++;
    }
}

Because the compiler generates these, it also controls access and visibility, so that you cannot actually access it in this way from your code directly. But it does all this automatically for you, and with fewer lines of code to achieve the same logical execution. It is also clearer from the lambda version that the calling method is the owner of the function, and no one else can or should share it.

Clearly, lambdas reduce our lines of code and are superior.

working with lambdas

Sometimes, though, your lambdas get complicated. When you have big complicated lambdas inside LINQ pipelines, it gets messy fast.

public List<Widget> GetWidgetsByPlumbob(string plumb, int restrictionNumber)
{
    var restriction =  _store.GetRestriction(restrictionNumber);
    
    
    // This `where` clause is only going to get more complicated from here.
    _store
        .GetWidgets()
        .Where(w => (w.PlumbBob.StartsWith(plumbob) || w.PlumbBob.EndsWith(plumbob)) && w.Restriction == restriction)
        .Select(Map)
        .ToList();
}

We have a few options prior to C# 7:

public List<Widget> GetWidgetsByPlumbob(string plumb, int restrictionNumber)
{
    var restriction =  _store.GetRestriction(restrictionNumber);
    
    _store
        .GetWidgets()
        .Where(w => Filter(w, plumb, restriction))
        .Select(Map)
        .ToList();
}

private bool Filter(Widget widget, string plumb, Restriction restriction)
{
    return (w.PlumbBob.StartsWith(plumbob) || w.PlumbBob.EndsWith(plumbob)) && w.Restriction > restriction;
}

There are a couple of problems with this. We are now passing all the parameters through to the nested function. This is fine, but from a maintenance point of view, we might add more parameters and we now end up maintaining two signatures every time we change.

Also, we open up for another function to start using the “sharable” filter function. Now we would be coupled to that new function. We can’t change our own filter without affecting that other new function, and that adds brittle fragility. Sometimes duplicating logic that has different reasons to change is worth the duplication, but the architecture of this code does not guard against this.

Another example is to pull the lambda into a variable:

public List<Widget> GetWidgetsByPlumbob(string plumb, int restrictionNumber)
{
    var restriction =  _store.GetRestriction(restrictionNumber);
    
    var filter = w => (w.PlumbBob.StartsWith(plumbob) || w.PlumbBob.EndsWith(plumbob)) && w.Restriction == restriction);
    
    _store
        .GetWidgets()
        .Where(filter)
        .Select(Map)
        .ToList();
}

Looking at the example, it adds some simplification. Because like the original example this lambda is a closure, the lambda can match the expected signature of the Where LINQ Extension.

However, sometimes this approach causes issues with type interpolation. That is, if you pull out into a variable, often you can’t use var or need to add explicit type casting to help the compiler out, or it will not compile.

I don’t think we have actually helped with the readability as much as we would like. It may not be obvious on the first pass of reading the function that this is a lambda, not a statement. This is also a simple example, and they can get more complex fast.

Put functions inside your functions!

With C# 7, we can now use a nested function. This gives us the benefits of not polluting the namespace of the class, while also making it more readable. It also makes it clearer that the function is owned by the caller as the only consumer.

public List<Widget> GetWidgetsByPlumbob(string plumb, int restrictionNumber)
{
    var restriction =  _store.GetRestriction(restrictionNumber);
    
    bool Filter(Widget widget)
    {
        return (w.PlumbBob.StartsWith(plumb) || w.PlumbBob.EndsWith(plumb)) && w.Restriction > restriction;
    }
    
    _store
        .GetWidgets()
        .Where(Filter)
        .Select(Map)
        .ToList();
}

Our compiler now ensures no one else can use this function. It is only callable from this method and gives the reader the knowledge that this is a specific implementation detail for this function only and not a shared common piece of logic. (Encapsulation.)

In this example it also still allows us to use the simplified Where call.

The best example of where this is really useful is recursion.

Often a recursive algorithm has a bootstrap function, that then calls the recursive part. Let’s print a tree of items with indentations.

public static void PrintLines(TextWriter out, Tree items)
{
    out.WriteLine(items.Title);
    foreach(var node in items.Children)
    {
        Print(out, node, "");
    }
}

private void Print(TextWriter out, TreeNode node, string indent)
{
    out.WriteLine("{0}{1}", indent, items.Title);
    if(node.HasChildren)
    {
        foreach(var node in items.Children)
        {
            Print(out, node indent + "    ");
        }
    }
}

For simplicity you can now write this:

public static void PrintLines(TextWriter out, Tree items)
{
    void Print(TreeNode node, string indent)
    {
        out.WriteLine("{0}{1}", indent, items.Title);
        if(node.HasChildren)
        {
            foreach(var node in items.Children)
            {
                Print(out, node indent + "    ");
            }
        }
    }

    out.WriteLine(items.Title);
    foreach(var node in items.Children)
    {
        Print(node, "");
    }
}

This example may not reduce lines of code by much, but the cognitive load of the encapsulation can be hugely beneficial when in a class with more service methods as well.

Summary

The cliché “another tool in the toolbelt” comes to mind but this is certainly that, and sprinkled through code strategically can really help with readability and maintainability. Not a “use often” but certainly something I can and have used in my dotnet apps.