tetsuo.github.io

Old-school LLM chatroom

noemail@noemail.org (Onur Gündüz) — Tue, 19 Aug 2025 13:37:00 GMT

burp is a chat server that connects clients to OpenAI and Anthropic APIs. It provides HTTP endpoints and a browser-based chat frontend.

Getting started

go install github.com/tetsuo/burp

This installs a burp executable to your $GOPATH/bin.

Start the server

burp

By default it binds to localhost:9042. You can change this with the -addr argument.

Set API keys via environment variables:

OPENAI_API_KEY for OpenAI models
ANTHROPIC_API_KEY for Claude models

Send messages

POST to /ask?id=<channel>&model=<model> with body text

curl --header "Content-Type: text/plain" \
  --request POST \
  --data "Tell a joke" \
  "http://localhost:9042/chat?id=emu&model=claude-3-haiku-20240307&temp=0.75"

Receive messages

/wait?id=<channel> - long-poll up to 30s
/recent?id=<channel> - fetch message history

Use the web UI

Open /chat?id=<channel>&model=<model> in a browser

How it works

Sending messages

Message history & Context

burp keeps a rolling, in-memory buffer of recent messages per channel (default: last 50 entries). Old messages are dropped after ~1 hour or when the buffer grows beyond the minimum keep size.

You can access the channel history by calling /recent?id=<channel>.

This history is also passed back into the provider on each /ask request, together with the system prompt, so replies remain contextual.

Message parameters

When you send a message, burp forwards generation parameters to the chosen provider. Supported fields include:

temp – temperature, always set (0.0–2.0 for OpenAI, 0.0–1.0 for Anthropic)
max_tokens – capped per-model, defaults to the model’s maximum
top_p – optional nucleus sampling
top_k – optional, Anthropic only

These values are stored alongside the channel state and displayed in the chat UI, so you always know what settings were used.

Receiving messages

Responses from OpenAI or Anthropic are streamed into an in-memory queue powered by github.com/tetsuo/bbq.

Each request gets its own small queue: the provider SDK writes token deltas into it, and a batching loop reads from it to publish messages back into the channel.

Long polling

When a client calls /wait?id=<channel>&after=<time>, the server will:

Hold the request open for up to 30 seconds.
If a new message arrives in that channel after the given after timestamp, it is immediately returned.
If no message arrives in that window, the server responds with a timeout marker message and the client retries with the latest cursor.

What's next?

I built this mainly as a console for my LLM dev environment. The chat UI reuses bits from a WebRTC-powered P2P chat app I hacked together years ago at a Mendix hackathon.

In the future I might add slash commands (IRC-style) or even Matrix integration, but for now the chatroom alone is already a solid base, especially for wiring into tools like Google Sheets or Excel as an extension backend/UI.

Old-school JSX syntax

noemail@noemail.org (Onur Gündüz) — Thu, 5 Jun 2025 00:00:00 GMT

restache extends plain HTML with curly braces and compiles to modern JSX, so you can write React components like it's 2013 again.

Example: Dashboard

The tetsuo/dashboard repository provides an example setup with ESBuild and a set of basic UI components demonstrating the current capabilities 👉 view it online.

Try it yourself

Here's an example

<ul>
  {#fruits}
    <li>{name}</li>
  {/fruits}
</ul>

restache v0 draft

restache is an HTML syntax extension that lets you write JSX-style templating directly inside HTML.

It extends HTML5 with Mustache-like primitives to support variables, conditionals, and loops.

Variables

Variables provide access to data in the current scope using dot notation.

Accessing component props

<article class="fruit-card" data-fruit-id={id}>
  <h3>{name}, eaten at {ateAt}</h3>
  <p>{color} on the outside, {flavor.profile} on the inside</p>
  <img src={image.src} alt={image.altText} />
</article>

They can only appear within text nodes or as full attribute values inside a tag.

✅ can insert variable {here}, or <img href={here}>.

When

Renders block when expression is truthy

{?loggedIn}
  <welcome-banner user={user} />
{/loggedIn}

Unless

Renders block when expression is falsy

{^hasPermission}
  <p>You do not have access to this section.</p>
{/hasPermission}

Range

Iterates over a list value

<ul>
  {#fruits}
    <li>
      {name}
      <ul>
        {#vitamins}
          <li>{name}</li>
        {/vitamins}
      </ul>
    </li>
  {/fruits}
</ul>

Range blocks create a new lexical scope. Inside the block, {name} refers to the local object in context; outer scope variables are not accessible.

⚠️ Control structures must wrap well-formed elements (or other well-formed control constructs), and cannot appear inside tags.

Comments

There are two types of comments

<!-- Comment example -->

<span>{! TODO: fix bugs } hi</span>

Standard HTML comments are removed from the generated output.

restache comments compile into JSX comments.

Generating JSX

restache transpiler generates a React JSX component from each input and handles JSX-specific quirks where necessary.

Fragment wrapping

Multiple root elements are wrapped in a Fragment

<h1>{title}</h1>
<p>{description}</p>

This also applies within control blocks.
If you only have one root element, then a Fragment is omitted.

Case conversion

React requires component names to start with a capital letter and prop names to use camelCase. In contrast, HTML tags and attribute names are not case-sensitive.

To ensure compatibility, restache applies the following transformations:

Elements written in kebab-case (e.g. <my-button>) are automatically converted to PascalCase (MyButton) in the output.
Similarly, kebab-case attributes (like disable-padding) are converted to camelCase (disablePadding).

kebab-case 🔜 React case

<search-bar
  hint-text="Type to search..."
  data-max-items="10"
  aria-label="Site search"
/>

ℹ️ Attributes starting with data- or aria- are preserved as-is, in line with React's conventions.

Attribute name normalization

Certain attributes are automatically renamed for React compatibility

<form enctype="multipart/form-data" accept-charset="UTF-8">
  <input name="username" popovertarget="hint">
  <textarea maxlength="200" autocapitalize="sentences"></textarea>
  <button formaction="/submit" formtarget="_blank">Submit</button>
</form>

<video controlslist="nodownload">
  <source src="video.mp4" srcset="video-480.mp4 480w, video-720.mp4 720w">
</video>

Attribute renaming only occurs when the attribute is valid for the tag. For instance, formaction isn't renamed on <img> since it isn't valid there.

However, some attributes are renamed globally, regardless of which element they're used on. These include:

All standard event handler attributes (onclick, onchange, etc.), which are converted to their camelCased React equivalents (e.g. onClick, onChange)
Common HTML aliases and reserved keywords like class and for, which are renamed to className and htmlFor
Certain accessibility- and editing-related attributes, such as spellcheck and tabindex

See table.go for the full list.

Implicit key insertion in loops

When rendering lists, restache inserts a key prop automatically, assigning it to the top-level element or to a wrapping Fragment if there are multiple root elements.

Key is passed to the root element inside a loop

{#images}
  <img src={src}>
{/images}

If there are multiple roots, it goes on the Fragment

{#items}
  <h1>{title}</h1>
  <h3>{description}</h3>
{/items}

Manually set key when there's single root

{#images}
  <img key={id} src={src}>
{/images}

Importing other components

restache supports an implicit module system where custom elements (i.e., tags that are not part of the HTML spec) are automatically resolved to file-based components.

Component imports are inferred from the tag names. The following examples show how different components are resolved:

HTML	JSX	Import path
`<my-button>`	`<MyButton>`	`./MyButton`
`<ui:card-header>`	`<UiCardHeader>`	`./ui/CardHeader`
`<main>`	`<main>`	Not resolved, standard tag
`<ui:div>`	`<UiDiv>`	`./ui/div`

Component resolution

Any tag that isn't a known HTML element is treated as a component.

When the parser encounters such a tag, it follows these steps:

1. Check for namespace

restache first determines whether the tag uses a namespace. Namespaced tags contain a prefix and a component name (e.g., <ui:button>).

2. Standard custom tags

If the tag does not contain a namespace (e.g., <my-button>):

restache first looks for an exact match in the build configuration's tagMappings.
If no mapping is found, it falls back to searching in the current directory. For example, <my-button> could resolve to either ./my-button or ./MyButton.

3. Namespaced tags

If the tag does contain a namespace (e.g., <ui:button>):

restache first checks the tagPrefixes configuration. If a prefix (e.g., ui) is defined, it uses the mapped path. For example, if mui is mapped to @mui/material, then <mui:app-bar> resolves to @mui/material/AppBar.
If no mapping is found, it attempts to resolve the component from a subdirectory: e.g., <ui:button> → ui/Button.js, ui/button.jsx, ui/button.tsx, etc.

ℹ️ Note: Standard HTML tags are not resolved as components, even if identically named files exist in the current directory. However, namespacing can override this behavior. For example, <ui:div> will resolve to ./ui/div (or ./ui/Div), even though <div> is a native HTML element.

ESBuild integration

restache includes an ESBuild plugin that makes integration simple and easy in Go:

Register .html loader and pass it to the plugin
Plugin uses restache compiler to convert to .jsx
No runtime library needed; everything is transpiled ahead of time

The dashboard project includes a working build script (build.go).

There's currently no support for Node.JS environment, but planned.

Roadmap

Integration with React hooks

Currently, most logic must live in a .jsx file next to the corresponding .html file.

The long-term plan is to introduce a minimal set of expressions into the language itself so that common hooks like useState, useContext, and useSelector from Redux can be inferred from markup and compiled automatically.

Relational and logical expressions

Support for predicates inside range blocks is planned for v1.

{#products: price < 100 && inStock}
  <product-card />
{/products}

This could be compiled as a filter, and potentially mapped to things like backend queries (e.g. MongoDB, CouchDB, Algolia, ...) as well.

Smarter code generation

Before adding expressions, there's still a lot that can be optimized with the syntax that's already in place.

Consider pattern matching. Since restache doesn't support expressions beyond dot notation, patterns have to be represented structurally. For example, using an object with a mutually exclusive key set:

{
  home: {...},
  settings: undefined,
  products: undefined
}

Then in the template:

{?home}    <home />    {/home}
{?settings}<settings />{/settings}
{?products}<products />{/products}

Compiles to:

if (props.home) <Home />
if (props.settings) <Settings />

This results in an O(n) operation instead of an O(1) equality check like switch(route), but the difference is negligible unless you're dealing with many conditions.

Future versions of restache will generate if/else or switch statements when keyed unions are used, along with other optimizations such as merging adjacent {?x} and {^x} blocks into a single conditional.

More codegen targets

Plans include:

Emitting a real JavaScript AST instead of raw JSX strings
Supporting things other than React
Option to emit TSX

When TypeScript is used and type information is available at build time, more advanced optimizations may be possible. But even without that, structural inference allows detecting optionals and iterables, which can be used to emit a generic type for the component.

Fortune as a Service

noemail@noemail.org (Onur Gündüz) — Mon, 31 Mar 2025 00:00:00 GMT

fortune is a simple HTTP API that delivers random fortune cookies, built as a ready-to-deploy reference for setting up observability and telemetry on GCP.

API

The fortune API has two endpoints. GET / returns a random fortune from the database. POST / accepts a plain text body with fortunes separated by %, as per the original format of the Unix fortune command.

Third-party code

I reused some internals from pkgsite (the Go package index) and adapted them to work with MySQL.

Specifically, I modified the internal/database package to support MySQL, along with the CLI tool in devtools/cmd/db. The other borrowed packages from pkgsite (memory, middleware, wraperr) remain largely unchanged, aside from minor cleanup.

Code

The repo is here: github.com/tetsuo/fortune

Docs cover local setup, development, and deployment.

Cubic Limit in Effect

noemail@noemail.org (Onur Gündüz) — Sat, 8 Feb 2025 14:55:00 GMT

Recreating the iconic artwork of Manfred Mohr the hard way, by rolling my own 3D graphics DSL in TypeScript with Effect.

Shown below is Cubic Limit, P-161 rendered on <canvas>, inspired by Mohr's 1975 plotter drawings.

Source code

The full source code that produces this image is available in the repository tetsuo/cubic-limit.

Under the hood it uses an Effect-based adaptation of graphics-ts (originally fp-ts) with added support for 3D rendering. You can clone it with:

git clone git@github.com:tetsuo/cubic-limit.git

The repository also includes a version of P-197, but this post will focus solely on P-161.

In this post, we'll use this artwork as a starting point and build a functional interface around it for drawing 3D shapes and applying transformations to them.

Cube vertices & edges

We'll work with a type Vec to hold [x, y, z] coordinates.

type Vec = NonEmptyReadonlyArray<number>
// e.g. [x, y, z]

A unit cube (scaled from -1 to +1) has 8 vertices:

const cubePoints: NonEmptyReadonlyArray<Vec> = [
  [-1, -1, -1],
  [ 1, -1, -1],
  [ 1,  1, -1],
  [-1,  1, -1],
  [-1, -1,  1],
  [ 1, -1,  1],
  [ 1,  1,  1],
  [-1,  1,  1],
]

It has 12 edges. We can index them by grouping bottom edges, top edges, and vertical edges. For each i in 0..3, we build three edges:

const getEdges = (i: number): NonEmptyReadonlyArray<Vec> => [
  [i, (i + 1) % 4],
  [i + 4, ((i + 1) % 4) + 4],
  [i, i + 4],
]

Combining all i values yields 12 edges. Each edge is a pair of indices into cubePoints.

Cube configurations

Mohr's piece shows a 31×31 grid of partially drawn cubes. Each cube is unique and displayed with exactly six edges (n = 6). To represent this mathematically:

Label each of the cube's 12 edges as a bit in a 12-bit integer.
Turning on an edge = setting the corresponding bit to 1.

This means each valid cube configuration is a 12-bit number with exactly six bits set to 1.

How many such configurations exist? Exactly 924, as determined by the formula C(12, 6) = 924.

These numbers correspond to OEIS sequence A023688, which lists integers with six 1s in their binary representation. Within the 12-bit range, the sequence starts at 63 (binary 000000111111) and ends just under 4095 (binary 111111111111):

63    -- 000000111111
95    -- 000001011111
111   -- 000001101111
.
.
4032  -- 111111000000

Generating all 12-bit numbers with six 1s

To iterate through all 12-bit integers that have exactly six bits set, I turned to a classic resource: Bit Twiddling Hacks. At the bottom of that page is the code to compute the lexicographically next bit permutation:

import { pipe } from 'effect/Function'

const nextNumber = (v: number) =>
  pipe(
    (v | (v - 1)) + 1,
    t => t | ((((t & -t) / (v & -v)) >> 1) - 1)
  )

This function finds the next integer with the same number of bits set. For example, if v has exactly 6 bits set, it will produce the next integer that also has 6 bits set.

With it, we can now accumulate all valid edge combinations:

import { unfold } from 'effect/Array'
import { none, some } from 'effect/Option'

const seq: number[] = unfold(63, a => a < 4095 ? some([a, nextNumber(a)]) : none()
)
// => [63, 95, 111, 119, ... 4032, etc.]

Each integer in seq encodes a unique cube with exactly 6 edges, totaling 924 configurations.

Representing geometry

Shape

The Shape type is originally defined in the purescript-drawing library:

type Point = { x :: Number, y :: Number }

data Shape
  = Path Boolean (List Point)
  | Composite (List Shape)

For our implementation, we borrow the graphics-ts port of Shape with one critical change: our points will be multi-dimensional. Therefore, the Point type will alias to Vec:

type Point = Vec

const point = (x: number, y: number, z: number): Point => [x, y, z]

So a Shape is a union of Path and Composite, meaning it can represent one of two things. In TypeScript, this translates to:

type Shape = Path | Composite

Path

A Path is a Chunk of points connected by lines, plus a boolean to indicate whether the path is closed (i.e., the last point connects back to the first):

import { Chunk } from '@effect/data/Chunk'

interface Path {
  readonly _tag: 'Path'
  readonly closed: boolean
  readonly points: Chunk<Point>
}

We define two constructors, path and closed, which take a Foldable, an abstraction for collections that can be reduced to a single value (e.g., an Array). We also define a Monoid instance for Path, which combines two paths by concatenating their points and checking if either path is closed.

import { MonoidSome } from '@effect/typeclass/data/Boolean'
import { constant } from 'effect/Function'
import { fromSemigroup, Monoid, struct } from '@effect/typeclass/Monoid'
import { make } from '@effect/typeclass/Semigroup'
import { Foldable } from '@effect/typeclass/Foldable'
import { Kind, TypeLambda } from 'effect/HKT'
import { Chunk, append, appendAll, empty } from '@effect/data/Chunk'

// Constructors

const closed =
  <F extends TypeLambda>(F: Foldable<F>): ((fa: Kind<F, unknown, unknown, unknown, Point>) => Path) =>
  fa =>
    F.reduce(fa, monoidPath.empty, (b, a) => ({
      _tag: 'Path',
      closed: true,
      points: append(b.points, a),
    }))

const path =
  <F extends TypeLambda>(F: Foldable<F>): ((fa: Kind<F, unknown, unknown, unknown, Point>) => Path) =>
  fa =>
    F.reduce(fa, monoidPath.empty, (b, a) => ({
      _tag: 'Path',
      closed: false,
      points: append(b.points, a),
    }))

// Monoid instance

const monoidPath: Monoid<Path> = struct({
  _tag: fromSemigroup<'Path'>(make(constant('Path')), 'Path'),
  closed: MonoidSome,
  points: fromSemigroup(make<Chunk<Point>>(appendAll), empty()),
})

Here, the monoidPath combines two Path objects by merging their point arrays. The MonoidSome ensures that if any of the paths are closed, the result is also considered closed.

Composite

The other variant of Shape is Composite, which basically serves as a container for multiple Shapes:

interface Composite {
  readonly _tag: 'Composite'
  readonly shapes: ReadonlyArray<Shape>
}

const composite = (shapes: ReadonlyArray<Shape>): Composite => ({
  _tag: 'Composite',
  shapes,
})

Constructing a full cube

Bringing it all together, here's how we can define a full cube (all 12 edges visible) as a Composite of 12 paths:

import { pipe } from 'effect/Function'
import { flatMap, map, range } from 'effect/Array'
import { Foldable } from '@effect/typeclass/data/Array'
import { tuple } from 'effect/Data'
import * as S from './Shape'

const path = S.path(Foldable)

const cubeShape = S.composite(
  pipe(
    range(0, 3),
    flatMap(i =>
      pipe(
        getEdges(i),
        map(ix =>
          path(
            pipe(
              tuple(points[ix[0]], points[ix[1]]),
              map(vec => S.point(vec[0], vec[1], vec[2]))
            )
          )
        )
      )
    )
  )
)

Cube as a composite of paths

The cubeShape structure represents the cube's 12 edges as pairs of points in 3D space. Each point is a 3D vector[x, y, z], and each edge is defined by two such points:

[
  [
    [-1, -1, -1], // Start point
    [1, -1, -1],  // End point
  ],
  [
    [-1, -1, 1],
    [1, -1, 1],
  ],
  ... (12 pairs in total)
]

Each innermost array defines a Point, represented by its x, y, and z coordinates.
Each array containing point tuples describes a Path, which corresponds to a single edge of the cube.
12 paths are grouped together into a Composite.

Toggling edges to form partial cubes

Since Mohr's P-161 selectively displays only 6 edges of the cube, we need a way to toggle edges (paths) on/off. We can do that by passing in a predicate that checks if an edge is active.

import { Predicate } from 'effect/Predicate'

const cubeShape = (shouldDrawEdge: Predicate<number>): S.Composite =>
  S.composite(
    pipe(
      range(0, 3),
      flatMapArray(i =>
        pipe(
          getEdges(i),
          mapArray((ix, j) =>
            path(
              shouldDrawEdge(i + j * 4)
                ? pipe(
                    tuple(points[ix[0]], points[ix[1]]),
                    mapArray(vec => S.point(vec[0], vec[1], vec[2]))
                  )
                : [] // ← empty when shouldDrawEdge(edge) returns false
            )
          )
        )
      )
    )
  )

We define a utility isBitSet, returning true if a given bit is set in n:

const isBitSet = (n: number) => (index: number) =>
  Boolean(n & (1 << index))

Then we build a function cubeFromNumber that uses isBitSet to determine which edges should be drawn:

import { flow } from 'effect/Function'

const cubeFromNumber = flow(isBitSet, cubeShape)

If n has bits 3, 5, 8, 10, etc., turned on, only those edges of the cube are visible.

For example, calling cubeFromNumber(63) (binary 000000111111) will omit edges 0 through 5 and include edges 6 through 11, producing a half-formed cube.

Generating all cube shapes

Finally, we can list all 924 cube shapes corresponding to n = 6 by:

Generating all 12-bit numbers with 6 bits set (using unfold and nextNumber).
Mapping each integer to a Shape with cubeFromNumber.

import { none, some } from 'effect/Option'
import { map, unfold } from 'effect/Array'

const cubes: Composite[] = pipe(
  unfold(63, a => (a < 4095 ? some([a, nextNumber(a)]) : none())),
  map(cubeFromNumber)
)

// cubes is now an array of 924 partial cube shapes

Representing styles & transformations

Drawing

Next, we define a new type that captures what we want to do with shapes, whether to fill them, outline them, apply transformations, or clip:

type Drawing =
  | { _tag: 'Translate'; translateX: number; translateY: number; translateZ: number; drawing: Drawing }
  | { _tag: 'Rotate'; rotateX: Angle; rotateY: Angle; rotateZ: Angle; drawing: Drawing }
  | { _tag: 'Scale'; scaleX: number; scaleY: number; scaleZ: number; drawing: Drawing }
  | { _tag: 'Many'; drawings: ReadonlyArray<Drawing> }
  | { _tag: 'Clipped'; shape: Shape; drawing: Drawing }
  | { _tag: 'Fill'; shape: Shape; style: FillStyle }
  | { _tag: 'Outline'; shape: Shape; style: OutlineStyle }

A Drawing acts like a scene graph, where transformations (Scale, Rotate, Translate) can be nested around shapes.

To convert a Shape into a Drawing, we use one of three options: Clipped, Fill, or Outline.

Outline

The Outline variant in Drawing represents shapes rendered with an outline. The function outline constructs an outlined shape with a given OutlineStyle (color, lineWidth, etc.):

const outline: (shape: Shape, style: OutlineStyle) => Drawing = (shape, style) => ({
  _tag: 'Outline',
  shape,
  style,
})

Applying outlines to cubes:

import * as D from './Drawing'

const lineColor = D.outlineColor(white)

const cubeLineStyle = D.monoidOutlineStyle.combine(lineColor, D.lineCap('round'))

const cubes = pipe(
  unfold(63, a => (a < 4095 ? some([a, nextNumber(a)]) : none())),
  map(flow(cubeFromNumber, cube => D.outline(cube, cubeLineStyle)))
)

Fill

Just like Outline defines shapes with an outline, the Fill variant in Drawing represents shapes rendered with a fill. The function fill creates a filled shape using a specified FillStyle.

const fill: (shape: Shape, style: FillStyle) => Drawing = (shape, style) => ({
  _tag: 'Fill',
  shape,
  style,
})

Drawing background:

import { Foldable } from '@effect/typeclass/data/Array'
import { hsl } from './Color'
import * as S from './Shape'
import * as D from './Drawing'

const closed = S.closed(Foldable)

const drawBackground = ({ width, height }: Size, bgColor: Color): D.Drawing =>
  D.fill(
    closed([
      [0, 0, 0],
      [width, 0, 0],
      [width, height, 0],
      [0, height, 0],
    ]),
    D.fillStyle(bgColor)
  )

Many

The Many variant allows combining multiple Drawing objects into a single composition, conceptually just a list.

const many: (drawings: ReadonlyArray<Drawing>) => Drawing = drawings => ({
  _tag: 'Many',
  drawings,
})

Drawing a grid of lines:

import { map, range } from 'effect/Array'
import { white } from './Color'
import * as D from './Drawing'

const lineColor = D.outlineColor(white)

// drawLines creates evenly spaced vertical or horizontal lines by dividing
// the given height into n segments and mapping them into outlined paths.
const drawLines = (n: number, height: number, vertical: boolean): D.Drawing => {
  const size = height / n
  return D.many(
    pipe(
      range(1, n - 1),
      map(i =>
        D.outline(
          path(
            vertical
              ? [
                  [i * size, 0, 0],
                  [i * size, height, 0],
                ]
              : [
                  [0, i * size, 0],
                  [height, i * size, 0],
                ]
          ),
          lineColor
        )
      )
    )
  )
}

Translate/Scale/Rotate

The each of the following constructors wraps a Drawing and includes X, Y, Z parameters to translate, rotate, or scale whatever is inside.

const translate = (
  translateX: number,
  translateY: number,
  translateZ: number,
  drawing: Drawing
): Drawing => ({ _tag: 'Translate', translateX, translateY, translateZ, drawing })

const scale = (
  scaleX: number,
  scaleY: number,
  scaleZ: number,
  drawing: Drawing
): Drawing => ({ _tag: 'Scale', scaleX, scaleY, scaleZ, drawing })

const rotate = (
  rotateX: Angle,
  rotateY: Angle,
  rotateZ: Angle,
  drawing: Drawing
): Drawing => ({ _tag: 'Rotate', rotateX, rotateY, rotateZ, drawing })

Rotating, scaling, and positioning cubes:

import { map, unfold } from 'effect/Array'
import { none, some } from 'effect/Option'
import * as D from './Drawing'
import * as S from './Shape'

// drawCubes generates a grid of outlined cubes.
const drawCubes = (numCells: number, cellSize: number): D.Drawing => {

  // translateCube positions each cube based on its row/column index.
  const translateCube = (drawing: D.Drawing, i: number): D.Drawing => {
    const translateX = cellSize * (numCells - Math.floor(i / numCells) - 0.5)
    const translateY = cellSize * (0.5 + (i % numCells))
    return D.translate(translateX, translateY, 0, drawing)
  }

  // Scale factor compensates for the original cube points being doubled.
  const scaleFactor = cellSize / 5

  return D.rotate(
    S.degrees(30),
    S.degrees(-60),
    S.degrees(0),
    D.scale(
      scaleFactor,
      scaleFactor,
      1, // Z remains unchanged.
      D.many(
        pipe(
          unfold(63, a => (a < 4095 ? some([a, nextNumber(a)]) : none())),
          map(flow(cubeFromNumber, cube => D.outline(cube, cubeLineStyle))),
          map(translateCube)
        )
      )
    )
  )
}

Drawing P-161

Finally, we end up with a Drawing that visually represents Cubic Limit, P-161 in its entirety.

const drawP161 = (size: Size, bgColor: Color): D.Drawing => {
  const numCells = 31
  const cellSize = size.width / numCells

  const background = drawBackground(size, bgColor)

  const lines = D.many([
    drawLines(numCells, size.width, false),
    drawLines(numCells, size.height, true)
  ])

  const cubes = drawCubes(numCells, cellSize)

  return D.many([background, lines, cubes])
}

Manfred Mohr and Estarose Wolfson look at the Benson plotter in the Centre de Calcul de la Météorologie Nationale, 1971 / © Manfred Mohr (Source)

The Cubic Limit series was originally drawn using a Benson 1284 flatbed plotter. Mohr developed his algorithms on a CDC 6400 mainframe with Fortran IV, storing his code on punch cards.

While details about this plotter's proprietary language are scarce, it probably shared similarities with Hewlett Packard's HP-GL. For example, drawing a rectangle in HP-GL might look like this:

SP1          // Select pen 1
PU 0,0       // Pen up (move without drawing)
PD 100,0     // Pen down (start drawing)
PD 100,100
PD 0,100
PD 0,0
PU           // Pen up (stop drawing)

Note that these plotter languages somewhat resemble the Canvas API. For example:

PU (Pen Up) corresponds to moveTo.
PD (Pen Down) corresponds to lineTo.

Here is a basic rectangle-drawing example:

const canvas = document.getElementById('myCanvas');
const ctx = canvas.getContext('2d');

ctx.beginPath();
ctx.moveTo(0, 0);
ctx.lineTo(100, 0);
ctx.lineTo(100, 100);
ctx.lineTo(0, 100);
ctx.closePath();
ctx.stroke();

The Drawing type we defined is essentially an algebraic data type (specifically, a sum type) that describes the semantics of a domain-specific language for vector graphics. It doesn't have its own syntax, nor does it need one.

All it requires is an interpreter to integrate with the Canvas API and, ultimately, generate canvas commands.

Interpreting geometry

That said, before we proceed, we must address a key limitation: the Canvas API only supports 2D coordinates, not 3D. To enable 3D functionality on <canvas>, we'll define Path3D (as an implementation of the CanvasPath interface), much like Path2D but with support for 3D coordinates.

Path3D

type Path3D = ReadonlyArray<ReadonlyArray<Point>>

Path3D is an intermediate representation that transforms high-level Shape definitions into a format more suitable for applying transformations and rendering. Essentially, it flattens a shape into an array of subpaths, where each subpath is a sequence of 3D points ([x, y, z]).

Converting a Shape to Path3D

The helper function fromShape performs this conversion:

const fromShape: (shape: Shape) => Path3D = shape => {
  switch (shape._tag) {
    case 'Composite':
      // For composites, recursively process each child shape and concatenate the results.
      return shape.shapes.flatMap(fromShape)
    case 'Path':
      // For a simple path, convert its points into a subpath.
      return pipe(
        toReadonlyArray(shape.points),
        matchLeft({
          onEmpty: empty,
          onNonEmpty: (head, tail) =>
            pipe(
              tail,
              map(lineTo),
              reduce(moveTo(head)([]), (acc, f) => f(acc)),
              path => (shape.closed ? closePath(path).slice(0, -1) : path)
            ),
        })
      )
  }
}

For a Composite, fromShape simply calls fromShape on each sub-shape and concatenates the resulting Path3D arrays.
For a Path, fromShape breaks the points into line segments (via moveTo and lineTo). If closed is true, it calls closePath, making the last point connect to the first.

Path3D combinators

These functions implement the path-drawing algorithm:

moveTo begins a new subpath at the specified point (if the point is finite).

const moveTo = (point: Point) => (path: Path3D): Path3D =>
  isPointFinite(point)
    ? // Create a new subpath with the specified point.
      append(path, [point])
    : path

lineTo extends the current subpath by connecting the last point to the new point.

const lineTo = (point: Point) => (path: Path3D): Path3D =>
  isPointFinite(point)
    ? isNonEmptyReadonlyArray(path)
      ? // Connect the last point in the subpath to the given point.
        pipe(path, modifyNonEmptyLast(append(point)))
      : // If path has no subpaths, ensure there is a subpath.
        moveTo(point)(path)
    : path

closePath closes the current subpath by connecting its last point back to the first.

const closePath = (path: Path3D): Path3D => {
  // Do nothing if path has no subpaths.
  if (!isNonEmptyReadonlyArray(path)) {
    return path
  }

  const cur = lastNonEmpty(path)

  // Do nothing if the last path contains a single point.
  if (!(isNonEmptyReadonlyArray(cur) && cur.length > 1)) {
    return path
  }

  const end = lastNonEmpty(cur)
  const start = cur[0]

  // Do nothing if both ends are the same point.
  if (end[0] === start[0] && end[1] === start[1] && end[2] === start[2]) {
    return path
  }

  // Mark the last path as closed adding a new subpath whose first point
  // is the same as the previous subpath's first point.
  return append(setNonEmptyLast(append(cur, start) as ReadonlyArray<Point>)(path), [start])
}

Transformation matrices

Transformations are typically encoded in 4×4 matrices when dealing with 3D points. We can define a matrix type Mat as follows:

type Mat = NonEmptyReadonlyArray<Vec>

Common 3D transforms

A standard 4×4 transform matrix looks like this:

[sx   0    0    tx]
[0    sy   0    ty]
[0    0    sz   tz]
[0    0    0    1 ]

sx, sy and sz control scaling on each axis.
tx, ty and tz control translation along each axis.
Off-diagonal terms can encode rotation and other transformations.

For example, a pure scale transformation might look like this:

const scale = (v: Vec): Mat => [
  [v[0], 0,    0,    0],
  [0,    v[1], 0,    0],
  [0,    0,    v[2], 0],
  [0,    0,    0,    1],
]

Likewise, a rotation about the X-axis by some angle can be represented as:

const sin = (deg: number) => Math.sin((deg * Math.PI) / 180)
const cos = (deg: number) => Math.cos((deg * Math.PI) / 180)

const rotateX = (angle: number): Mat => [
  [1, 0,           0,          0],
  [0, cos(angle),  sin(angle), 0],
  [0, -sin(angle), cos(angle), 0],
  [0, 0,           0,          1],
]

Combining multiple transforms

Often we need to combine several transformations (e.g. translate first, then rotate, then scale). In matrix math, combining transformations is done by matrix multiplication:

declare function mul(y: Mat): (x: Mat) => Mat

pipe(
  identity, // The 4×4 identity matrix (no transformation)
  mul(translate([10, 0, 0])),
  mul(rotateX(45)),
  mul(rotateY(30)),
  mul(scale([2, 2, 2]))
)

The final result is a single 4×4 matrix encoding all these operations in the correct order. When you apply that matrix to a point [x, y, z, 1], it performs the entire sequence of transformations-translation, then rotation on X, then rotation on Y, then scaling.

📄 See the full implementation in Mat.ts file.

Reminder: Matrix multiplication is not commutative. The order you multiply matters.

Semigroup & Monoid for matrices

Because matrix multiplication is associative, we can define a Semigroup for Mat:

import { Semigroup, make } from '@effect/typeclass/Semigroup'

// semigroupMat: given two matrices x and y, how do we combine them?
const semigroupMat: Semigroup<Mat> = make((x, y) => {
  // matrix multiplication logic
})

And from that Semigroup, we get a Monoid by adding the identity matrix (which leaves any vector unchanged):

import { Monoid, fromSemigroup } from '@effect/typeclass/Monoid'

const monoidMat: Monoid<Mat> = fromSemigroup(
  semigroupMat,
  identity // 4×4 identity matrix
)

This way, we can compose an array of transformations neatly:

import { monoidMat } from './Mat'

const combined = monoidMat.combineAll([
  translate([5, 0, 0]),
  scale([2, 2, 2]),
  rotateZ(90),
])

Applying 3D transformations

The toCoords helper function takes a Shape, extracts its constituent subpaths, applies a transformation matrix, and outputs the final coordinates to be sent to the Canvas API.

const toCoords = (shape: Shape, transform: Mat): ReadonlyArray<ReadonlyArray<Point>> =>
  pipe(
    // 1. Convert the shape into an array of subpaths.
    fromShape(shape),
    // 2. Attempt to convert each subpath into a NonEmpty array.
    map(validateNonEmpty),
    // 3. Filter out any empty subpaths.
    compactArray,
    // 4. Append a 1 to each coordinate for homogeneous transformation.
    map(map(append(1))),
    // 5. Multiply each coordinate by the transformation matrix.
    map(mul(transform))
  )

Render service

Now that everything is in place, let's define the Render service as a tagged interface that encapsulates methods mirroring those of the native CanvasRenderingContext2D (e.g. lineTo, moveTo, fill, stroke, etc.).

Here's the full definition:

class Render extends Tag('Render')<
  Render,
  {
    readonly lineTo: (point: Vec) => Micro<void>
    readonly moveTo: (point: Vec) => Micro<void>
    readonly fill: (fillRule?: CanvasFillRule) => Micro<void>
    readonly clip: (fillRule?: CanvasFillRule) => Micro<void>
    readonly stroke: () => Micro<void>
    readonly beginPath: () => Micro<void>
    readonly closePath: () => Micro<void>
    readonly save: () => Micro<void>
    readonly restore: () => Micro<void>
    readonly setFillStyle: (style: string) => Micro<void>
    readonly setStrokeStyle: (style: string) => Micro<void>
    readonly setLineWidth: (width: number) => Micro<void>
    readonly setLineCap: (cap: D.LineCap) => Micro<void>
    readonly setLineJoin: (join: D.LineJoin) => Micro<void>
  }
>() {}

Producing render effect

The function renderDrawing is our core interpreter for the Drawing DSL. It takes a Drawing and recursively transforms it into a series of canvas commands, while internally maintaining a transformation matrix that accumulates and applies all transformations.

const renderDrawing = (d: D.Drawing): Micro<void, never, Render> =>
  service(Render).pipe(
    andThen(c => {
      // Wrap an operation with save/restore for context isolation.
      const withContext = (fa: Micro<void>) =>
        pipe(
          c.save(),
          andThen(() => fa),
          andThen(c.restore)
        )

      const applyStyle: <A>(o: Option<A>, f: (a: A) => Micro<void>) => Micro<void> = (fa, f) =>
        isSome(fa) ? f(fa.value) : success

      // Render a single sub-path using moveTo and lineTo.
      const renderSubPath: (subPath: ReadonlyArray<Point>) => Micro<void> = matchLeft({
        onEmpty: () => success,
        onNonEmpty: (head, tail) =>
          pipe(
            c.moveTo(head),
            andThen(() => forEach(tail, c.lineTo, { discard: true }))
          ),
      })

      // Convert a Shape to a Path3D using fromShape, transform it, then render each sub-path.
      const renderShape = (shape: Shape, transform: Mat) =>
        forEach(toCoords(shape, transform), renderSubPath, { discard: true })

      // The recursive interpreter that handles all variants of Drawing.
      const go: (drawing: D.Drawing, transform: Mat) => Micro<void> = (d, t) => {
        switch (d._tag) {
          case 'Many':
            return forEach(d.drawings, d => go(d, t), { discard: true })
          case 'Scale':
            return go(d.drawing, semigroupMat.combine(t, scale([d.scaleX, d.scaleY, d.scaleZ])))
          case 'Rotate':
            return go(
              d.drawing,
              semigroupMat.combineMany(t, [
                rotateZ(angle(d.rotateZ)),
                rotateY(angle(d.rotateY)),
                rotateX(angle(d.rotateX)),
              ])
            )
          case 'Translate':
            return go(
              d.drawing,
              semigroupMat.combine(t, translate([d.translateX, d.translateY, d.translateZ]))
            )
          case 'Outline':
            return withContext(
              pipe(
                applyStyle(d.style.color, flow(Color.toCss, c.setStrokeStyle)),
                andThen(() => applyStyle(d.style.lineWidth, c.setLineWidth)),
                andThen(() => applyStyle(d.style.lineCap, c.setLineCap)),
                andThen(() => applyStyle(d.style.lineJoin, c.setLineJoin)),
                andThen(c.beginPath),
                andThen(() => renderShape(d.shape, t)),
                andThen(c.stroke)
              )
            )
          case 'Fill':
            return withContext(
              pipe(
                applyStyle(d.style.color, flow(Color.toCss, c.setFillStyle)),
                andThen(c.beginPath),
                andThen(() => renderShape(d.shape, t)),
                andThen(() => c.fill())
              )
            )
          case 'Clipped':
            return withContext(
              pipe(
                c.beginPath(),
                andThen(() => renderShape(d.shape, t)),
                andThen(() => c.clip()),
                andThen(() => go(d.drawing, t))
              )
            )
        }
      }
      return go(d, identity)
    })
  )

Transformation composition

The recursive function go walks through a Drawing, updating the transform matrix (with operations like Scale, Rotate, and Translate) as it recurses through nested drawings.

Style and context management

For instructions like Outline and Fill, the function saves the canvas state, applies style settings, begins a new path, renders the shape, executes the appropriate drawing command (stroke or fill), and finally restores the canvas state.

Sub-path rendering

The helper renderSubPath converts a sub-path (a list of 3D points) into the corresponding canvas calls (moveTo for the first point, followed by lineTo for subsequent points).

Wiring up the real canvas

The final step is the render function, which "plugs in" a real CanvasRenderingContext2D implementation by providing a concrete instance of the Render service. This function builds the effect (from renderDrawing) and then supplies a real implementation that calls the actual Canvas API:

const render = (d: D.Drawing, ctx: CanvasRenderingContext2D): Micro<void, never, never> =>
  provideService(renderDrawing(d), Render, {
    fill(fillRule) {
      ctx.fill(fillRule)
      return succeed(undefined)
    },
    clip(fillRule) {
      ctx.clip(fillRule)
      return succeed(undefined)
    },
    setFillStyle(style: string) {
      ctx.fillStyle = style
      return succeed(undefined)
    },
    setStrokeStyle(style: string) {
      ctx.strokeStyle = style
      return succeed(undefined)
    },
    setLineWidth(width: number) {
      ctx.lineWidth = width
      return succeed(undefined)
    },
    setLineJoin(join: D.LineJoin) {
      ctx.lineJoin = join
      return succeed(undefined)
    },
    setLineCap(cap: D.LineCap) {
      ctx.lineCap = cap
      return succeed(undefined)
    },
    stroke() {
      ctx.stroke()
      return succeed(undefined)
    },
    save() {
      ctx.save()
      return succeed(undefined)
    },
    restore() {
      ctx.restore()
      return succeed(undefined)
    },
    lineTo(p) {
      ctx.lineTo(p[0], p[1])
      return succeed(undefined)
    },
    moveTo(p) {
      ctx.moveTo(p[0], p[1])
      return succeed(undefined)
    },
    beginPath() {
      ctx.beginPath()
      return succeed(undefined)
    },
    closePath() {
      ctx.closePath()
      return succeed(undefined)
    },
  })

Putting it all together

Finally, the renderTo function ties everything together. It retrieves the canvas element, adjusts its size for the device pixel ratio, gets the 2D context, and runs the rendering effect:

const renderTo = (f: (size: Size) => Drawing, canvasId: string): void =>
  pipe(
    // 1. Obtain a canvas.
    getCanvasElementById(canvasId),
    andThen(canvas => {
      // 2. Scale the canvas according to the device pixel ratio.
      const rect = canvas.getBoundingClientRect()
      return dpr.pipe(
        andThen(dpr => {
          canvas.width = rect.width * dpr
          canvas.height = rect.height * dpr
          canvas.style.width = `${rect.width}px`
          canvas.style.height = `${rect.height}px`
          return pipe(
            // 3. Get the 2D context.
            getContext2D(canvas),
            andThen(ctx => {
              ctx.scale(dpr, dpr)
              // 4. Compute the Drawing by calling f(size) and render it on the context.
              return render(f({ height: canvas.height / dpr, width: canvas.width / dpr }), ctx)
            })
          )
        })
      )
    })
  ).pipe(runSync) // 5. Perform the side-effect on the canvas.

Rendering P-161

const renderP161 = (canvasId: string, bgColor: string) =>
  renderTo(size => drawP161(size, hex(bgColor)), canvasId)

Triggers and notifications in PostgreSQL

noemail@noemail.org (Onur Gündüz) — Sun, 17 Nov 2024 00:00:00 GMT

Building a pub/sub application with libpq to batch-process account onboarding jobs on a self-managed PostgreSQL-backed job queue.

mailroom uses PostgreSQL triggers and notification events to detect account status changes and send relevant email notifications.

In this post, we'll explore the architecture in detail, starting with the schema and triggers for tracking account updates in a PostgreSQL-backed queue, then building a collector in C with libpq to consume those events and batch-process action tokens.

Overview

The system comprises these actors:

User: Responsible for creating and activating accounts.
Admin: Can suspend accounts (not the main focus though).

Components include:

Accounts: A table for storing users and their lifecycle states.
Tokens: A table for managing activation and recovery tokens.
Triggers: Automate processes like status updates, notifications, and timestamp modifications.

Here's the sequence diagram outlining the workflows:

Accounts table

The accounts table manages user data and tracks account lifecycle states.

CREATE TYPE account_status AS ENUM (
    'provisioned',
    'active',
    'suspended'
);

CREATE TABLE accounts (
    id                  BIGSERIAL PRIMARY KEY,
    email               VARCHAR(254) UNIQUE NOT NULL,
    status              account_status DEFAULT 'provisioned' NOT NULL,
    login               VARCHAR(254) UNIQUE NOT NULL,
    created_at          INTEGER DEFAULT EXTRACT(EPOCH FROM NOW()) NOT NULL,
    status_changed_at   INTEGER,
    activated_at        INTEGER,
    suspended_at        INTEGER,
    unsuspended_at      INTEGER
);

Here, the status field tracks the current state of the account (provisioned, active, or suspended), while timestamps like status_changed_at and activated_at capture important lifecycle events, helping to maintain the status field correctly during transitions and ensuring accurate tracking of account states over time.

Tokens table

The tokens table tracks actionable tokens, such as those used for activation or password recovery.

CREATE TYPE token_action AS ENUM (
    'activation',
    'password_recovery'
);

CREATE TABLE tokens (
    id          BIGSERIAL PRIMARY KEY,
    action      token_action NOT NULL,
    secret      BYTEA DEFAULT gen_random_bytes(32) UNIQUE NOT NULL,
    code        VARCHAR(5) DEFAULT LPAD(TO_CHAR(RANDOM() * 100000, 'FM00000'), 5, '0'),
    account     BIGINT NOT NULL,
    expires_at  INTEGER DEFAULT EXTRACT(EPOCH FROM NOW() + INTERVAL '15 minute') NOT NULL,
    consumed_at INTEGER,
    created_at  INTEGER DEFAULT EXTRACT(EPOCH FROM NOW()) NOT NULL,

    FOREIGN KEY (account) REFERENCES accounts (id) ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED
);

Key columns:

action – Specifies the token type (activation or password recovery).
secret – A unique and secure token string.
code – A short, human-readable security code.
expires_at – Defines the expiration time for tokens, defaulting to 15 minutes.

This table complements the accounts table by managing token-based actions, with relationships maintained through the foreign key account.

Triggers

PostgreSQL triggers allow us to automate processes in response to data changes. Below are the triggers to ensure seamless management of account status transitions, token consumption, and notifications.

1. Before account insert

Event: Before an account is inserted into the accounts table.
Purpose: Automatically creates an activation token when a new account is provisioned.

CREATE OR REPLACE FUNCTION trg_before_account_insert()
RETURNS TRIGGER AS $$
BEGIN
    IF (NEW.status = 'provisioned') THEN
        INSERT INTO
        tokens
            (account, action)
        VALUES
            (NEW.id, 'activation');
    END IF;
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER before_account_insert
    BEFORE INSERT ON accounts
    FOR EACH ROW
    EXECUTE FUNCTION trg_before_account_insert ();

Why not an AFTER trigger?

While it may seem logical to create the token after confirming the account's existence (since the token is ultimately tied to the account), this approach has a critical flaw: if the token insertion fails, we could end up with an account that lacks a corresponding activation token, which would break downstream processes.

The BEFORE trigger ensures that token creation and account insertion are part of the same transaction, guaranteeing the consistency we need. If token creation fails, the entire transaction will be rolled back, preventing the system from entering an invalid state.

This is why the DEFERRABLE INITIALLY DEFERRED constraint is applied to the tokens table. It allows a token to be inserted even before the associated account is created, provided both operations occur within the same transaction.

2. Before account status change

Event: Before an account's status is updated.
Purpose: Updates timestamps for key status changes (e.g., activated, suspended, unsuspended).

CREATE OR REPLACE FUNCTION trg_before_account_status_change ()
    RETURNS TRIGGER
    AS $$
DECLARE
    ts integer := extract(epoch FROM now());
BEGIN
    IF (NEW.status = OLD.status) THEN
        RETURN NEW;
    END IF;

    NEW.status_changed_at = ts;

    IF (NEW.status = 'active') THEN
        IF (OLD.status = 'provisioned') THEN
            NEW.activated_at = ts;
        ELSIF (OLD.status = 'suspended') THEN
            NEW.unsuspended_at = ts;
            NEW.suspended_at = NULL;
            -- Revert status to 'provisioned' if never activated
            IF (OLD.activated_at IS NULL) THEN
              NEW.status = 'provisioned';
            END IF;
        END IF;
    ELSIF (NEW.status = 'suspended') THEN
        NEW.suspended_at = ts;
        NEW.unsuspended_at = NULL;
    END IF;
    RETURN new;
END;
$$
LANGUAGE plpgsql;

CREATE TRIGGER before_account_status_change
    BEFORE UPDATE OF status ON accounts
    FOR EACH ROW
    EXECUTE FUNCTION trg_before_account_status_change ();

3. After token consumed

Event: After a token's consumed_at field in tokens is updated.
Purpose: Activates the associated account when an activation token is consumed.

CREATE OR REPLACE FUNCTION trg_after_token_consumed ()
    RETURNS TRIGGER
    AS $$
BEGIN
    IF (NEW.action != 'activation') THEN
        RETURN NULL;
    END IF;
    -- Activate account
    UPDATE
        accounts
    SET
        status = 'active'
    WHERE
        id = NEW.account
        AND status = 'provisioned';
    RETURN NULL;
END;
$$
LANGUAGE plpgsql;

CREATE TRIGGER after_token_consumed
    AFTER UPDATE OF consumed_at ON tokens
    FOR EACH ROW
    WHEN (NEW.consumed_at IS NOT NULL AND OLD.consumed_at IS NULL)
    EXECUTE FUNCTION trg_after_token_consumed ();

4. After token inserted

Event: After a token is inserted into the tokens table.
Purpose: Notifies external services that a new token has been created.

CREATE OR REPLACE FUNCTION trg_after_token_inserted()
    RETURNS TRIGGER
    LANGUAGE plpgsql
AS $$
BEGIN
    NOTIFY token_insert;
    RETURN NULL;
END;
$$;

CREATE TRIGGER after_token_inserted
    AFTER INSERT ON tokens
    FOR EACH ROW
    EXECUTE FUNCTION trg_after_token_inserted ();

Let's try it out!

Follow these steps to test the triggers and notifications in action:

Setting your environment

(Skip this section if you've already set up the tables and triggers.)

Clone the tetsuo/mailroom repository:

git clone https://github.com/tetsuo/mailroom.git

Run the following command to create a new database in PostgreSQL:

createdb mailroom

Then, navigate to the migrations folder and run:

psql -d mailroom < 0_init.up.sql

Alternatively, you can use go-migrate which is often my preference.

Inspect the initial state

Before adding any data, let's take a look at the initial state of the jobs table:

psql -d mailroom -c "SELECT * FROM jobs;"

You should see one row with job_type set to mailroom and last_seq set to zero:

 job_type | last_seq
----------+----------
 mailroom |        0
(1 row)

Create a new account

Insert a new account into the accounts table. This should automatically generate an activation token.

INSERT INTO accounts (email, login)
    VALUES ('user@example.com', 'user123');

Tip: To insert three records with randomized email and login fields, use the following command:

printf "%.0sINSERT INTO accounts (email, login) VALUES ('user' || md5(random()::text) || '@fake.mail', 'user' || substr(md5(random()::text), 1, 20));\n" {1..3} | \
    psql -d mailroom

Expected outcome:

A new account with status = 'provisioned' is added to accounts.
An activation token is automatically inserted into the tokens table, linked to the account.

Verify:

SELECT * FROM accounts WHERE id = 1;
SELECT * FROM tokens WHERE account = 1;

Here's an example account record:

-[ ACCOUNT 1 ]-------------------------------------------------------------------
id                | 1
email             | usere3213152e8cdf722466a011b1eaa3c98@fake.mail
status            | provisioned
login             | user85341405cb33cbe89a5f
created_at        | 1735709763
status_changed_at |
activated_at      |
suspended_at      |
unsuspended_at    |

The corresponding token record generated by the trigger function:

-[ TOKEN 1 ]---------------------------------------------------------------------
id          | 1
action      | activation
secret      | \x144d3ba23d4e60f80d3cb5cf25783539ba267af34aecd71d7cc888643c912fb7
code        | 06435
account     | 1
expires_at  | 1735710663
consumed_at |
created_at  | 1735709763

Consume the activation token

Simulate token consumption by updating the consumed_at field in the tokens table.

UPDATE
    tokens
SET
    consumed_at = extract(epoch FROM now())
WHERE
    account = 1
    AND action = 'activation';

Expected outcome:

The account's status in accounts should change to active.
The activated_at timestamp should be updated in accounts.

Verify:

SELECT * FROM accounts WHERE id = 1;
SELECT * FROM tokens WHERE account = 1;

Suspend the account

Change the account's status to suspended to test the suspension flow.

UPDATE accounts SET status = 'suspended' WHERE id = 1;

Expected outcome:

The account's suspended_at timestamp is updated.
The unsuspended_at field is cleared.

Verify:

SELECT * FROM accounts WHERE id = 1;

Unsuspend the account

Restore the account's status to active.

UPDATE accounts SET status = 'active' WHERE id = 1;

Expected outcome:

The account's unsuspended_at timestamp is updated.
The suspended_at field is cleared.

Verify:

SELECT * FROM accounts WHERE id = 1;

Observe notifications

Listen for token creation notifications on the token_insert channel using LISTEN:

LISTEN token_insert;

Next, insert some dummy data into the accounts table (or directly into tokens).

Expected outcome:

The LISTEN session should immediately display a notification like:

Asynchronous notification "token_insert" with payload "" received.

psql might need a little nudge (empty ;) to display notifications:

mailroom=# LISTEN token_insert;
LISTEN
mailroom=# ;
Asynchronous notification "token_insert" received from server process with PID 5148.
Asynchronous notification "token_insert" received from server process with PID 5148.
Asynchronous notification "token_insert" received from server process with PID 5148.

These notifications signal that new tokens have arrived—it's time to start processing them.

Jobs table

Now we need to build a mechanism to collect newly added tokens. To do that, we'll define a query that manages their progression through the queue.

We use the jobs table to maintain a cursor that advances through tokens. This table simply tracks the last processed token (last_seq) for each job type:

CREATE TYPE job_type AS ENUM (
    'mailroom'
);

CREATE TABLE jobs (
    job_type job_type PRIMARY KEY,
    last_seq BIGINT
);

Initialize the mailroom queue:

INSERT INTO
jobs
    (last_seq, job_type)
VALUES
    (0, 'mailroom');

Retrieving pending jobs

The following query retrieves relevant job data (tokens and account details), ensuring only valid, unexpired, and unprocessed tokens are selected, with accounts in the correct status for the intended action.

SELECT
    t.account,
    t.secret,
    t.code,
    t.expires_at,
    t.id,
    t.action,
    a.email,
    a.login
FROM
    jobs
    JOIN tokens t
        ON t.id > jobs.last_seq
        AND t.expires_at > EXTRACT(EPOCH FROM NOW())
        AND t.consumed_at IS NULL
        AND t.action IN ('activation', 'password_recovery')
    JOIN accounts a
    ON a.id = t.account
    AND (
      (t.action = 'activation' AND a.status = 'provisioned')
      OR (t.action = 'password_recovery' AND a.status = 'active')
    )
WHERE
    jobs.job_type = 'mailroom'
ORDER BY
    id ASC
LIMIT 10

Joins & filters explained:

Jobs table: We filter for rows where job_type is mailroom.
Tokens table:
- We join tokens with jobs using the condition tokens.id > jobs.last_seq, which ensures we only process tokens that haven't been handled yet.
- We further filter tokens to include only those that are not expired (expires_at is in the future), have not been consumed (consumed_at is NULL), and have an action of either activation or password_recovery.
Accounts table:
- We join accounts on accounts.id = tokens.account.
- For tokens with the activation action, the account must be in the provisioned state.
- For tokens with the password_recovery action, the account must be active.

Dequeueing and advancing the cursor

Next, we integrate this query into a common table expression:

WITH token_data AS (
    -- Insert SELECT query here
),
updated_jobs AS (
  UPDATE
    jobs
  SET
    last_seq = (SELECT MAX(id) FROM token_data)
  WHERE
    EXISTS (SELECT 1 FROM token_data)
  RETURNING last_seq
)
SELECT
  td.action,
  td.email,
  td.login,
  td.secret,
  td.code
FROM
  token_data td

This accomplishes two key tasks:

Retrieves tokens generated after the current last_seq along with the corresponding user data.
Updates the last_seq value to prevent processing the same tokens again.

Output example:

-[ RECORD 1 ]--------------------------------------------------------------
action | activation
email  | usere3213152e8cdf722466a011b1eaa3c98@fake.mail
login  | user85341405cb33cbe89a5f
secret | \x144d3ba23d4e60f80d3cb5cf25783539ba267af34aecd71d7cc888643c912fb7
code   | 06435
-[ RECORD 2 ]--------------------------------------------------------------
action | activation
email  | user41e8b6830c76870594161150051f8215@fake.mail
login  | user2491d87beb8950b4abd7
secret | \x27100e07220b62e849e788e6554fede60c96e967c4aa62db7dc45150c51be23f
code   | 80252
-[ RECORD 3 ]--------------------------------------------------------------
action | activation
email  | user7bb11e235c85afe12076884d06910be4@fake.mail
login  | user91ab8536cb05c37ff46a
secret | \xa9763eec727835bd97b79018b308613268d9ea0db70493fd212771c9b7c3bcb2
code   | 31620

Indexes

To optimize the query performance, the following composite indexes are recommended:

CREATE INDEX accounts_id_status_idx ON accounts (id, status);

CREATE INDEX tokens_id_expires_consumed_action_idx ON tokens
    (id, expires_at, consumed_at, action);

Indexing Strategy:

Equality Conditions First: Since columns used in equality conditions (= or IN) are typically the most selective, they should come first.
Range Conditions Next: Columns used in range conditions (>, <, BETWEEN) should follow.

Collector behavior

Instead of continuously polling the database for new batches, a lightweight service is set up to subscribe to a notification channel, monitor incoming events, and trigger the previously defined job retrieval query when either a certain row limit is reached (based on received notifications) or a timeout occurs.

Here's how the job retrieval and batch execution are controlled:

Batch limit

The maximum number of email destinations in a single batch.

A collector queries the database for at most N tokens at a time (where N is the batch limit). Even if 500 tokens are waiting in the database, the collector will only take, say, 10 at a time. This imposes a hard cap on the throughput of tokens that can leave the database at once.

Batch timeout

The time to wait for accumulating enough notifications to fill a batch.

A collector waits up to X milliseconds before processing incoming notifications (where X is the batch timeout). If fewer than the batch limit have arrived during that period, the collector will still dequeue whatever did arrive, but it won't pull more immediately. In effect, this sets an upper limit on how long new tokens can linger before being handed over to the email sender.

Example

If you set:

A batch timeout of 30 seconds.
A limit of 10 notifications.

This means:

If 10 notifications arrive in quick succession, the batch is triggered immediately.
If fewer than 10 arrive over 30 seconds, the batch is triggered when the timeout ends.

⚠️ Keep in mind that a collector doesn't impose rate limiting; it primarily controls database roundtrips and batch size. A large influx of notifications will keep triggering the batch limit, effectively bypassing the timeout, so the overall token throughput downstream remains largely unaffected.

Implementation details

collector is written in C and interacts with PostgreSQL via libpq. It is responsible solely for collecting jobs and writing them to output in a structured format.

Dequeuing jobs

When the connection is established, collector issues a LISTEN command on the specified channel and creates the prepared statements for subsequent queries.

As notifications arrive, it fetches tokens in batches and writes the results directly to stdout. Processing continues until all queued tokens are exhausted or an error occurs.

📄 See the full implementation in db.c

Batch output structure

The results are output as line-delimited batches, formatted as comma-separated values in the following order:

action,email,username,secret,code

Each batch is represented as a single line, where every row follows this schema:

action – Numeric representation of the email action type (e.g., 1 for activation, 2 for password recovery).
email – Recipient's email address.
username – Recipient's login name.
secret – A base64 URL-encoded string containing the signed token.
code – (Optional) Numeric code (e.g., for password recovery).

Example output

In this example, the first line contains a batch of three jobs, including both password recovery and account activation. The second line contains a single activation job:

2,john.doe123@fakemail.test,johndoe,0WEKrnjY_sTEqogrR6qsp7r7Vg4SQ_0iM_1La5hHp5p31nbkrHUBS0Cz9T24iBDCk6CFqO7tJTihpsOVuHYgLg,35866,1,jane.smith456@notreal.example,janesmith,BfQXx31qfY2IJFTtzAp21IdeW0dDIxUT1Ejf3tYJDukNsfaxxOfldwL-lEfVy4SEkZ_v18rf-EWsvWXH5qgvIg,24735,1,emma.jones789@madeup.mail,emmajones,jxrR5p72UWTQ8JiU2DrqjZ-K8L4t8i454S9NtPkVn4-1-bin3ediP0zHMDQU2J_iIyzH4XmNtzpXZhjV0n5xcA,25416
1,sarah.connor999@unreal.mail,resistance1234,zwhCIthd12DqpQSGB57S9Ky-OXV_8H0e8aHOv_kWoggIuAZ2sc-aQVpIoQ-M--PjwVfdIIxiXkv_WjRjGI57zA,38022

Signing and validating tokens

During the dequeue operation, the token's secret is signed using HMAC-SHA256 and encoded in URL-safe Base64 format. This enables the frontend to verify the authenticity of a token without performing an immediate database lookup.

The encoded secret consists of:

A path name (e.g., /activate or /recover).
The original secret (and code, in the case of recovery).
A cryptographic signature generated from the secret.

static size_t construct_signature_data(char *output, const char *action,
                                       const unsigned char *secret, const char *code)
{
  size_t offset = 0;

  if (strcmp(action, "activation") == 0)
  {
    memcpy(output, "/activate", 9); // "/activate" is 9 bytes
    offset = 9;
    memcpy(output + offset, secret, 32);
    offset += 32;
  }
  else if (strcmp(action, "password_recovery") == 0)
  {
    memcpy(output, "/recover", 8); // "/recover" is 8 bytes
    offset = 8;
    memcpy(output + offset, secret, 32);
    offset += 32;
    memcpy(output + offset, code, 5); // code is 5 bytes
    offset += 5;
  }

  return offset; // Total length of the constructed data
}

ℹ️ If you'd like to see how verification works on the backend, check out verifyHmac.js in the repo.

Security considerations

Handle expired tokens properly. One approach is to include expires_at in the payload so expiration can be checked without a DB call. For stronger protection, cache consumed tokens until they naturally expire to prevent reuse within their validity window.
Regularly rotate your signing key.

Main loop logic

At a high level, the main loop continuously:

🔄 Dequeues and processes ready batches
📩 Checks for new notifications
⏳ Waits on select() for database activity or a timeout
🩺 Performs periodic health checks
🔌 Reconnects to the database if needed

When the batch limit is reached or the timeout expires, the collector executes the dequeue query. If a broken connection is detected, it attempts to reconnect and resume processing once stable.

Pseudo-code representation:

// 🌟 Main processing loop
WHILE the application is running 🔄
    // 🔌 Handle reconnection if needed
    IF the connection is not ready ❌ THEN
        reconnect to the database 🔄
        initialize the connection ✅
        reset counters 🔢
        CONTINUE to the next iteration ⏩
    END IF

    // 📦 Process ready batches
    IF ready for processing ✅ THEN
        dequeue and process a batch of items 📤
        reset state for the next cycle 🔁
        CONTINUE to the next iteration ⏩
    END IF

    // 🛎️ Handle pending notifications
    process all incoming notifications 📥
    IF notifications exceed the batch limit 🚨 THEN
        mark ready for processing ✅
        CONTINUE to the next iteration ⏩
    END IF

    // ⏱️ Wait for new events or timeout
    wait for activity on the connection 📡 or timeout ⌛
    IF interrupted by a signal 🚨 THEN
        handle the signal (e.g., shutdown) ❌
        CONTINUE to the next iteration ⏩
    ELSE IF timeout occurs ⏳ THEN
        IF notifications exist 📋 THEN
            mark ready for processing ✅
            CONTINUE to the next iteration ⏩
        END IF
        perform periodic health checks 🩺
    END IF

    // 🛠️ Consume available data
    consume data from the connection 📶
    prepare for the next cycle 🔁
END WHILE

📄 See the full implementation in main.c

The select() system call plays a key role in the program's operation. It is a UNIX mechanism that monitors file descriptors (e.g., sockets) to check if they are ready for I/O operations like reading or writing. In this code, select() is used to monitor the socket for new notifications and enforce a timeout for batch processing.

Once select() signals that data is available, PQconsumeInput is called to read incoming data into libpq's internal buffers. Then PQnotifies is invoked to retrieve any pending notifications and update the counter.

📖 Learn more about libpq's async API

Compile & run

To compile, verify that you have openssl@3 and libpq@5 installed, then use the provided Makefile.

Run the collector

Use the following command to build and run the collector executable with example configuration variables:

make && \
  MAILROOM_BATCH_LIMIT=3 \
  MAILROOM_BATCH_TIMEOUT=5000 \
  MAILROOM_DATABASE_URL="dbname=mailroom" \
  MAILROOM_SECRET_KEY="cafebabecafebabecafebabecafebabecafebabecafebabecafebabecafebabe" \
  ./collector

Refer to the README file for the full list of environment variables

Once configured and started, it will log its activity:

2024/04/20 13:37:00 [PG] configured; channel=token_insert queue=mailroom limit=3 timeout=5000ms healthcheck-interval=270000ms
2024/04/20 13:37:00 [PG] connecting to host=/tmp port=5432 dbname=mailroom user=ogu sslmode=disable
2024/04/20 13:37:00 [PG] connected

Insert accounts and observe batching

In another terminal, insert 5 accounts:

printf "%.0sINSERT INTO accounts (email, login) VALUES ('user' || md5(random()::text) || '@fake.mail', 'user' || substr(md5(random()::text), 1, 20));\n" {1..5} | \
    psql -d mailroom

You'll observe that collector immediately process the first batch of three items. After a 5-second delay (as defined by the MAILROOM_BATCH_TIMEOUT), it processes the remaining two in a second batch:

2024/04/20 13:37:00 [PG] NOTIFY called; waking up
2024/04/20 13:37:00 [PG] processing 3 rows... (max reached)
1,userb183abb7a25d04027061e6b8d8d8e7fa@fake.mail,userb0bf075b82b892f53d97,gVRNesi-opSvs3ntPfr9DzSn_JwbOD04VVIurQSCOFzzd3BOM3WBDL3SOtDjMxKLd6csSn8_p9hemXHIUxIjPg,78092,1,user43b01ba9686c886473e526429dd2c672@fake.mail,userf420078dba4fd5a91de2,--DTy5LsbDeLP_AweXIPSjL3_avQMT5cH_bRxPy1uxQLVhXKaw7Oxd7NYkcJ6MZmnnqWqTcBPHA5z7bqunXEAA,25778,1,user46f81dfd34b91a1904ac4524193575aa@fake.mail,user6d91baab56d2823b326d,ryooWewe3OTxIGF1Gjl5Vvl8BsXoqWVbCAt1t6J--_KX1SM4DbyCes4yn75OWVe60G4MMZdv4byRh1wy-Clvxw,78202
2024/04/20 13:37:00 [PG] NOTIFY called; waking up
2024/04/20 13:37:05 [PG] processing 2 rows... (timeout)
1,user12d2722e1c07b0a531ea69ae125d4697@fake.mail,user853ae29eefc5d44a6bc6,4pmew2o2EOAZBDHWvJBcixJftpRCb8uyXZhzN12EOcrLBmzc4ic9avwd9dla09pIiKIoqW5iIwMfoXLEM3_LGw,38806,1,user9497d0e033019fcf3198eecb053ba40e@fake.mail,userfcde338dba96cc419613,ANLMa-1y37VLCDqK0wnfEFhUVzHsWpaNGV2ttI8m3o6_lbbYOKmp3hP7Q8H8ZQRNMPAj4xsSqC26nesfVZLgzQ,89897

Testing reconnect behavior

To simulate a dropped connection, open another terminal, connect to the mailroom db via psql, and run:

SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE
  datname = 'mailroom'
  AND pid <> pg_backend_pid();

After the connection is killed, select() wakes up, causing PQconsumeInput() to fail with an error. The process logs a reconnect attempt, and once reconnected, it resumes processing without losing track of queued tokens during the downtime.

2024/04/20 13:37:42 [PG] WARN: error consuming input: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

2024/04/20 13:37:42 [PG] connecting to host=/tmp port=5432 dbname=mailroom user=ogu sslmode=disable
2024/04/20 13:37:42 [PG] connected

Further improvements

Building on this foundation, you can extend your triggers to support more complex workflows and fine-tune the collector to operate under stricter constraints, all while keeping the database at the core of event processing.

Before we conclude, let's explore some potential improvements, starting with support for multiple worker setups.

1. Multiple workers

When you update last_seq, PostgreSQL locks the jobs row being updated, preventing other processes from modifying it until the transaction is complete. However, PostgreSQL does not prevent multiple processes from attempting to read the same cursor before one updates it. This can lead to duplicate processing if you're not careful.

If there's any chance of concurrent execution, using FOR UPDATE is essential:

...
    FROM
        jobs
        -- Lock the `jobs` record to prevent concurrent access
        FOR UPDATE
        JOIN tokens t ON t.id > jobs.last_seq
...

Without locking:

Consumer A reads jobs.last_seq = 100.
Consumer B also reads jobs.last_seq = 100 before A updates it.
Both consumers select tokens where t.id > 100, potentially processing the same tokens.

With FOR UPDATE:

Consumer A locks the jobs record and reads last_seq = 100.
Consumer B tries to read jobs.last_seq but is blocked until Consumer A's transaction completes.
Consumer A updates last_seq to, say, 150 and releases the lock.
Consumer B then reads the updated last_seq = 150, processing the next set of tokens.

Alternatively, to efficiently handle multiple consumers, you might consider eliminating the jobs table altogether. Instead, add a new field, such as processed_at, to the tokens table. This field will indicate when a token has been processed. By updating processed_at during token retrieval, you can use FOR UPDATE SKIP LOCKED to support a multi-consumer setup in a safe fashion.

However, if you're certain that only a single consumer runs this query at any given time, I recommend sticking with the jobs table as a single point of reference. This approach avoids the need for complex locking mechanisms, and you can further enhance the jobs table to keep a history of job executions, parameters, and statuses, which can be valuable for auditing purposes.

2. Priority queues

The current queueing mechanism processes tokens without distinguishing between their types and lacks the ability to prioritize critical ones, such as password recovery, over less urgent emails like account activations. At present, '10 emails per second' could mean 10 emails of the same type or a mix, depending on the batch. While effective, this design leaves room for improvement, such as introducing prioritization or smarter batching strategies.

3. Adaptive batching

User activity is rarely consistent—there are bursts of high traffic that may far exceed daily or hourly quotas, followed by periods of minimal activity. Rather than using fixed limits and timeouts, batch size and timeout values can be dynamically adjusted based on real-time conditions. During low-traffic periods, the batch size can be increased to improve efficiency. During peak hours, it can be reduced to minimize delays.

What's next?

⏭ While not covered in this post, a Rust-based AWS SES email sender is available in the sender folder of the repository. It consumes the collector's output to send bulk emails.

Making sense of Haskell

noemail@noemail.org (Onur Gündüz) — Tue, 30 May 2023 12:41:00 GMT

An attempt at a beginner-friendly explanation of denotational semantics and how it maps to Haskell code 😬

What is a function?

Functions in Haskell are pure, meaning they're fixed mappings from inputs to outputs with no side effects like performing I/O or throwing exceptions.

Here is an example demonstrating the Fibonacci sequence:

fib :: Integer -> Integer
fib 0 = 0
fib 1 = 1
fib n = fib (n-1) + fib (n-2)

This defines fib as a function that takes an integer n and returns the nth Fibonacci number. Each match corresponds to an equation defining the function's meaning in terms of input-output relationships.

This view of a function is called denotational, in contrast to operational semantics, where functions are sequences of operations executed over time, like in imperative languages.

Denotational semantics

Denotational semantics is powerful because it turns programs into mathematical objects in some semantic domain, be it numbers, functions, sets, or even stranger things.

Formally, a language's meaning is defined by its semantic function, which maps program syntax to a chosen semantic domain. Think of it as a box ⟦⟧ where you place a syntactic expression inside and get back its value in that domain. For example:

⟦E⟧ : V

means the expression E is assigned a value in the semantic domain V (which could be numbers, functions, etc.).

Example: Calculator

Consider arithmetic expressions written in prefix notation:

⟦add 1 2⟧ = 1 + 2
⟦mul 2 5⟧ = 2 × 5
     ⟦42⟧ = 42
 ⟦neg 42⟧ = -42

We can define the abstract syntax for these expressions using Backus-Naur Form (BNF):

n ∈ Int ::= ... | -1 | 0 | 1 | 2 | ...
e ∈ Exp ::= add e e
         |  mul e e
         |  neg e
         |  n

Here, every expression evaluates to an integer, so the semantic domain is ℤ (the set of all integers). In Haskell, we might denote this with:

type ℤ = Integer

-- | An expression representing a numeral structure.
data Exp = Lit ℤ         -- Literal integer
         | Neg Exp       -- Negation of an expression
         | Add Exp Exp   -- Addition of two expressions
         | Mul Exp Exp   -- Multiplication of two expressions

The valuation function then assigns a mathematical meaning to each expression:

      ⟦Exp⟧ : ℤ
⟦add e1 e2⟧ = ⟦e1⟧ + ⟦e2⟧
⟦mul e1 e2⟧ = ⟦e1⟧ × ⟦e2⟧
    ⟦neg e⟧ = -⟦e⟧
        ⟦n⟧ = n

or, in Haskell:

eval :: Exp -> ℤ
eval (Lit n)     = n                   -- ⟦n⟧ = n
eval (Neg e)     = - (eval e)          -- ⟦neg e⟧ = -⟦e⟧
eval (Add e1 e2) = eval e1 + eval e2   -- ⟦add e1 e2⟧ = ⟦e1⟧ + ⟦e2⟧
eval (Mul e1 e2) = eval e1 * eval e2   -- ⟦mul e1 e2⟧ = ⟦e1⟧ × ⟦e2⟧

Move language

Consider Move, a made-up DSL for controlling a robot.

The Move language specifies commands such as go E 3, which instruct a robot to move a given number of steps in a specified direction:

go E 3; go N 4; go S 1;

Each go command constructs a Step, representing an n-unit movement in one of the cardinal directions.
A Move is a sequence of steps separated by semicolons.

The abstract syntax for the Move language might be defined as:

n ∈ Nat  ::= 0 | 1 | 2 | ...
d ∈ Dir  ::= N | S | E | W
s ∈ Step ::= go d n
m ∈ Move ::= ε | s ; m

We can explore two interpretations (semantic domains) for Move programs:

1. Total distance calculation

In this interpretation, the semantic domain is ℕ (the natural numbers), representing the total distance traveled.

For Step expressions:

  S⟦Step⟧ : Nat
S⟦go d k⟧ = k

For Move expressions:

  M⟦Move⟧ : Nat
   M⟦ε⟧ = 0
 M⟦s;m⟧ = S⟦s⟧ + M⟦m⟧

2. Target position calculation

Here, the semantic domain is the set of functions that map a starting position (x, y) to a final position. We denote this using lambda calculus (λ-calculus):

⟦Expr⟧ : Pos → Pos

For each Step:

  S⟦Step⟧ : Pos → Pos
S⟦go N k⟧ = λ(x,y).(x,y+k)
S⟦go S k⟧ = λ(x,y).(x,y−k)
S⟦go E k⟧ = λ(x,y).(x+k,y)
S⟦go W k⟧ = λ(x,y).(x−k,y)

A Move expression composes these functions in sequence. For an empty Move, the function simply returns the starting position:

M⟦Move⟧ : Pos → Pos
   M⟦ε⟧ = λp.p
 M⟦s;m⟧ = M⟦m⟧ ∘ S⟦s⟧

Implementing Move in Haskell

Here's the thing, the Move DSL can be implemented directly in Haskell by mirroring its BNF grammar with algebraic data types and defining its semantics as pure functions.

-- Abstract syntax
data Dir = N | S | E | W

data Step = Go Dir Int

data Move
  = Empty
  | Seq Step Move

-- Semantics: total distance
stepDist :: Step -> Int
stepDist (Go _ k) = k

moveDist :: Move -> Int
moveDist Empty       = 0
moveDist (Seq s m)   = stepDist s + moveDist m

-- Semantics: final position
type Pos = (Int, Int)

stepPos :: Step -> Pos -> Pos
stepPos (Go N k) (x, y) = (x    , y + k)
stepPos (Go S k) (x, y) = (x    , y - k)
stepPos (Go E k) (x, y) = (x + k, y    )
stepPos (Go W k) (x, y) = (x - k, y    )

movePos :: Move -> Pos -> Pos
movePos Empty       p = p
movePos (Seq s m)   p = movePos m (stepPos s p)

To test it, save the code as Move.hs and run ghci Move.hs.

Then define a program:

let prog = Seq (Go E 3) (Seq (Go N 4) (Seq (Go S 1) Empty))

Total distance traveled:

moveDist prog
-- 8

Final position from the origin:

movePos prog (0, 0)
-- (3, 3)

Final position from an arbitrary point:

movePos prog (10, -2)
-- (13, 1)

Building a full-stack web app with fp-ts

noemail@noemail.org (Onur Gündüz) — Wed, 25 Jan 2023 12:41:00 GMT

In this post, we take a quick tour of flixbox, a movie trailer search app built entirely with fp-ts and libraries from the fp-ts community.

Server

The flixbox server is powered by hyper-ts, which is a partial porting of Hyper.

Internally, a set of middlewares is defined like get, put, movie, and results for interacting with the TMDb API and managing caching. These functions are arranged into pipelines that can short-circuit on failure, handling things like input validation, TMDb errors, or missing resources.

Example: Movie middleware

The following middleware handles /movie/ID requests.

When /movie/3423 is called:

🗂️ Check the internal cache:
- ✅ Return cached data if available.
- 🔄 Otherwise, fetch data from TMDb, store it in the cache, and return the result.
📦 Respond with a JSON object.

import * as H from 'hyper-ts/lib/Middleware'

// ...

function getMovieMiddleware(
  tmdb: TMDb,
  store: Storage<Document>
): (route: MovieRoute) => H.Middleware<StatusOpen, ResponseEnded, AppError, void> {
  return route =>
    pipe(
      GET, // ensure this is a GET request
      H.apSecond(
        pipe(
          // try to get the movie from the cache
          get(store, `/movies/${String(route.id)}`),
          H.map(entry => entry.value),
          // if not found, fetch from TMDb and cache it
          H.orElse(() =>
            pipe(
              movie(tmdb, route.id),
              H.chain(value =>
                pipe(
                  put(store, `/movies/${String(route.id)}`, value),
                  H.map(entry => entry.value)
                )
              )
            )
          )
        )
      ),
      // respond with JSON
      H.ichain(res =>
        pipe(
          H.status<AppError>(200),
          H.ichain(() => sendJSON(res))
        )
      )
    )
}

📄 See the full implementation in server/Flixbox.ts.

Shared modules

On both client and server, flixbox utilizes a set of common modules including:

logging-ts for structured logging
io-ts for runtime type validation
monocle-ts for optics support
fp-ts-routing for declarative route parsing

Among these, io-ts is especially valuable for robust type validation throughout the application, with applications such as:

io-ts is highly recommended even for projects that do not fully adopt fp-ts.

Extending fp-ts modules to new effect types

While these modules integrate smoothly within the fp-ts v2 ecosystem, certain scenarios may require explicit type-level configuration.

For instance, when using logging-ts with a custom effect type, you must provide an instance of the Logger algebra that conforms to that effect. logging-ts facilitates this by exposing getLoggerM, which abstracts over any monad.

To integrate logging-ts with the effects flixbox generates, a new HKT LoggerTaskEither is defined and registered in fp-ts's URItoKind2, thereby allowing type class instance support for logging within the TaskEither context, the most frequently used effect type in the project.

Client

The client uses elm-ts, which provides an fp-ts adaptation of Elm.

Elm shares conceptual similarities with Redux. Messages in Elm correspond to Redux actions, and the Elm update function closely mirrors Redux reducers, responsible for state changes.

Here are the message types used in the flixbox UI:

type Msg =
  | Navigate
  | PushUrl
  | UpdateSearchTerm
  | SubmitSearch
  | SetHttpError
  | SetNotification
  | SetSearchResults
  | SetPopularResults
  | SetMovie

Elm architecture in a nutshell

📄 Initial state: You define an initial application state, the model.
🖼️ View function: A view function renders visual elements based on the current state.
🔁 Update function: When user interaction triggers a message (e.g., clicking a link triggers a Navigate message), the update function is called.
- 📥 It receives the message and the current state as its inputs.
- ⚙️ It processes the current state and returns a new state and potentially a new message.
🔄 State updates: The new state is sent to subscribers (like the view function).
🌀 Continuous processing: New actions are processed until no further actions remain.

📄 See the full implementation in app/Effect.ts.

Optics for state updates

The client also uses monocle-ts, a port of Monocle, allowing composable structures like Lens and Traversal for state updates without mutations.

Consider the following comparison to Immer.js:

Immer.js example:

import produce from "immer"

const toggleTodo = produce((draft, id) => {
    const todo = draft.find(todo => todo.id === id)
    todo.done = !todo.done
})

const nextState = toggleTodo(baseState, "Immer")

monocle-ts equivalent:

import * as _ from 'monocle-ts/lib/Traversal'

type Todo = { id: number; done: boolean }

type Todos = ReadonlyArray<Todo>

const toggleTodoDone = (id: number) =>
  pipe(
    _.id<Todos>(),
    _.findFirst(todo => todo.id === id),
    _.prop('done'),
    _.modify(done => !done)
  )

const nextState = toggleTodoDone(42)(baseState)

CouchDB design document bundler

noemail@noemail.org (Onur Gündüz) — Mon, 2 Jan 2023 14:55:00 GMT

couchilla is a bundler for packing design documents for CouchDB with CommonJS support.

Overview

In CouchDB, design documents are special database entries that contain JavaScript functions, such as view and update functions. These functions, executed on demand, generate secondary indexes, often termed MapReduce views.

JavaScript support in CouchDB is based on the Mozilla SpiderMonkey engine (and, starting with version 3.4.1, also QuickJS). However, because design functions are language-independent, CouchDB does not include a dedicated tool for creating them.

That's where couchilla comes in. It scans a directory of JavaScript files containing view and filter functions, then builds a JSON design document that's ready to deploy.

Example directory layout

For instance, your directory might look like this:

.
├── filters
│   └── quu.js
├── views
│   ├── foo.map.js
│   └── bar.reduce.js
└── validate_doc_update.js

View functions reside in the views directory. Files with .map.js (or simply .js) are converted into map functions.
- Reduce functions are defined in files with .reduce.js extensions.
Filter functions belong in the filters directory.

View functions

Map functions

Emit key/value pairs to store them in a view.

views/foo.map.js

export default doc => emit(doc._id, 42)

Reduce functions

Take sum of mapped values:

views/foo.reduce.js

export default (keys, values, rereduce) => {
  if (rereduce) {
    return sum(values)
  } else {
    return values.length
  }
}

Builtin reduce functions

You can opt to use Erlang native functions using the builtin annotation. For example the sum function above can be rewritten using _sum.

views/foo.reduce.js

/* builtin _sum */

During compilation this will be replaced with a call to the builtin _sum function.

Filter functions

Filter by field:

filters/foo.js

export default (doc, req) => {
  if (doc && doc.title && doc.title.startsWith('C')) {
    return true
  }
  return false
}

Validate document update functions

Log incoming requests and respond with forbidden:

export default (newDoc, oldDoc, userCtx, secObj) => {
  log(newDoc)
  log(oldDoc)
  log(userCtx)
  log(secObj)
  throw { forbidden: 'not able now!' }
}

Requiring other modules

All code, including require() statements, must be enclosed within the exported default function.

views/gamma.map.js

export default doc => {
  const gamma = require('gamma')

  emit(doc._id, gamma(doc.value))
}