Now, what's Lisp, and why would I want to do that? Well, let me tell you a little story about that.

Several decades ago, I got a New Year's present from my roommate's girlfriend. It was a book. She got it at a used bookstore in the UK. She wasn't sure what to get me, so she turned for advice to the store clerk. He asked her what kind of person I was, and she aptly described me as "a guy who loves math and mutters to himself". The clerk, without batting an eyelash, handed her a book with a yellow bird on its cover and told her I would like it.

He turned out to be right. I did like the book. It was written by Raymond M. Smullyan, and its title was "To Mock a Mockingbird".

By that time, I was already familiar with the author. I had read his books before. They were collections of logic puzzles, set in the now-popular Knights-and-Knaves world of his own invention. "To Mock a Mockingbird", at first glance, was the same. Its first two chapters were collections of logic puzzles as well. The puzzles weren't too easy, nor were they too hard. Each of them would make you flex your brain just the right amount. Back then, I used to commute a lot every day, and these puzzles turned out to be perfect time killers during long commutes. It took me something like two weeks to get through the first two chapters.

The third chapter, as is typical for the genre, started with a preface explaining the setup. Instead of knights and knaves, this chapter had a forest inhabited by magic birds. The birds are programmed to call out each other's names. When, say, a finch hears "parrot", it would (always) respond with "jay". When a parrot would hear "jay", it would say "crane", and so on, and so forth. All the birds have to say something when they hear another bird's name. This chain never stops.

The very first puzzle introduced a couple of conditions (which didn't look like much), and asked to prove some facts about the birds.

At first, I thought it would be the same old, same old. It wasn't. I couldn't solve it. Not in five minutes, not in an hour, not in a day.

I lost sleep for a week.

I had been reading a lot of math puzzle books, so I sort of knew there were math concepts behind the fantasy setup. It was obviously some kind of logic. I was pretty good with regular, first-order logic. It was easy. There's data: numbers and facts. There are functions, or formulas, which operate on the data. The two never mix. You can add two numbers together, or you can combine two theorems to prove a new one. But you can't "subtract" the Pythagorean theorem from the law of the excluded middle, or "multiply" Bézout's lemma by two; it doesn't make sense. You know exactly what's what.

With those damned birds, all the bets were off. If it's logic, are the birds data or functions in that logic? Remember the finch. It hears "parrot", it says "jay". It maps one to the other. When something maps things to things, it's a function, so "parrot" and "jay" must be data. Ok, parrots are data. But at the same time, the parrot maps "jay" to "crane", so the parrot is a function. So now the birds are functions? And the data? Should we put their name in quotes when they're data? Double quotes when they're a function again? It never stops. You think you've put your finger on it, and then it eludes you again.

It wasn't just that I couldn't find the solution; I didn't even know where to start looking. I spent (I believe) two months of my life pondering this problem. And then, in a surge of inspiration, I just saw the solution, clear as day. I still remember where I was at that moment.

The book had promised that the solution would be "extremely ingenious". It was. To this day, the fact that I found it without looking it up fills me with pride. I won't spoil it for you. I'll leave finding the book and going through the same experience as an exercise for the curious reader.

That exact feeling came back to me several times after that.

I had that feeling when I started learning programming and came across the concept of first-class functions. Store functions as if they were data? Call a variable as if it were a function? What a weird idea at first! And what a brilliant one, when you start thinking about it.

I had it again when I came across what turned out to be the theory behind the bird puzzles: the lambda calculus and its evil twin, the combinatory logic. It takes the idea of combining data and functions to the extreme: There's no data at all, only functions. There's no starting point, nor the stopping one; it's the functions all the way down. And yet, if you're clever enough, you can build arithmetic, algebra, and programming languages using only these functions that have no beginning and no end.

And then I had it again, almost as a déjà vu, when I first came across Lisp.

What is Lisp?

Lisp is a really old programming language (or, rather, a family of languages). It was invented by John McCarthy in 1958. Some would even say it was "discovered" rather than invented. It wasn't even supposed to be a programming language. It was a clever way to describe computations.

Back then, people would describe computer languages in terms of Turing machines. A Turing machine is a set of rules, telling the computer how to move from one state to another, depending on what's in memory and CPU registers. It's powerful, but really tedious. McCarthy didn't like it, so he invented a notation of his own, called S-expressions. A typical S-expression looks like this:

((defn factorial [n]
  (reduce * (range 1 (inc n)))) 5)

This expression is a set of nested lists, each list defined by a pair of parentheses: ( ). All dialects of Lisp look like someone dropped a bucket of parens. Lists and symbols within these lists can denote either data or computations on data. Computations serve to simplify, or shrink, the expressions. E.g. (+ 1 2) can be simplified to 3, and (reduce * (1 2 3 4 5)) could be simplified to 120. How exactly it's done is left up to the language designer. Thinking of language design in terms of "how do I go from (+ 1 2) to 3?" was supposed to be much easier than in terms of manipulating, saving, and restoring register states.

McCarthy wanted to prove that S-expressions were easier than Turing machines. To this end, he wrote an algorithm to "evaluate" every S-expression and simplify it to the bare minimum. Such an algorithm is called a "universal function". Of course, he described this algorithm in what he thought was the best way to describe any algorithm: using S-expressions. McCarthy called his algorithm "eval". As expected, it came out very neat and concise. This algorithm was only intended for publication. In McCarthy's own words:

Another way to show that LISP was neater than Turing machines was to write a universal LISP function and show that it is briefer and more comprehensible than the description of a universal Turing machine. This was the LISP function eval[e,a], which computes the value of a LISP expression e - the second argument a being a list of assignments of values to variables. (a is needed to make the recursion work). Writing eval required inventing a notation representing LISP functions as LISP data, and such a notation was devised for the purposes of the paper with no thought that it would be used to express LISP programs in practice.

To his surprise, one of his students, Steve Russell, offered to implement this "eval" algorithm, and promptly did it. Thus, the first Lisp "evaluator" was born. You could feed it Lisp expressions, and it would compute their values.

If you fed the evaluator something like (* 10 50), it would simplify it to 500. If you fed it (defun fac (n) (if (= n 1) 1 (* n (fac (- n 1))))), it would simplify it to a function that computes factorials. And if you fed the evaluator the expression for "eval", it would simplify it to a function capable of processing Lisp expressions — a working evaluator.

There is no beginning and no end to this endless chain. Lisp expressions turn into data, which turns into expressions, and so on, and so forth, with such ease and grace that few other languages can dream of. Just like Smullyan's birds from "To Mock the Mockingbird". Back in the day, it took me some time to notice the resemblance, but when I did, it hit me hard.

I was fascinated with Lisp ever since.

If you've been programming computers for an even remotely extended period of time, chances are you've heard this spiel already. Apparently, Lisp is supposed to make you a better programmer. Lisp can make you rich. It's supposed to be the best programming language ever designed.

You might be thinking that I'm one of those weird people who are trying to get others on the Lisp bandwagon.

You're right. I think every computer programmer should learn Lisp. In case I wasn't clear enough: If you haven't written a single Lisp program in your life, go and do it.

SQL implementation of Lisp

A quick reminder for the new readers. Every New Year's Eve, I publish an article where I do what I think is some cool stuff in SQL. I use SQL to draw ray-traced 3D pictures, play music, and implement working Large Language Models. I do this all in pure SQL, as a single query (usually in PostgreSQL), with no user-defined functions and no use of procedural languages.

This year will be no exception. I'm about to write a single SQL query that will take a Lisp expression (as a string parameter), evaluate it, and return the result of the evaluation, also as a string.

Lisp is a family of languages. To implement it in SQL, I will need to pick a dialect. There are many available, but I've picked a really cool a simple one, called Mal. Mal, which stands for "Make A Lisp", is a toy Lisp dialect, specifically designed as a learning tool for aspiring compiler hackers. It has a ton of implementations in other languages and comes with a manual that guides you through all the steps of implementing a Lisp interpreter.

Lisp programs consist of nested expressions, which lend themselves easily to evaluation in recursive functions. Pure SQL, however, doesn't have a concept of recursive functions (or any kind of functions, for that matter), so we don't have this luxury. We will need to implement our own "virtual machine" of sorts to be able to evaluate Lisp expressions.

Here's how it's gonna work.

SQL doesn't have recursive functions, but it does have recursive CTEs. They can be used to emulate a loop. Because loops are all that we have, we'll need to do something that John McCarthy loathed so much, and implement a Turing machine.

Each iteration of the recursive CTE will return exactly one row. This row will contain the whole state of the machine. It will keep the call stack, the heap, and the registers, each in a separate field.

On every invocation of the recursive part of the CTE, it will look at the state of the stack, heap, and the registers, and "manipulate" them according to some hardcoded rules. Queries don't have the concept of memory, so we cannot really "write" anything on the stack or heap. Every invocation will need to copy the full value of the stack and the heap, and apply the rules to make the changes. As is often the case with abusing SQL for the purposes of general programming, it's not going to be fast or efficient.

PostgreSQL has first-class support for JSON values, and a very rich set of functions and operators to manipulate them. It also has a native datatype for efficient storage of JSON data, called JSONB. We'll use it to store our virtual machine's data.

For the stack, we're going to use a single JSONB array, with nested arrays serving as stack frames. Regular programming languages, of course, use the stack as well. On a CPU, the stack is usually flat, and stack frames are separated with special instructions, if at all. Here, we can explicitly mark frame boundaries with JSONB arrays.

We will encode Lisp structures in JSONB. It's an almost one-to-one mapping. Lisp lists will become JSONB arrays; symbols, numbers, booleans, and nils will become JSONB strings, numbers, booleans, and nulls, respectively; all other datatypes will be stored as JSONB objects with the properties t and v (for "type" and "value", respectively).

The heap will be a flat JSONB array. Our implementation doesn't have garbage collection, so we never remove anything from the heap, only add.

Functions and macros will be stored by value and copied between the stack and the heap when accessed. It's a little bit wasteful, but easier to implement. Every function has a reference to the parent environment, a list of parameters (as a list of strings), and its body (the AST expression).

The rules for transforming the stack, the heap, and the registers will be defined according to the value of the last stack frame. Every stack frame is always an array. Its first value will define a special form, a function, or a built-in operator; the rest are the parameters and intermediate variables.

Functions on the stack frames can return values by appending them to the previous stack frame and removing themselves from the stack. This is not unlike how functions work on a regular CPU, except that we won't be doing repeated popping and pushing, and will do all the replacements in a single operation, in one fell swoop.

Stack frames won't have a clear distinction between parameters, variables, and values returned by functions down the stack. This is intended because it will allow some elegant optimizations.

Here's a brief outline of what the virtual machine would do:

As you can see, on each step, the query mechanically transforms the stack and other fields, according to some internal rules, which we will hardcode in SQL.

Print

Let's start writing our query by implementing print. It's a very simple operation: it should put its first argument to the field output, remove itself from the stack, and pass the value null as a result of its operation. (In Lisp, all expressions return some kind of result).

We'll write a simple CTE query that will look at the first value of the last stack frame, and treat it as an "opcode" to our virtual machine.

To do this, we will add some boilerplate code that will come in handy later:

1. We'll implement every opcode as a separate subquery, which we'll call in a LEFT JOIN LATERAL, and use the value of the opcode as a join condition.
2. Technically, each subquery only needs the value of the stack. However, there are some pieces of the stack that the opcode processors will need more often. This includes the value of the first item on the last stack frame (the opcode itself), and everything on the last stack frame except the opcode (it would be arguments/variables/return values from subsequent calls). It makes sense to expose them explicitly as fields, so the opcode queries down the line could access them more easily.
3. There are several ways we can manipulate the stack. For now, we can only return values and remove the current frame. This is what most functions do anyway. To make the implementation easier, we will expose the return value as a separate field from every opcode query that does return a value.
4. There will be some improvements to the boilerplate code further down the line

And here's our print processor:

WITH    RECURSIVE
        loop (stack, output, step) AS
        (
        SELECT  '[["print"], ["print"], ["print", {"t": "str", "v": "Hello world"}]]'::JSONB, NULL::TEXT, 0
        UNION ALL
        SELECT  new.new_stack, new.new_output, step + 1
        FROM    loop
        -- The subquery below exposes the value of the opcode and arguments for easier use
        CROSS JOIN LATERAL (SELECT stack #>> '{-1, 0}', JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]')) q_call (opcode, args)
        -- This query will only return something if the first value of the last frame on the stack equals to the string "print".
        -- We can say that it will process the opcode "print"
        LEFT JOIN LATERAL
                (
                SELECT  'null'::JSONB ret,
                        CASE WHEN args #>> '{0, t}' = 'str' THEN args #>> '{0, v}' ELSE args[0]::TEXT END new_output
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        -- For now, we only have one opcode processor, but in the future, there will be many.
        -- They are mutually exclusive (all the join conditions are different).
        -- If any of them returned anything, the COALESCE will take care of that.
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(print.ret),
                        COALESCE(print.new_output)
                ) rets (ret, new_output)
        -- If we have a return value, the expression below will append in to the previous stack frame,
        -- and remove the current one from the stack. This is how function returns work in most languages
        CROSS JOIN LATERAL
                (
                SELECT  JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE) AS new_stack,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT  *
FROM    loop

Here's what the processor does on each step:

1. Call the subquery that's responsible for the current opcode (we only have one for now, print)
2. Look at the value in its ret field
3. Append this value to the previous stack frame
4. Remove the last stack frame from the stack

The result is the same as if we called (print(print(print("Hello world"))) in a regular programming language. The innermost print prints "Hello world", and the outermost ones print the results returned to them by the "prints" deeper down (that is, NULL values).

Eval

Now, we'll make a really simple expression evaluator. With some changes (that we'll make later), we can use it as a calculator that works in reverse Polish notation.

The algorithm is as follows:

The opcode for it will be called #eval. A hash symbol in front of the opcode means its arguments need to be sanitized and can't come directly from the user-provided input.

1. If the first argument is not a list, it's returned as is.
2. If the first argument is an empty list, and there are no further arguments, it's returned as is.
3. If the first argument is a non-empty list, its first member is put for further evaluation (passed into a new #eval down the stack), and removed from the list.
4. If the first argument is an empty list, and there are some further arguments, then it's a function call, and needs to be "applied" (that's Lisp-speak for executing a function). Put it on the stack as is, replacing the current frame.

Let's make a print-eval stack and run it up until the point where it would need to make the actual calculations (it can't do them yet):

WITH    RECURSIVE
        loop (stack, output, step) AS
        (
        SELECT  '[["print"], ["#eval", ["+", 1, 2]]]'::JSONB, NULL::TEXT, 0
        UNION ALL
        SELECT  new.new_stack, new.new_output, step + 1
        FROM    loop
        CROSS JOIN LATERAL (SELECT stack[-1], stack #>> '{-1, 0}', JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]')) q_call (current_frame, opcode, args)
        LEFT JOIN LATERAL
                (
                SELECT  'null'::JSONB ret,
                        CASE WHEN args #>> '{0, t}' = 'str' THEN args #>> '{0, v}' ELSE args[0]::TEXT END new_output
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        LEFT JOIN LATERAL
                (
                -- Our first argument is not a list or an empty list with no follow-up. Return it as is
                SELECT  CASE WHEN calls IS NULL THEN eval_ast END AS ret,
                        calls
                FROM    (SELECT args['0'], JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]'), current_frame) q (eval_ast, eval_ret)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(eval_ast) = 'array') q_array (ast_is_array)
                CROSS JOIN LATERAL (SELECT CASE WHEN ast_is_array THEN JSONB_ARRAY_LENGTH(eval_ast) = 0 ELSE FALSE END, JSONB_ARRAY_LENGTH(eval_ret) = 0) q_empty (ast_array_empty, ret_empty)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                -- Our first argument is a non-empty list. Remove the first argument from that list and pass it to a new #eval.
                                -- [..., ["#eval", ["+", 1, 2]]] -> [..., ["#eval", [1, 2]], ["eval", "+"]]
                                WHEN ast_is_array AND NOT ast_array_empty THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#eval', eval_ast[0]))
                                -- Our first argument is an empty list, and we have something after it.
                                -- It means that whatever was there, has already been evaluated and needs to be applied.
                                -- We just take the evaluated list and put it on the stack as a new frame, replacing the current eval
                                -- [..., ["#eval", [], "+", 1, 2]] -> [..., ["+", 1, 2]]
                                WHEN ast_array_empty AND NOT ret_empty THEN JSONB_BUILD_ARRAY(eval_ret)
                                END
                        ) q_calls (calls)
                ) eval ON (opcode = '#eval')
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(print.ret, eval.ret),
                        COALESCE(eval.calls) AS calls,
                        COALESCE(print.new_output)
                ) rets (ret, calls, new_output)
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(
                                JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE),
                                (stack - (-1)) || COALESCE(rets.calls, '[]'::JSONB)) AS new_stack,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT  *
FROM    loop
WHERE   step <= 7

By that time, our evaluator has run up until the point where it's about to execute the addition operator on the arguments 1 and 2. We don't have code for it yet. Let's add it!

Addition

The plus sign will serve as an opcode. We'll need to add a hard-coded processor for it in SQL code. It's gonna be a one-liner.

There's a trick to that, however. The way we're organizing our code, putting each opcode into its own LEFT JOIN subquery, has a downside. PostgreSQL query planner works in such a way that all LEFT JOIN LATERAL queries are evaluated first, and only then is the join condition applied. All the join conditions don't depend on the results of the queries at all; it would make sense to evaluate them first, and then, if they come out to be false, not run the query at all. This is, unfortunately, not how PostgreSQL works.

This behavior has two implications for us.

First, it's a little bit of a waste of resources: all the queries get evaluated regardless of the value of the opcode, and their results are discarded.

Second, some SQL expressions, like casting non-numeric JSONB values to numbers, will fail at runtime if these values can't be safely cast. Since all opcode queries are always executed, the arguments to every stack frame will try to be cast to numbers, even if this stack frame has nothing to do with the numeric addition.

For this reason, we need to make sure all type casts and other potentially unsafe operations won't throw SQL-level exceptions. To this end, we need to wrap them in COALESCE or CASE expressions.

It's not ideal, but all the alternatives I could think of look even uglier, so this solution is the lesser evil.

The query below will evaluate an AST that comes from the expression (+ 1 2):

WITH    RECURSIVE
        loop (stack, output, step) AS
        (
        SELECT  '[["print"], ["#eval", ["+", 1, 2]]]'::JSONB, NULL::TEXT, 0
        UNION ALL
        SELECT  new.new_stack, new.new_output, step + 1
        FROM    loop
        CROSS JOIN LATERAL (SELECT stack[-1], stack #>> '{-1, 0}', JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]')) q_call (current_frame, opcode, args)
        LEFT JOIN LATERAL
                (
                SELECT  'null'::JSONB ret,
                        CASE WHEN args #>> '{0, t}' = 'str' THEN args #>> '{0, v}' ELSE args[0]::TEXT END new_output
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN calls IS NULL THEN eval_ast END AS ret, calls
                FROM    (SELECT args['0'], JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]'), current_frame) q (eval_ast, eval_ret)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(eval_ast) = 'array') q_array (ast_is_array)
                CROSS JOIN LATERAL (SELECT CASE WHEN ast_is_array THEN JSONB_ARRAY_LENGTH(eval_ast) = 0 ELSE FALSE END, JSONB_ARRAY_LENGTH(eval_ret) = 0) q_empty (ast_array_empty, ret_empty)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                WHEN ast_is_array AND NOT ast_array_empty THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#eval', eval_ast[0]))
                                WHEN ast_array_empty AND NOT ret_empty THEN JSONB_BUILD_ARRAY(eval_ret)
                                END
                        ) q_calls (calls)
                ) eval ON (opcode = '#eval')

        -- This part is responsible for calculating addition

        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN JSONB_TYPEOF(args[0]) = 'number' AND JSONB_TYPEOF(args[1]) = 'number' THEN
                        TO_JSONB(args[0]::DOUBLE PRECISION + args[1]::DOUBLE PRECISION) END AS ret
                ) plus ON (opcode = '+')

        --

        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(print.ret, eval.ret, plus.ret),
                        COALESCE(eval.calls) AS calls,
                        COALESCE(print.new_output)
                ) rets (ret, calls, new_output)
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(
                                JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE),
                                (stack - (-1)) || COALESCE(rets.calls, '[]'::JSONB)) AS new_stack,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT  *
FROM    loop

We just built an expression evaluator. For now, it can only do addition, but that's already something.

Let's try to feed it a more complex expression, like (+ (+ 1 2) 3):

WITH    RECURSIVE
        loop (stack, output, step) AS
        (
        SELECT  '[["print"], ["#eval", ["+", ["+", 1, 2], 3]]]'::JSONB, NULL::TEXT, 0
        UNION ALL
        SELECT  new.new_stack, new.new_output, step + 1
        FROM    loop
        CROSS JOIN LATERAL (SELECT stack[-1], stack #>> '{-1, 0}', JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]')) q_call (current_frame, opcode, args)
        LEFT JOIN LATERAL
                (
                SELECT  'null'::JSONB ret,
                        CASE WHEN args #>> '{0, t}' = 'str' THEN args #>> '{0, v}' ELSE args[0]::TEXT END new_output
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN calls IS NULL THEN eval_ast END AS ret, calls
                FROM    (SELECT args['0'], JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]'), current_frame) q (eval_ast, eval_ret)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(eval_ast) = 'array') q_array (ast_is_array)
                CROSS JOIN LATERAL (SELECT CASE WHEN ast_is_array THEN JSONB_ARRAY_LENGTH(eval_ast) = 0 ELSE FALSE END, JSONB_ARRAY_LENGTH(eval_ret) = 0) q_empty (ast_array_empty, ret_empty)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                WHEN ast_is_array AND NOT ast_array_empty THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#eval', eval_ast[0]))
                                WHEN ast_array_empty AND NOT ret_empty THEN JSONB_BUILD_ARRAY(eval_ret)
                                END
                        ) q_calls (calls)
                ) eval ON (opcode = '#eval')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN JSONB_TYPEOF(args[0]) = 'number' AND JSONB_TYPEOF(args[1]) = 'number' THEN
                        TO_JSONB(args[0]::DOUBLE PRECISION + args[1]::DOUBLE PRECISION) END AS ret
                ) plus ON (opcode = '+')
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(print.ret, eval.ret, plus.ret),
                        COALESCE(eval.calls) AS calls,
                        COALESCE(print.new_output)
                ) rets (ret, calls, new_output)
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(
                                JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE),
                                (stack - (-1)) || COALESCE(rets.calls, '[]'::JSONB)) AS new_stack,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT  *
FROM    loop

Not only can it do addition, but it can also nest addition expressions. Nice!

Variables and environments

Programming languages need to store data and reference it later. Lisp is no exception. Like most other languages, it has a concept of variables.

Lisp variables live in a special data structure called an "environment". It's a key-value map, which can store values keyed by (usually) a string. In addition, environments are nested; that is, every environment can have a parent environment.

When eval comes across a symbol it doesn't know about, it assumes it's a variable and tries to evaluate it by looking up its value. Every eval is accompanied by a reference to an environment. At first, eval tries to find the value in the key-value map of the current environment, using the symbol as a key. If it can't find it in the current environment, it looks it up in the parent of the current environment, then in the grandparent, and so on, until it reaches the topmost environment. If the value still can't be found, the virtual machine throws a runtime exception.

On JavaScript

As a side note, the notion of environments turned out to be extremely useful to help implement programming languages with lexical scope. Take JavaScript. It was heavily influenced by Lisp (to the point that some people even call it "Lisp in C's clothing").

In JavaScript, you can create a function within another function. The innermost function will be able to see and even change the variables declared in the outermost function. It works because the names and values of the variables live in a special hidden object called "scope" that is implicitly created when a function runs. The child function saves ("captures") to its definition when it's created, and uses it as a parent for its own scope when it's run. When JavaScript looks up a variable, it traverses the chain of scopes all the way up to the topmost scope until it finds the variable.

If you look closely, you can see that JavaScript scopes are Lisp environments in a very thin disguise.

Environments in SQL Mal

To support environments in the SQL implementation, we'll make some non-trivial additions to our query.

1. Introduce two more fields: heap and env.
   • heap will be a JSONB array. It will store data that needs to stay around for a long time, regardless of what's on the stack.
   • env will be an integer field. It will store the reference to the current environment. The environment itself will live in the heap, and its reference will be just the index to its position in the JSONB array.

   Our toy implementation of Mal doesn't use garbage collection, so we never delete anything from the heap.

2. Add a new opcode, #get-env. It will treat its first argument as a symbol, look it its value recursively in the current environment, and return it, if found.
3. To support the perfectly expected situation that a variable is not found in the environment, we'll add another opcode, #throw. For now, it will accept one argument, a string. When executed, this opcode will print the string and empty the stack, so that the program terminates right away. This will require some more boilerplate code.
4. Change the eval logic. When it comes across a symbol (that we encode as JSONB strings), and it's not a built-in function (like the addition operator +), it will put an instruction on the stack to look this symbol up.

Since eval (and, potentially, any opcode) now needs a reference to an existing environment, we'll need to add at least one environment to the heap on startup. We'll do it manually. An environment is just a two-element array: the reference to the parent, and the key-value map (a JSONB object).

We'll set the variable four to the value 4, and will try to evaluate the expression (+ four four):

WITH    RECURSIVE
        constants AS
        (
        -- These are the symbols that we won't be looking up during evaluation, they will always evaluate to themselves
        SELECT  ARRAY['+'] AS builtin_functions
        ),
        loop (heap, stack, env, output, step) AS
        (
        -- We add the variable four = 4 to the default environment
        SELECT  '[[null, {"four": 4}]]'::JSONB, '[["print"], ["#eval", ["+", "four", "four"]]]'::JSONB, 0, NULL::TEXT, 0
        UNION ALL
        SELECT  new.new_heap, new.new_stack, new.new_env, new.new_output, step + 1
        FROM    loop
        CROSS JOIN constants
        CROSS JOIN LATERAL (SELECT stack[-1], stack #>> '{-1, 0}', JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]')) q_call (current_frame, opcode, args)
        LEFT JOIN LATERAL
                (
                SELECT  'null'::JSONB ret,
                        CASE WHEN args #>> '{0, t}' = 'str' THEN args #>> '{0, v}' ELSE args[0]::TEXT END new_output
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN calls IS NULL THEN eval_ast END AS ret, calls
                FROM    (SELECT args['0'], JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]'), current_frame) q (eval_ast, eval_ret)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(eval_ast) = 'array', JSONB_TYPEOF(eval_ast) = 'string') q_array (ast_is_array, ast_is_symbol)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE WHEN ast_is_array THEN JSONB_ARRAY_LENGTH(eval_ast) = 0 ELSE FALSE END,
                                JSONB_ARRAY_LENGTH(eval_ret) = 0
                        ) q_empty (ast_array_empty, ret_empty)
                CROSS JOIN LATERAL (SELECT COALESCE(eval_ast #>> '{}' = ANY(builtin_functions), FALSE)) q_callable (is_builtin_function)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                WHEN ast_is_array AND NOT ast_array_empty THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#eval', eval_ast[0]))
                                WHEN ast_array_empty AND NOT ret_empty THEN JSONB_BUILD_ARRAY(eval_ret)
                                -- If we are evaluating a symbol that is not a built-in function, we need to look it up
                                WHEN ast_is_symbol AND NOT is_builtin_function THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#get-env', eval_ast))
                                END
                        ) q_calls (calls)
                ) eval ON (opcode = '#eval')
        LEFT JOIN LATERAL
                (
                SELECT  value AS ret,
                        CASE WHEN value IS NULL THEN FORMAT('Variable %s not found', key) END AS throws
                FROM    (SELECT args ->> 0) AS q_key (key)
                LEFT JOIN LATERAL
                        (
                        -- This is a nested recursive subquery (PostgreSQL supports those, which is extremely nice)
                        -- It follows the chain of environments and tries to look it the value of its first argument in each of them
                        WITH    RECURSIVE get (current_env) AS
                                (
                                SELECT  heap -> env
                                UNION ALL
                                SELECT  heap -> (current_env ->> 0)::INT
                                FROM    get
                                WHERE   current_env IS NOT NULL
                                )
                        SELECT  vals -> key
                        FROM    get
                        CROSS JOIN LATERAL (SELECT current_env -> 1) q (vals)
                        WHERE   vals ? key
                        LIMIT   1
                        ) AS env_value (value) ON TRUE
                ) get_env ON (opcode = '#get-env')

        -- This is the processor for the opcode "#throw". It rewrites the stack.

        LEFT JOIN LATERAL (SELECT args #>> '{0}' AS new_output, '[]'::JSONB AS new_stack) throw ON (opcode = 'throw')

        --

        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN JSONB_TYPEOF(args[0]) = 'number' AND JSONB_TYPEOF(args[1]) = 'number' THEN
                        TO_JSONB(args[0]::DOUBLE PRECISION + args[1]::DOUBLE PRECISION) END AS ret
                ) plus ON (opcode = '+')
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(heap),
                        COALESCE(env),
                        COALESCE(print.new_output, throw.new_output),
                        COALESCE(throw.new_stack),
                        COALESCE(print.ret, eval.ret, plus.ret, get_env.ret),
                        COALESCE(eval.calls) AS calls,
                        COALESCE(get_env.throws) AS throws
                ) rets (new_heap, new_env, new_output, new_stack, ret, calls, throws)
        CROSS JOIN LATERAL
                (
                SELECT  rets.new_heap,
                        COALESCE(
                                -- If an opcode decides to overwrite the stack, it trumps throws, returns, and calls
                                rets.new_stack,
                                -- We have a new field, throws, that can come from opcodes. It overrides returns and calls
                                CASE WHEN rets.throws IS NOT NULL THEN stack || JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('throw', rets.throws)) END,
                                JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE),
                                (stack - (-1)) || COALESCE(rets.calls, '[]'::JSONB)) AS new_stack,
                        rets.new_env,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT  *
FROM    loop

It came out just right.

What if we try to evaluate (+ five five)? Remember that the symbol five is undefined:

WITH    RECURSIVE
        constants AS
        (
        SELECT  ARRAY['+'] AS builtin_functions
        ),
        loop (heap, stack, env, output, step) AS
        (
        SELECT  '[[null, {"four": 4}]]'::JSONB, '[["print"], ["#eval", ["+", "five", "five"]]]'::JSONB, 0, NULL::TEXT, 0
        UNION ALL
        SELECT  new.new_heap, new.new_stack, new.new_env, new.new_output, step + 1
        FROM    loop
        CROSS JOIN constants
        CROSS JOIN LATERAL (SELECT stack[-1], stack #>> '{-1, 0}', JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]')) q_call (current_frame, opcode, args)
        LEFT JOIN LATERAL
                (
                SELECT  'null'::JSONB ret,
                        CASE WHEN args #>> '{0, t}' = 'str' THEN args #>> '{0, v}' ELSE args[0]::TEXT END new_output
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN calls IS NULL THEN eval_ast END AS ret, calls
                FROM    (SELECT args['0'], JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]'), current_frame) q (eval_ast, eval_ret)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(eval_ast) = 'array', JSONB_TYPEOF(eval_ast) = 'string') q_array (ast_is_array, ast_is_symbol)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE WHEN ast_is_array THEN JSONB_ARRAY_LENGTH(eval_ast) = 0 ELSE FALSE END,
                                JSONB_ARRAY_LENGTH(eval_ret) = 0
                        ) q_empty (ast_array_empty, ret_empty)
                CROSS JOIN LATERAL (SELECT COALESCE(eval_ast #>> '{}' = ANY(builtin_functions), FALSE)) q_callable (is_builtin_function)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                WHEN ast_is_array AND NOT ast_array_empty THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#eval', eval_ast[0]))
                                WHEN ast_array_empty AND NOT ret_empty THEN JSONB_BUILD_ARRAY(eval_ret)
                                -- If we are evaluating a symbol that is not a built-in function, we need to look it up
                                WHEN ast_is_symbol AND NOT is_builtin_function THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#get-env', eval_ast))
                                END
                        ) q_calls (calls)
                ) eval ON (opcode = '#eval')
        LEFT JOIN LATERAL
                (
                SELECT  value AS ret,
                        CASE WHEN value IS NULL THEN FORMAT('Variable %s not found', key) END AS throws
                FROM    (SELECT args ->> 0) AS q_key (key)
                LEFT JOIN LATERAL
                        (
                        -- This is a nested recursive subquery (PostgreSQL supports those, which is extremely nice)
                        -- It follows the chain of environments and tries to look it the value of its first argument in each of them
                        WITH    RECURSIVE get (current_env) AS
                                (
                                SELECT  heap -> env
                                UNION ALL
                                SELECT  heap -> (current_env ->> 0)::INT
                                FROM    get
                                WHERE   current_env IS NOT NULL
                                )
                        SELECT  vals -> key
                        FROM    get
                        CROSS JOIN LATERAL (SELECT current_env -> 1) q (vals)
                        WHERE   vals ? key
                        LIMIT   1
                        ) AS env_value (value) ON TRUE
                ) get_env ON (opcode = '#get-env')
        LEFT JOIN LATERAL (SELECT args #>> '{0}' AS new_output, '[]'::JSONB AS new_stack) throw ON (opcode = 'throw')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN JSONB_TYPEOF(args[0]) = 'number' AND JSONB_TYPEOF(args[1]) = 'number' THEN
                        TO_JSONB(args[0]::DOUBLE PRECISION + args[1]::DOUBLE PRECISION) END AS ret
                ) plus ON (opcode = '+')
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(heap),
                        COALESCE(env),
                        COALESCE(print.new_output, throw.new_output),
                        COALESCE(throw.new_stack),
                        COALESCE(print.ret, eval.ret, plus.ret, get_env.ret),
                        COALESCE(eval.calls) AS calls,
                        COALESCE(get_env.throws) AS throws
                ) rets (new_heap, new_env, new_output, new_stack, ret, calls, throws)
        CROSS JOIN LATERAL
                (
                SELECT  rets.new_heap,
                        COALESCE(
                                -- If an opcode decides to overwrite the stack, it trumps throws, returns, and calls
                                rets.new_stack,
                                -- We have a new field, throws, that can come from opcodes. It overrides returns and calls
                                CASE WHEN rets.throws IS NOT NULL THEN stack || JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('throw', rets.throws)) END,
                                JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE),
                                (stack - (-1)) || COALESCE(rets.calls, '[]'::JSONB)) AS new_stack,
                        rets.new_env,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT  *
FROM    loop

We have an exception, which is printed on the program output.

Special forms

In Lisp, expressions (pieces of data) that can be fed to eval without an error are called "forms".

By default, eval works by recursively evaluating all items in the forms, and then, if it's a list, calculate its value by "applying" the first element of the list (as a function) to the rest of them (as its arguments).

Such evaluations are powerful, but their power is limited. In particular, lazy evaluation of expressions (and, hence, recursion) is impossible in this setup: all arguments to every function need to be evaluated before they can be provided to the function.

To solve this problem, Lisps introduce the concept of "special forms".

Special forms are lists that start with a particular symbol, which can be thought of as a keyword. As the name suggests, they have special evaluation rules. In particular, members of special forms can be evaluated in a different order than they normally are, or not at all. This opens a whole new range of possibilities.

Here are the special forms that exist in Mal:

• (def! var value) — set the symbol var (unevaluated) to value (evaluated) in the current environment.

  Analog of var / let / const in JavaScript.

• (fn* (arg1 arg2 …) form) — Create a function, capture the current environment.

  Analog of function in JavaScript.

• (let (arg1 value1 arg2 value2 …) form)
  1. Create a new environment
  2. Evaluate value1, and set it to arg1 in the new environment
  3. Evaluate value2 (with arg1 already defined), and set it to arg2
  4. Repeat until the list is exhausted. Finally, evaluate form in the new environment

  Analog of defining variables within an anonymous block in JavaScript: { let b = 2; }

• (do form1 form2) — Evaluate all the forms in the list one by one, left to right, and return the value of the last evaluation.

  Analog of statements in JavaScript.

• (if cond form-yes form-no) — Evaluate cond. If it evaluates to a truthy value (anything other than false or nil), evaluate and return form-yes; else, evaluate and return form-no.

  Analog of if / else in JavaScript.

• (defmacro! var form) — Define a macro. Macro is a special type of function that generates AST (rather than a value) and then evaluates it.

  No direct analog in JavaScript.

• (quote ast) — Don't evaluate AST; return it as is.

  No direct analog in JavaScript.

• (quasiquote ast) — Don't evaluate AST; return it as is, unless marked.

  No direct analog in JavaScript.

• (unquote form) — Evaluate form, when inside a quasiquote. Normally, forms within quasiquote are not evaluated.

  (quasiquote a (unquote (+ 1 2)) b) will evaluate to (a 3 b)

• (splice-unquote form) — Evaluate form, when inside a quasiquote. If it returns a list, splice it into the AST.

  (quasiquote a (splice-unquote ((+ 1 2) (+ 2 3))) b) will evaluate to (a 3 5 b)

• (try* form (catch* exception-var handler)) — catch exceptions.

  Analog of the try/catch construct in JavaScript

Functions

Of all the special forms, fn*, the special form that creates functions, is of greatest interest, because, obviously, functions are what make the computers tick.

We will represent functions as JSONB objects of the following shape:

{"t": "fn*", "v": [env, [arg1, arg2, …], ast]}

, where:

• env is the environment (integer) which was active at the time the function was created
• [arg1, arg2, …] is a list of symbols that will be set to function arguments when the function gets applied (executed)
• ast is the code of the function

In our implementation, functions are value types rather than reference types, and they won't be put on the heap when created. It will mean, among other things, that two different functions that happen to be created in the same environment with the same list of parameters and the same AST will be treated as equal. I don't think it will be much of a problem.

The list of arguments and the function body are, in fact, already present on the AST in the same exact shape that they will be in the function value. So, to create a function, we just need to append them to the value of env, and wrap all this into an object.

As I said before, fn* is a special form. We'll need to add support for special forms to our eval code. That is, if we're evaluating a list and the first element in this list happens to be a special form symbol, we'll put this list on the stack as is and let the opcode processors handle it.

Of course, we'll need to add an opcode handler for fn*. This opcode comes straight from the AST, so we'll need to validate it: just make sure that the first argument is a list of symbols. Mal specification says that only two arguments matter for fn*, but it's not a problem if there are more: even if they end up on the stack, we'll just ignore them.

Let's try to create a function that doubles its argument. We'll do it by evaluating the expression (fn* (a) (+ a a))

WITH    RECURSIVE
        constants AS
        (
        SELECT  ARRAY['+'] AS builtin_functions,
                -- We're keeping a list of special forms here
                ARRAY['fn*'] AS special_forms,
                -- We'll be using it quite a lot, so we'll add a constant for it and save us several dozens of keystrokes.
                'null'::JSONB AS c_null
        ),
        loop (heap, stack, env, output, step) AS
        (
        SELECT  '[[null, {}]]'::JSONB, '[["print"], ["#eval", ["fn*", ["a"], ["+", "a", "a"]]]]'::JSONB, 0, NULL::TEXT, 0
        UNION ALL
        SELECT  new.new_heap, new.new_stack, new.new_env, new.new_output, step + 1
        FROM    loop
        CROSS JOIN constants
        CROSS JOIN LATERAL (SELECT stack[-1], stack #>> '{-1, 0}', JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]')) q_call (current_frame, opcode, args)
        LEFT JOIN LATERAL
                (
                SELECT  'null'::JSONB ret,
                        CASE WHEN args #>> '{0, t}' = 'str' THEN args #>> '{0, v}' ELSE args[0]::TEXT END new_output
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN calls IS NULL THEN eval_ast END AS ret, calls
                FROM    (SELECT args['0'], JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]'), current_frame) q (eval_ast, eval_ret)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(eval_ast) = 'array', JSONB_TYPEOF(eval_ast) = 'string') q_array (ast_is_array, ast_is_symbol)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE WHEN ast_is_array THEN JSONB_ARRAY_LENGTH(eval_ast) = 0 ELSE FALSE END,
                                JSONB_ARRAY_LENGTH(eval_ret) = 0
                        ) q_empty (ast_array_empty, ret_empty)
                CROSS JOIN LATERAL
                        (
                        SELECT  COALESCE(eval_ast #>> '{0}' = ANY(special_forms), FALSE),
                                COALESCE(eval_ast #>> '{}' = ANY(builtin_functions), FALSE)
                        ) q_callable (is_special_form, is_builtin_function)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                -- This condition handles special forms. We just place the form on the stack as is. It should be handled first
                                WHEN ret_empty AND is_special_form THEN JSONB_BUILD_ARRAY(eval_ast)
                                WHEN ast_is_array AND NOT ast_array_empty THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#eval', eval_ast[0]))
                                WHEN ast_array_empty AND NOT ret_empty THEN JSONB_BUILD_ARRAY(eval_ret)
                                WHEN ast_is_symbol AND NOT is_builtin_function THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#get-env', eval_ast))
                                END
                        ) q_calls (calls)
                ) eval ON (opcode = '#eval')
        LEFT JOIN LATERAL
                (
                SELECT  value AS ret,
                        CASE WHEN value IS NULL THEN FORMAT('Variable %s not found', key) END AS throws
                FROM    (SELECT args ->> 0) AS q_key (key)
                LEFT JOIN LATERAL
                        (
                        WITH    RECURSIVE get (current_env) AS
                                (
                                SELECT  heap -> env
                                UNION ALL
                                SELECT  heap -> (current_env ->> 0)::INT
                                FROM    get
                                WHERE   current_env IS NOT NULL
                                )
                        SELECT  vals -> key
                        FROM    get
                        CROSS JOIN LATERAL (SELECT current_env -> 1) q (vals)
                        WHERE   vals ? key
                        LIMIT   1
                        ) AS env_value (value) ON TRUE
                ) get_env ON (opcode = '#get-env')
        LEFT JOIN LATERAL (SELECT args #>> '{0}' AS new_output, '[]'::JSONB AS new_stack) throw ON (opcode = 'throw')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN JSONB_TYPEOF(args[0]) = 'number' AND JSONB_TYPEOF(args[1]) = 'number' THEN
                        TO_JSONB(args[0]::DOUBLE PRECISION + args[1]::DOUBLE PRECISION) END AS ret
                ) plus ON (opcode = '+')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_OBJECT('t', 'fn*', 'v', JSONB_BUILD_ARRAY(env, args[0], COALESCE(args[1], c_null))) ret,
                        CASE
                        WHEN JSONB_TYPEOF(args[0]) <> 'array' OR EXISTS (SELECT FROM JSONB_ARRAY_ELEMENTS(args[0]) arg WHERE JSONB_TYPEOF(arg) <> 'string') THEN
                                'The first argument to fn* should be a list of symbols'
                        END throws
                ) fn ON (opcode = 'fn*')
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(heap),
                        COALESCE(env),
                        COALESCE(print.new_output, throw.new_output),
                        COALESCE(throw.new_stack),
                        COALESCE(print.ret, eval.ret, plus.ret, get_env.ret, fn.ret),
                        COALESCE(eval.calls) AS calls,
                        COALESCE(get_env.throws, fn.throws) AS throws
                ) rets (new_heap, new_env, new_output, new_stack, ret, calls, throws)
        CROSS JOIN LATERAL
                (
                SELECT  rets.new_heap,
                        COALESCE(
                                rets.new_stack,
                                CASE WHEN rets.throws IS NOT NULL THEN stack || JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('throw', rets.throws)) END,
                                JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE),
                                (stack - (-1)) || COALESCE(rets.calls, '[]'::JSONB)) AS new_stack,
                        rets.new_env,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT  *
FROM    loop

We have successfully created a function value. Now we need a way to apply it.

Function application

To apply a function, we need to do some preparation first:

1. Create a new environment. To do that, we need to modify our heap, something that we haven't done yet. We'll add a new opcode, #add-heap, to handle this. Later, it will come in handy for other things. This opcode will return an integer, a reference to the position of the new environment (or any other value passed to #add-heap) in the heap array.
2. Set the newly created environment as active. We'll need to modify the value of the env field in our loop. That's also something we haven't done before. To do this, we'll create a new opcode, #push-env.
3. Populate the now-active environment with argument values. We have a list of argument names in the function definition object, and the list of their values on the call stack. We need to put these names and values into the environment map object.

   To do this, we'll create a new opcode #fill-args. In this opcode's processor, we'll need to zip an array of names with an array of values to create a key-value map object, and then merge this object with the environment.

   PostgreSQL supports object merging natively, but not array zipping. We'll have to code it up in SQL. Fortunately, it's not that hard.

4. Save the reference to the old environment on the stack. Once the function application is over, we will need to restore the old environment. This will be done by the new opcode #pop-env.
5. Finally, we just need to replace the current stack frame with an #eval, giving it the AST from the function definition as a parameter.

After all these preparations, the stack will look like this:

["#pop-env", <ENV>]
["#eval", <FN-AST>]
["#fill-args", <FN-ARGS>, <ARGS>]
["#push-env", <FN-ENV>]


Tail-call optimization

In many places of our code, when we need to emit a new opcode, we just replace the current stack frame with it, instead of pushing it on the stack. This is an optimization trick.

Let's imagine we're evaluating a form, and we have the values and the opcode ready. We could have coded our #eval the straightforward way: push the opcode on the stack, wait for it to return the value, then, when we have the value, return it to the caller.

This would require at least three "ticks" of our virtual machine. We would also need to add some kind of a marker for #eval to distinguish between the states when it needs to call the opcode, and when it needs to return the value. That's, as a matter of fact, what compilers for many languages actually do when generating machine code.

But why waste two ticks and stack space, if we know that the only thing we'll do is pass the value back? We could (and we did) just replace the #eval frame with the next opcode, and have it return the value straight to the caller.

This trick is called "tail-call optimization". If we're in a state where we're waiting for a result of an operation downstream with the sole intent to pass this result upstream, we can safely remove ourselves from this state. We just need to tell the guy downstream to talk directly to the guy upstream, and cut the middleman.

Normally, when you watch the execution stack of a language without tail-call optimization, it grows until the very end, and then decreases sharply, when the last operation in the chain needs to pass its result forty layers up the bucket line. All the callers are patiently sitting there, waiting for the result, taking up precious stack space, and wasting CPU cycles on stack unwinding.

In a language with tail-call optimization, the stack usually stays low. Once everything is done to prepare the work for the next guy, the caller knows that it's not needed anymore. It gives the floor to whoever is doing the real job, tells them where to deliver the result, and removes itself from the process. Just the way it works at a well-run company with good management.

It's very satisfactory to watch tail-call optimization done right. It's reassuring to see that control flow, through a series of well-orchestrated moves, reaches a measly service function ten levels of indirection deep, calls it, and has it return its result straight to His Majesty the End User. It conveys a very powerful message: "We've done all the preparatory work, and we're not babysitting anyone anymore. This guy is right for the job. Whatever he does is what you've asked of us." It gives off an air of confidence.

And, of course, it becomes a pain to debug when something inevitable doesn't go according to plan.

But I digress.

TCO in function calls

We did the TCO in #eval implementation, but we can't do it in function application, not without a bit of hackery.

The reason is the #pop-env opcode, which goes before #eval. #eval doesn't return its result straight to the caller. It needs to go through #pop-env first. #pop-env will forward the result to the caller anyway, but it will also restore the old environment.

If we have a lot of functions calling each other, even in a tail position, the stack will look like this:


["#pop-env", <ENV1>]
["#pop-env", <ENV2>]
["#pop-env", <ENV3>]
["#pop-env", <ENV4>]
["#pop-env", <ENV5>]
["#eval", <FN-AST>]
["#fill-args", <FN-ARGS>, <ARGS>]
["#push-env", <FN-ENV>]


The result will have to bubble up through all the #pop-env's, which is inefficient. In addition, if our call chain is deep enough, it can lead to a stack overflow.

Can we fix it?

The #pop-env's are doing exactly two things: restore the environment, and pass the result up the stack. We can see that once control flow reaches the uppermost #pop-env, the execution environment will get set <ENV1> regardless of its previous history.

It means that all the #pop-env's after the first one are, in essence, useless. The environments they restore will change again right on the next step. We might as well not even bother putting them there, except for the very first one.

That's exactly what we'll do in our implementation. When applying a function, we'll check if the previous frame on the call stack is a #pop-env. If it is, we won't put a new one there.

Let's try to apply the number-doubling function to an argument: ((fn* (a) (+ a a)) 5). This expression might be a little bit hard to parse if you're not used to Lisp notation, but it's very similar to IIFE (Immediately Invoked Function Expressions) in JavaScript. Basically, we're defining a function (something we did on the previous step), wrap it with parens, and provide it with an argument, so it's immediately applied right after definition.

Here's the SQL:

WITH    RECURSIVE
        constants AS
        (
        SELECT  ARRAY['+'] AS builtin_functions,
                ARRAY['fn*'] AS special_forms,
                'null'::JSONB AS c_null
        ),
        loop (heap, stack, env, output, step) AS
        (
        SELECT  '[[null, {}]]'::JSONB, '[["print"], ["#eval", [["fn*", ["a"], ["+", "a", "a"]], 5]]]'::JSONB, 0, NULL::TEXT, 0
        UNION ALL
        SELECT  new.new_heap, new.new_stack, new.new_env, new.new_output, step + 1
        FROM    loop
        CROSS JOIN constants
        CROSS JOIN LATERAL
                (
                SELECT  stack[-1], stack #> '{-1, 0}' AS callable, JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]'), stack #>> '{-2, 0}' = '#pop-env'
                ) q_args (current_frame, callable, args, can_tco)
        CROSS JOIN LATERAL
                (
                SELECT  CASE
                        WHEN JSONB_TYPEOF(callable) = 'string' THEN callable #>> '{}'
                        -- Function value is not a string, so we can't map it straight to the opcode. We'll need to make up one.
                        WHEN callable ->> 't' = 'fn*' THEN '#fn-apply'
                        END
                ) q_call (opcode)
        LEFT JOIN LATERAL
                (
                SELECT  'null'::JSONB ret,
                        CASE WHEN args #>> '{0, t}' = 'str' THEN args #>> '{0, v}' ELSE args[0]::TEXT END new_output
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN calls IS NULL THEN eval_ast END AS ret, calls
                FROM    (SELECT args['0'], JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]'), current_frame) q (eval_ast, eval_ret)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(eval_ast) = 'array', JSONB_TYPEOF(eval_ast) = 'string') q_array (ast_is_array, ast_is_symbol)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE WHEN ast_is_array THEN JSONB_ARRAY_LENGTH(eval_ast) = 0 ELSE FALSE END,
                                JSONB_ARRAY_LENGTH(eval_ret) = 0
                        ) q_empty (ast_array_empty, ret_empty)
                CROSS JOIN LATERAL
                        (
                        SELECT  COALESCE(eval_ast #>> '{0}' = ANY(special_forms), FALSE),
                                COALESCE(eval_ast #>> '{}' = ANY(builtin_functions), FALSE)
                        ) q_callable (is_special_form, is_builtin_function)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                -- This condition handles special forms. We just place the form on the stack as is. It should be handled first
                                WHEN ret_empty AND is_special_form THEN JSONB_BUILD_ARRAY(eval_ast)
                                WHEN ast_is_array AND NOT ast_array_empty THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#eval', eval_ast[0]))
                                WHEN ast_array_empty AND NOT ret_empty THEN JSONB_BUILD_ARRAY(eval_ret)
                                WHEN ast_is_symbol AND NOT is_builtin_function THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#get-env', eval_ast))
                                END
                        ) q_calls (calls)
                ) eval ON (opcode = '#eval')
        LEFT JOIN LATERAL
                (
                SELECT  value AS ret,
                        CASE WHEN value IS NULL THEN FORMAT('Variable %s not found', key) END AS throws
                FROM    (SELECT args ->> 0) AS q_key (key)
                LEFT JOIN LATERAL
                        (
                        WITH    RECURSIVE get (current_env) AS
                                (
                                SELECT  heap -> env
                                UNION ALL
                                SELECT  heap -> (current_env ->> 0)::INT
                                FROM    get
                                WHERE   current_env IS NOT NULL
                                )
                        SELECT  vals -> key
                        FROM    get
                        CROSS JOIN LATERAL (SELECT current_env -> 1) q (vals)
                        WHERE   vals ? key
                        LIMIT   1
                        ) AS env_value (value) ON TRUE
                ) get_env ON (opcode = '#get-env')
        LEFT JOIN LATERAL (SELECT args #>> '{0}' AS new_output, '[]'::JSONB AS new_stack) throw ON (opcode = 'throw')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN JSONB_TYPEOF(args[0]) = 'number' AND JSONB_TYPEOF(args[1]) = 'number' THEN
                        TO_JSONB(args[0]::DOUBLE PRECISION + args[1]::DOUBLE PRECISION) END AS ret
                ) plus ON (opcode = '+')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_OBJECT('t', 'fn*', 'v', JSONB_BUILD_ARRAY(env, args[0], COALESCE(args[1], c_null))) ret,
                        CASE
                        WHEN JSONB_TYPEOF(args[0]) <> 'array' OR EXISTS (SELECT FROM JSONB_ARRAY_ELEMENTS(args[0]) arg WHERE JSONB_TYPEOF(arg) <> 'string') THEN
                                'The first argument to fn* should be a list of symbols'
                        END throws
                ) fn ON (opcode = 'fn*')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN can_tco THEN next_frames ELSE JSONB_INSERT(next_frames, '{0}', JSONB_BUILD_ARRAY('#pop-env', call_env), FALSE) END AS calls
                FROM    (SELECT CASE WHEN callable ->> 't' = 'fn*' THEN (callable #>> '{v, 0}') END::INT) AS q_call_args (call_env)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_BUILD_ARRAY(
                                        JSONB_BUILD_ARRAY('#eval', callable #> '{v, 2}'),
                                        JSONB_BUILD_ARRAY('#fill-args', callable #> '{v, 1}', args),
                                        JSONB_BUILD_ARRAY('#push-env', call_env)
                                )
                        ) AS q_next_frames (next_frames)
                ) fn_apply ON (callable ->> 't' = 'fn*')
        LEFT JOIN LATERAL (SELECT CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0)::INT END new_env, args['1'] ret) pop_env ON (opcode = '#pop-env')
        LEFT JOIN LATERAL
                (
                SELECT  new_env,
                        CASE
                        WHEN new_env IS NULL THEN
                                JSONB_BUILD_ARRAY(
                                        current_frame,
                                        JSONB_BUILD_ARRAY('#add-heap', JSONB_BUILD_ARRAY(old_env, JSONB_BUILD_OBJECT()))
                                )
                        END AS calls
                FROM    (SELECT CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0)::INT END, CASE WHEN JSONB_TYPEOF(args['1']) = 'number' THEN (args ->> 1)::INT END) q_args (old_env, new_env)
                ) push_env ON (opcode = '#push-env')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_SET(heap, ARRAY[env::TEXT, 1::TEXT], env_data || new_data) AS new_heap
                FROM    (
                        SELECT  CASE WHEN opcode = '#fill-args' AND JSONB_TYPEOF(args['0']) = 'array' THEN args[0] END,
                                CASE WHEN opcode = '#fill-args' AND JSONB_TYPEOF(args['1'])  = 'array' THEN args[1] END
                        ) q_args (keys, values)
                CROSS JOIN LATERAL (SELECT heap[env::TEXT]['1']) q_env_data (env_data)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_OBJECT_AGG(key #>> '{}', COALESCE(value, c_null) ORDER BY index)
                        FROM    JSONB_ARRAY_ELEMENTS(keys) WITH ORDINALITY q_keys (key, index)
                        LEFT JOIN
                                JSONB_ARRAY_ELEMENTS(values) WITH ORDINALITY q_values (value, index)
                        USING   (index)
                        ) q_new_data (new_data)
                ) fill_args ON (opcode = '#fill-args')
        LEFT JOIN LATERAL
                (
                SELECT  new_heap, TO_JSONB(JSONB_ARRAY_LENGTH(new_heap) - 1) ret
                FROM    (SELECT JSONB_INSERT(heap, '{-1}', args[0], TRUE)) AS q_heap (new_heap)
                ) add_heap ON (opcode = '#add-heap')
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(fill_args.new_heap, add_heap.new_heap, heap),
                        COALESCE(pop_env.new_env, push_env.new_env, env),
                        COALESCE(print.new_output, throw.new_output),
                        COALESCE(throw.new_stack),
                        COALESCE(print.ret, eval.ret, plus.ret, get_env.ret, fn.ret, pop_env.ret, add_heap.ret),
                        COALESCE(eval.calls, fn_apply.calls, push_env.calls) AS calls,
                        COALESCE(get_env.throws, fn.throws) AS throws
                ) rets (new_heap, new_env, new_output, new_stack, ret, calls, throws)
        CROSS JOIN LATERAL
                (
                SELECT  rets.new_heap,
                        COALESCE(
                                rets.new_stack,
                                CASE WHEN rets.throws IS NOT NULL THEN stack || JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('throw', rets.throws)) END,
                                JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE),
                                (stack - (-1)) || COALESCE(rets.calls, '[]'::JSONB)) AS new_stack,
                        rets.new_env,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT  *
FROM    loop

It starts to feel like a real language now!

Try/catch

In the implementation we've made so far, all exceptions are show-stoppers. Let's make a way to handle them.

In Mal, there's a special form for that, which has the following shape: (try* <FORM> (catch* <EXCEPTION-VAR> <HANDLER-FORM>)). It evaluates <FORM>. If it results in an exception, <EXCEPTION-VAR> gets set to the value of that exception in a new environment, and <HANDLER-FORM> is evaluated.

To add support for handling exceptions, we'll do the following:

1. Before evaluating forms that come from a try* block, put the instruction #catch on the stack with four preset parameters:
   1. The constant false
   2. The environment active at the moment try* was called
   3. The name of the variable that will hold the exception value
   4. The AST of the catch* block
2. Change the handler of the opcode throw:
   1. Go up the stack and find the closest #catch instruction.
   2. If it's there, change its first parameter to true and put the value of the exception as its last (fifth) parameter.
   3. If it's not there, invoke the default handler: wipe the stack completely and call the opcode print with the value of the exception.
3. Add a handler for the opcode #catch. This handler will look at the first parameter and, based on its value, decide what to do next:
   1. If the parameter is false, just return the last (fifth) argument. This argument will be the result of evaluating the form in the try* block if there was no exception.
   2. If the parameter is true, create a new environment, set the value of the exception variable to the exception's value (which has been put on the stack by this moment), and evaluate the handler. This is done in a way similar to how fn* and let* handlers work.

You'll be able to see the implementation of try*/catch* in the next chapter.

Other special forms

For the sake of brevity, I won't dive into the implementation details of other special forms. By this moment, it's pretty straightforward.

One little thing I find interesting: to implement do, I found it handy to implement an additional opcode, #nop. As the name suggests, it just swallows the result and passes control up the stack. It doesn't return anything, doesn't call new opcodes, doesn't change the heap or the current environment, and doesn't throw exceptions.

With all the boilerplate code we've written so far, here's what the implementation of this opcode looks like in all its glory:

LEFT JOIN LATERAL (SELECT) nop ON (opcode = '#nop')

It doesn't really do anything, and we could just as well remove it. But I decided to leave it in. I think it's pretty neat and conveys the intent really clearly.

Parser

So far, I've been manually converting S-expressions into our JSONB representation and putting them on the stack. It's time we automated this. To this end, we'll need to implement a reader and a parser.

The creators of Mal have provided a regular expression that parses a string and returns Mal tokens. We'll wrap these tokens into an object called "reader". It will be a stateful opaque object that will live on the heap: basically, an array of string representations of tokens, and a pointer to the position in this array. This reader will be created by the opcode #tokenize. It will use #add-heap to put the reader on the heap, and then wrap its reference into a reader object that can be safely used on the stack.

The parser will read the stream of (string) tokens and create the AST out of them.

The opcode responsible for that will be called #read-form. It will be a "stateful" opcode, which means that it will call itself in a loop and change the values of its arguments. If the first argument to #read-form is an opening token of some sort (paren, regular bracket, or curly bracket), then it will keep accruing tokens and forms (read recursively) until it encounters a closing token. Otherwise, it will return the token as is.

The opcode responsible for converting a string representation of a token into JSONB representation is called #next-reader. It does a double job: converts the tokens and manages the state of the reader.

To mark the end of the token stream, we'll introduce a new value, the EOF. It will be represented by the JSONB object {"t": "eof"}

Because #read-form is stateful, we need its user-callable counterpart, read-form. It serves a double purpose. First, it sanitizes the arguments that it passes to #read-form. Second, #read-form (with the hash), unless it's in a list/vector/map evaluation state, passes closing tokens down the line as is. If we have an unbalanced closing paren (that is, one closing paren too many), read-form is the only one able to catch it.

Let's try to evaluate the expression (+ 1 (+ 2 3)). Note that this time, we're putting it on the stack as a string, and not as a manually-parsed AST as we did before. Because of that, we're not using the opcode #eval directly, but rather its user-facing version eval. Otherwise, we won't be able to catch errors like unbalanced closing brackets.

That's how it works:

WITH    RECURSIVE
        constants AS
        (
        SELECT  ARRAY[
                'tokenize', 'print', 'to-json', 'read-input', 'read-form', 'throw', 'eval', 'list', 'cons', 'concat', 'pr-str',
                '+', '-', '*', '/', '='] AS builtin_functions,
                ARRAY['fn*', 'def!', 'let*', 'if', 'do', 'quote', 'quasiquote', 'defmacro!'] AS special_forms,
                ARRAY['(', ')', '[', ']', '{', '}', '''', '`', '~', '~@'] AS symbols,
                TO_JSONB('('::TEXT) AS c_open_paren, TO_JSONB(')'::TEXT) AS c_closed_paren,
                TO_JSONB('['::TEXT) AS c_open_bracket, TO_JSONB(']'::TEXT) AS c_closed_bracket,
                TO_JSONB('{'::TEXT) AS c_open_curly, TO_JSONB('}'::TEXT) AS c_closed_curly,
                TO_JSONB('unquote'::TEXT) AS c_unquote, TO_JSONB('splice-unquote'::TEXT) AS c_splice_unquote,
                'null'::JSONB AS c_null, '{"t": "eof"}'::JSONB AS c_eof,
                '{"''": "quote", "`": "quasiquote", "~": "unquote", "~@": "splice-unquote"}'::JSONB AS c_reader_macros
        ),
        loop (heap, stack, env, output, step) AS
        (
        SELECT  '[[null, {"eof": {"t": "eof"}}]]'::JSONB,
                '[["print"], ["eval"], ["read-form"], ["tokenize", {"t": "str", "v": "(+ 1 (+ 2 3))"}]]'::JSONB,
                0, NULL::TEXT, 1
        FROM    constants
        UNION ALL
        SELECT  new.*, step + 1
        FROM    loop
        CROSS JOIN constants
        CROSS JOIN LATERAL (SELECT stack #> '{-1, 0}', JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]'), stack['-1'], stack #>> '{-2, 0}' = '#pop-env') q_call (callable, args, current_frame, can_tco)
        CROSS JOIN LATERAL
                (
                SELECT  CASE
                        WHEN JSONB_TYPEOF(callable) = 'string' THEN callable #>> '{}'
                        WHEN callable ->> 't' = 'fn*' THEN '#fn-apply'
                        WHEN callable ->> 't' = 'macro' THEN '#macro-apply'
                        END
                ) q_opcode (opcode)
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#tokenize', args[0])) AS calls,
                        CASE WHEN (args #>> '{0, t}' = 'str') IS DISTINCT FROM TRUE THEN 'Usage: (tokenize string)' END AS throws
                ) tokenize ON (opcode = 'tokenize')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN args[1] IS NOT NULL THEN JSONB_BUILD_OBJECT('t', 'reader', 'v', args[1]) END AS ret,
                        CASE WHEN args[1] IS NULL THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#add-heap', JSONB_BUILD_ARRAY(0, tokens))) END AS calls
                FROM    (
                        SELECT  COALESCE(JSONB_AGG(TO_JSONB(token) ORDER BY index), '[]'::JSONB)
                        FROM    REGEXP_MATCHES(args #>> '{0, v}', '[\s,]*(~@|[\[\]{}()''`~^@]|"(?:\\.|[^\\"])*"?|;.*|[^\s\[\]{}(''"`~^,;)]*)', 'gm') WITH ORDINALITY AS q (match, index)
                        CROSS JOIN LATERAL (SELECT match[1]) q_token (token)
                        WHERE   token > '' AND NOT token ^@ ';'
                        ) q_tokens (tokens)
                ) tokenize2 ON (opcode = '#tokenize')
        LEFT JOIN LATERAL (SELECT NULL::JSONB AS ret) read_input ON (opcode = 'read-input')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN is_string THEN c_null END AS ret,
                        CASE WHEN is_string THEN args #>> '{0, v}' END new_output,
                        CASE WHEN NOT is_string THEN JSONB_BUILD_ARRAY(JSON_BUILD_ARRAY(opcode), JSONB_BUILD_ARRAY('pr-str', COALESCE(args[0], c_null))) END AS calls
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        LEFT JOIN LATERAL (SELECT TO_JSONB(args['0']::TEXT) AS ret) _to_json ON (opcode = 'to-json')
        LEFT JOIN LATERAL
                (
                SELECT  *,
                        CASE
                        WHEN type = 'array' AND JSONB_ARRAY_LENGTH(value) > 0 THEN NULL
                        ELSE JSONB_BUILD_OBJECT('t', 'str', 'v', TO_JSONB(
                                CASE
                                WHEN type IN ('number', 'boolean') THEN value::TEXT
                                WHEN type = 'string' THEN value #>> '{}'
                                WHEN type = 'null' THEN 'nil'
                                WHEN complex_type = 'str' THEN (value -> 'v')::TEXT
                                WHEN type = 'array' AND JSONB_ARRAY_LENGTH(value) = 0 THEN
                                (
                                SELECT  '(' || COALESCE(STRING_AGG(element #>> '{v}', ' ' ORDER BY index), '') || ')'
                                FROM    JSONB_ARRAY_ELEMENTS(rest) WITH ORDINALITY q (element, index)
                                )
                                WHEN complex_type IS NOT NULL THEN FORMAT('#<%s>', complex_type)
                                ELSE 'Unprintable value'
                                END))
                        END
                        AS ret,
                        CASE WHEN type = 'array' AND JSONB_ARRAY_LENGTH(value) > 0 THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('pr-str', value['0'])) END AS calls
                FROM    (SELECT args[0], JSONB_TYPEOF(args[0]), args[0] ->> 't', JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]')) q (value, type, complex_type, rest)
                ) pr_str ON (opcode = 'pr-str')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN catch_index IS NOT NULL THEN
                                JSONB_PATH_QUERY_ARRAY(stack, '$[0 to $index - 1]', JSONB_BUILD_OBJECT('index', catch_index)) ||
                                JSONB_BUILD_ARRAY(JSONB_SET(stack[catch_index], '{1}', 'true'::JSONB) || COALESCE(args[0], c_null))
                        ELSE JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('print', args[0]))
                        END
                        AS new_stack
                FROM    (SELECT) q
                LEFT JOIN LATERAL
                        (
                        SELECT  index::INT - 1
                        FROM    JSONB_ARRAY_ELEMENTS(stack) WITH ORDINALITY q_frames (frame, index)
                        WHERE   (frame ->> 0) = '#catch*'
                        ORDER BY
                                index DESC
                        LIMIT 1
                        ) q_catch_index (catch_index) ON TRUE
                ) throw ON (opcode = 'throw')
        LEFT JOIN LATERAL
                (
                SELECT  new_heap, TO_JSONB(JSONB_ARRAY_LENGTH(new_heap) - 1) ret
                FROM    (SELECT JSONB_INSERT(heap, '{-1}', args[0], TRUE)) AS q_heap (new_heap)
                ) add_heap ON (opcode = '#add-heap')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN value IS NULL THEN c_eof
                        WHEN value = 'nil' THEN c_null
                        WHEN value ~ '^[-+]?\d+' OR value IN ('true', 'false') THEN value::JSONB
                        WHEN value ^@ '"' THEN JSONB_BUILD_OBJECT('t', 'str', 'v', value::JSONB)
                        ELSE TO_JSONB(value)
                        END AS ret,
                        CASE WHEN value IS NOT NULL THEN JSONB_SET(heap, ARRAY[reader_ptr, '0'], TO_JSONB(position + 1)) END AS new_heap
                FROM    (SELECT args['0']) AS q_reader_ref (reader_ref)
                CROSS JOIN LATERAL (SELECT reader_ref ->> 'v') AS q_reader_ptr (reader_ptr)
                CROSS JOIN LATERAL (SELECT heap[reader_ptr]) q_reader (reader)
                CROSS JOIN LATERAL (SELECT CASE WHEN JSONB_TYPEOF(reader['0']) = 'number' THEN (reader['0'] #> '{}')::INT END) q_position (position)
                CROSS JOIN LATERAL (SELECT reader['1'] ->> position) AS q_value (value)
                ) next_reader ON (opcode = '#next-reader')
        LEFT JOIN LATERAL
                (
                SELECT  ret,
                        CASE WHEN ret IS NULL THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#read-form', reader_ref)) END AS calls,
                        CASE
                        WHEN (reader_ref ->> 't') = 'reader' IS DISTINCT FROM TRUE THEN 'Invalid reader reference'
                        WHEN ret #>> '{}' IN (')', ']', '}') THEN 'Unbalanced parens'
                        END AS throws
                FROM    (SELECT  args[0], args[1]) q_args (reader_ref, ret)
                ) read_form ON (opcode = 'read-form')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN calls IS NOT NULL THEN NULL
                        WHEN is_open_paren AND is_closed_paren THEN chunk
                        WHEN is_open_bracket AND is_closed_bracket THEN JSONB_BUILD_OBJECT('t', 'vec', 'v', chunk)
                        WHEN is_open_curly AND is_closed_curly AND JSONB_ARRAY_LENGTH(chunk) % 2 = 0 THEN
                                (
                                SELECT  JSONB_BUILD_OBJECT('t', 'map', 'v', COALESCE(JSONB_OBJECT_AGG(key #>> '{}', value), '{}'::JSONB))
                                FROM    JSONB_ARRAY_ELEMENTS(chunk) WITH ORDINALITY keys (key, index)
                                JOIN    JSONB_ARRAY_ELEMENTS(chunk) WITH ORDINALITY values (value, index)
                                ON      values.index = keys.index + 1
                                WHERE   keys.index % 2 = 1
                                )
                        WHEN NOT is_open THEN first_token
                        END AS ret,
                        calls,
                        CASE
                        WHEN (is_open AND is_eof) OR
                             (is_closed AND (
                                     (is_open_paren AND NOT is_closed_paren) OR
                                     (is_open_bracket AND NOT is_closed_bracket) OR
                                     (is_open_curly AND NOT is_closed_curly)
                                     )) THEN 'Unbalanced parens'
                        WHEN is_open_curly AND NOT is_closed_curly AND JSONB_ARRAY_LENGTH(chunk) % 2 = 1 AND JSONB_TYPEOF(chunk[-1]) <> 'string' THEN
                                FORMAT('Cannot use token "%s" as a map key', last_token)
                        END AS throws
                FROM    (SELECT  args[0] reader_ref, args[1] first_token, args[-1] last_token, JSONB_PATH_QUERY_ARRAY(args, '$[2 to last - 1]') chunk) q_tokens
                CROSS JOIN LATERAL
                        (
                        SELECT  COALESCE(first_token = c_open_paren, FALSE) is_open_paren,
                                COALESCE(last_token = c_closed_paren, FALSE) is_closed_paren,
                                COALESCE(first_token = c_open_bracket, FALSE) is_open_bracket,
                                COALESCE(last_token = c_closed_bracket, FALSE) is_closed_bracket,
                                COALESCE(first_token = c_open_curly, FALSE) is_open_curly,
                                COALESCE(last_token = c_closed_curly, FALSE) is_closed_curly,
                                COALESCE(last_token = c_eof, FALSE) is_eof,
                                c_reader_macros ->> (first_token #>> '{}') reader_macro_form
                        ) q_state
                CROSS JOIN LATERAL
                        (
                        SELECT  is_open_paren OR is_open_curly OR is_open_bracket AS is_open,
                                is_closed_paren OR is_closed_curly OR is_closed_bracket AS is_closed
                        ) q_state2
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                WHEN first_token IS NULL THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#next-reader', reader_ref))
                                WHEN is_open AND NOT (is_closed OR is_eof) THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#read-form', reader_ref))
                                WHEN reader_macro_form IS NOT NULL THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('list', reader_macro_form), JSONB_BUILD_ARRAY('#read-form', reader_ref))
                                END
                        ) q_calls (calls)
                ) read_form2 ON (opcode = '#read-form')
        LEFT JOIN LATERAL (SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', COALESCE(args[0], c_null))) AS calls) eval ON (opcode = 'eval')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN eval_calls IS NULL THEN eval_ast END AS ret,
                        eval_calls AS calls
                FROM    (SELECT args['0'], JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]')) q (eval_ast, eval_ret)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(eval_ast) = 'array', JSONB_TYPEOF(eval_ast) = 'string') q_array (ast_is_array, ast_is_symbol)
                CROSS JOIN LATERAL (SELECT CASE WHEN ast_is_array THEN JSONB_ARRAY_LENGTH(eval_ast) = 0 ELSE FALSE END, JSONB_ARRAY_LENGTH(eval_ret) = 0) q_empty (ast_array_empty, ret_empty)
                CROSS JOIN LATERAL
                        (
                        SELECT  COALESCE(eval_ast #>> '{0}' = ANY(special_forms), FALSE),
                                COALESCE(eval_ast #>> '{}' = ANY(builtin_functions), FALSE)
                        ) q_callable (is_special_form, is_builtin_function)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                WHEN ret_empty AND is_special_form THEN JSONB_BUILD_ARRAY(eval_ast)
                                WHEN ast_is_array AND NOT ast_array_empty THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#eval', eval_ast[0]))
                                WHEN ast_array_empty AND NOT ret_empty THEN JSONB_BUILD_ARRAY(eval_ret)
                                WHEN ast_is_symbol AND NOT is_builtin_function THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#get-env', eval_ast))
                                END
                        ) q_calls (eval_calls)
                ) eval2 ON (opcode = '#eval')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_OBJECT('t', 'fn*', 'v', JSONB_BUILD_ARRAY(env, args[0], COALESCE(args[1], c_null))) ret,
                        CASE
                        WHEN JSONB_TYPEOF(args[0]) <> 'array' OR EXISTS (SELECT FROM JSONB_ARRAY_ELEMENTS(args[0]) arg WHERE JSONB_TYPEOF(arg) <> 'string') THEN
                                'The first argument to fn* should be a list of symbols'
                        END throws
                ) fn ON (opcode = 'fn*')
        LEFT JOIN LATERAL
                (
                SELECT  ret,
                        CASE WHEN ret IS NULL THEN FORMAT('Error applying operator: (%s %s %s)', opcode, args -> 0, args -> 1) END AS throws
                FROM    (
                        SELECT  CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0) END::DOUBLE PRECISION,
                                CASE WHEN JSONB_TYPEOF(args['1']) = 'number' THEN (args ->> 1) END::DOUBLE PRECISION
                        ) q_args (arg1, arg2)
                LEFT JOIN LATERAL
                        (
                        SELECT  TO_JSONB(
                                CASE opcode
                                WHEN '+' THEN TO_JSONB(arg1 + arg2)
                                WHEN '-' THEN TO_JSONB(arg1 - arg2)
                                WHEN '*' THEN TO_JSONB(arg1 * arg2)
                                WHEN '/' THEN TO_JSONB(arg1 / arg2)
                                END
                                )
                        ) q_ret (ret) ON TRUE
                ) math_ops ON (opcode IN ('+', '-', '*', '/'))
        LEFT JOIN LATERAL
                (
                SELECT  TO_JSONB(COALESCE(args[0] = args[1], FALSE)) AS ret
                ) eq ON (opcode = '=')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN can_tco THEN next_frames ELSE JSONB_INSERT(next_frames, '{0}', JSONB_BUILD_ARRAY('#pop-env', env), FALSE) END AS calls
                FROM    (SELECT CASE WHEN callable ->> 't' = 'fn*' THEN (callable #>> '{v, 0}') END::INT) AS q_call_args (call_env)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_BUILD_ARRAY(
                                        JSONB_BUILD_ARRAY('#eval', callable #> '{v, 2}'),
                                        JSONB_BUILD_ARRAY('#fill-args', callable #> '{v, 1}', args),
                                        JSONB_BUILD_ARRAY('#push-env', call_env)
                                )
                        ) AS q_next_frames (next_frames)
                ) fn_apply ON (callable ->> 't' = 'fn*')
        LEFT JOIN LATERAL (SELECT JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval'), JSONB_SET(current_frame, '{0, t}', TO_JSONB('fn*'::TEXT))) AS calls) macro_apply ON (callable ->> 't' = 'macro')
        LEFT JOIN LATERAL
                (
                SELECT  value AS ret,
                        CASE WHEN value IS NULL THEN FORMAT('Variable %s not found', key) END AS throws
                FROM    (SELECT args ->> 0) AS q_key (key)
                LEFT JOIN LATERAL
                        (
                        WITH    RECURSIVE get (current_env) AS
                                (
                                SELECT  heap -> env
                                UNION ALL
                                SELECT  heap -> (current_env ->> 0)::INT
                                FROM    get
                                WHERE   current_env IS NOT NULL
                                )
                        SELECT  vals -> key
                        FROM    get
                        CROSS JOIN LATERAL (SELECT current_env -> 1) q (vals)
                        WHERE   vals ? key
                        LIMIT   1
                        ) AS env_value (value) ON TRUE
                ) get_env ON (opcode = '#get-env')
        LEFT JOIN LATERAL
                (
                SELECT  value AS ret,
                        JSONB_SET(heap, ARRAY[env::TEXT, 1::TEXT, key], value, TRUE) AS new_heap
                FROM    (SELECT args ->> 0, args['1']) q (key, value)
                ) set_env ON (opcode = '#set-env')
        LEFT JOIN LATERAL
                (
                SELECT  new_env,
                        CASE
                        WHEN new_env IS NULL THEN
                                JSONB_BUILD_ARRAY(
                                        current_frame,
                                        JSONB_BUILD_ARRAY('#add-heap', JSONB_BUILD_ARRAY(old_env, JSONB_BUILD_OBJECT()))
                                )
                        END AS calls
                FROM    (SELECT CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0)::INT END, CASE WHEN JSONB_TYPEOF(args['1']) = 'number' THEN (args ->> 1)::INT END) q_args (old_env, new_env)
                ) push_env ON (opcode = '#push-env')
        LEFT JOIN LATERAL (SELECT CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0)::INT END new_env, args['1'] ret) pop_env ON (opcode = '#pop-env')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_SET(heap, ARRAY[env::TEXT, 1::TEXT], env_data || new_data) AS new_heap
                FROM    (
                        SELECT  CASE WHEN opcode = '#fill-args' AND JSONB_TYPEOF(args['0']) = 'array' THEN args[0] END,
                                CASE WHEN opcode = '#fill-args' AND JSONB_TYPEOF(args['1'])  = 'array' THEN args[1] END
                        ) q_args (keys, values)
                CROSS JOIN LATERAL (SELECT heap[env::TEXT]['1']) q_env_data (env_data)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_OBJECT_AGG(key #>> '{}', COALESCE(value, c_null) ORDER BY index)
                        FROM    JSONB_ARRAY_ELEMENTS(keys) WITH ORDINALITY q_keys (key, index)
                        LEFT JOIN
                                JSONB_ARRAY_ELEMENTS(values) WITH ORDINALITY q_values (value, index)
                        USING   (index)
                        ) q_new_data (new_data)
                ) fill_args ON (opcode = '#fill-args')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN can_tco THEN next_frames
                        ELSE JSONB_INSERT(next_frames, '{0}', JSONB_BUILD_ARRAY('#pop-env', env), FALSE)
                        END AS calls,
                        CASE
                        WHEN NOT vars_is_array THEN 'The first argument to let* should be a list'
                        WHEN body IS NULL THEN 'There should be two arguments to let*'
                        END AS throws
                FROM    (SELECT args['0'] IS NOT NULL AND JSONB_TYPEOF(args['0']) = 'array', args['0'], args['1']) q_vars_array (vars_is_array, vars, body)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_BUILD_ARRAY(
                                        JSONB_BUILD_ARRAY('#let*', vars, body),
                                        JSONB_BUILD_ARRAY('#push-env', env)
                                )
                        ) q_next_frames (next_frames)
                ) let ON (opcode = 'let*')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN vars_is_array AND JSONB_ARRAY_LENGTH(vars) = 0 THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', body))
                        ELSE JSONB_BUILD_ARRAY(
                                JSONB_SET(current_frame, '{1}', JSONB_PATH_QUERY_ARRAY(vars, '$[2 to last]')),
                                JSONB_BUILD_ARRAY('#set-env', key ->> 'v'),
                                JSONB_BUILD_ARRAY('#eval', value)
                        )
                        END AS calls,
                        CASE WHEN JSONB_TYPEOF(key) <> 'string' THEN 'let*: variable names should be symbols' END AS throws
                FROM    (SELECT args[0], args[1]) q_vars_body (vars, body)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(vars) = 'array', vars[0], COALESCE(vars[1], c_null)) q_key_value (vars_is_array, key, value)
                ) let2 ON (opcode = '#let*')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#set-env', key), JSONB_BUILD_ARRAY('#eval', value)) AS calls,
                        CASE
                        WHEN (JSONB_TYPEOF(key) = 'string') IS DISTINCT FROM TRUE THEN 'def!: variable name should be a symbol'
                        WHEN value IS NULL THEN 'def!: value should be provided'
                        END AS throws
                FROM    (SELECT args[0], args[1]) q_vars (key, value)
                ) def ON (opcode = 'def!')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#set-env', key), JSONB_BUILD_ARRAY('#fn-to-macro'), JSONB_BUILD_ARRAY('#eval', value)) AS calls,
                        CASE
                        WHEN JSONB_TYPEOF(key) = 'string' IS DISTINCT FROM TRUE THEN 'defmacro!: variable name should be a symbol'
                        END AS throws
                FROM    (SELECT args[0], args[1]) q_vars (key, value)
                ) defmacro ON (opcode = 'defmacro!')
        LEFT JOIN LATERAL
                (
                SELECT  CASE args #>> '{0, t}'
                        WHEN 'macro' THEN args[0]
                        WHEN 'fn*' THEN JSONB_SET(args[0], '{t}', TO_JSONB('macro'::TEXT), FALSE)
                        END AS ret,
                        CASE
                        WHEN NOT COALESCE(args #>> '{0, t}' IN ('fn*', 'macro'), FALSE) THEN 'defmacro!: the second argument should evaluate to a function or a macro'
                        END AS throws
                ) macro ON (opcode = '#fn-to-macro')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(
                                JSONB_BUILD_ARRAY('#if2', yes, COALESCE(no, c_null)),
                                JSONB_BUILD_ARRAY('#eval', condition)
                        ) AS calls,
                        CASE WHEN condition IS NULL OR yes IS NULL THEN 'if should have at least two arguments' END throws
                FROM    (SELECT args[0], args[1], args[2]) q_args (condition, yes, no)
                ) if ON (opcode = 'if')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN result IS NULL OR result IN (c_null, 'false'::JSONB) THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', no))
                        ELSE JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', yes))
                        END AS calls
                FROM    (SELECT args[0], args[1], args[2]) q_args (yes, no, result)
                ) if2 ON (opcode = '#if2')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN total > 1 THEN
                                JSONB_BUILD_ARRAY(
                                        current_frame - 1,
                                        JSONB_BUILD_ARRAY('#nop'),
                                        eval_frame
                                )
                        ELSE    JSONB_BUILD_ARRAY(eval_frame)
                        END AS calls
                FROM    (SELECT args['0'], JSONB_ARRAY_LENGTH(args)) AS q_args (next, total)
                CROSS JOIN LATERAL (SELECT JSONB_BUILD_ARRAY('#eval', COALESCE(next, c_null))) AS q_eval_frame (eval_frame)
                ) _do ON (opcode = 'do')
        LEFT JOIN LATERAL (SELECT  COALESCE(args[0], c_null) AS ret) quote ON (opcode = 'quote')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN NOT is_array THEN COALESCE(value, '[]'::JSONB)
                        WHEN is_array AND JSONB_ARRAY_LENGTH(value) = 0 THEN JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]')
                        END ret,
                        CASE
                        WHEN maybe_special = c_unquote THEN JSONB_BUILD_ARRAY(patched_frame, JSONB_BUILD_ARRAY('eval', form[1]))
                        WHEN maybe_special = c_splice_unquote THEN JSONB_BUILD_ARRAY(patched_frame, JSONB_BUILD_ARRAY('#splice'), JSONB_BUILD_ARRAY('eval', form[1]))
                        ELSE JSONB_BUILD_ARRAY(patched_frame, JSONB_BUILD_ARRAY(opcode, form))
                        END calls
                FROM    (SELECT args[0], args[0][0], args[0][0][0], COALESCE(JSONB_TYPEOF(args[0]) = 'array', FALSE)) q_args (value, form, maybe_special, is_array)
                CROSS JOIN LATERAL (SELECT current_frame #- '{1, 0}') q_patched_frame (patched_frame)
                ) quasiquote ON (opcode = 'quasiquote')
        LEFT JOIN LATERAL (SELECT JSONB_SET(stack - (-1), '{-1}', stack['-2'] || args['0']) AS new_stack) splice ON (opcode = '#splice')
        LEFT JOIN LATERAL (SELECT args AS ret) list ON (opcode = 'list')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(head) || list AS ret,
                        CASE WHEN is_list IS DISTINCT FROM TRUE THEN 'Second argument to cons should be a list' END AS throws
                FROM    (SELECT args['0'], args['1'], JSONB_TYPEOF(args['1']) = 'array') AS q_args (head, list, is_list)
                ) cons ON (opcode = 'cons')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN NOT has_non_arrays THEN
                                (
                                SELECT  JSONB_AGG(element ORDER BY arg_index, element_index)
                                FROM    JSONB_ARRAY_ELEMENTS(args) WITH ORDINALITY q_args (arg, arg_index)
                                CROSS JOIN LATERAL
                                        JSONB_ARRAY_ELEMENTS(arg) WITH ORDINALITY q_element (element, element_index)
                                )
                        END AS ret,
                        CASE WHEN has_non_arrays THEN 'All arguments to concat should be lists' END AS throws
                FROM    (
                        SELECT  COALESCE(BOOL_OR(JSONB_TYPEOF(arg) <> 'array'), FALSE)
                        FROM    JSONB_ARRAY_ELEMENTS(args) arg
                        ) q (has_non_arrays)
                ) _concat ON (opcode = 'concat')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(
                                JSONB_BUILD_ARRAY('#catch*', FALSE, env, exception_symbol, handler_body),
                                JSONB_BUILD_ARRAY('#eval', args[0])
                                ) AS calls,
                        CASE WHEN NOT COALESCE(catch_symbol #>> '{}' = 'catch*', FALSE) OR NOT COALESCE(JSONB_TYPEOF(exception_symbol) = 'string', FALSE) THEN
                                'Usage: (try* form (catch* exception handler_form))'
                        END AS throws
                FROM    (SELECT args[1][0] AS catch_symbol, args[1][1] AS exception_symbol, args[1][2] AS handler_body) q_args
                ) try ON (opcode = 'try*')
        LEFT JOIN LATERAL
                (
                SELECT  CASE caught WHEN 'false'::JSONB THEN args[4] END ret,
                        CASE
                        WHEN caught = 'false'::JSONB THEN NULL
                        WHEN can_tco THEN next_frames
                        ELSE JSONB_INSERT(next_frames, '{0}', JSONB_BUILD_ARRAY('#pop-env', catch_env), FALSE)
                        END AS calls
                FROM    (SELECT args[0] caught, args[1] catch_env, args[2] exception_symbol, args[3] handler_body, args[4] value) q_args
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_BUILD_ARRAY(
                                        JSONB_BUILD_ARRAY('#eval', handler_body),
                                        JSONB_BUILD_ARRAY('#nop'),
                                        JSONB_BUILD_ARRAY('#set-env', exception_symbol, value),
                                        JSONB_BUILD_ARRAY('#push-env', catch_env)
                                )
                        ) q (next_frames)
                ) catch ON (opcode = '#catch*')
        LEFT JOIN LATERAL (SELECT) nop ON (opcode = '#nop')
        LEFT JOIN LATERAL (SELECT FORMAT('Invalid opcode: %s', callable) AS throws) invalid_opcode ON (opcode IS NULL)
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(add_heap.new_heap, next_reader.new_heap, fill_args.new_heap, set_env.new_heap, heap),
                        COALESCE(throw.new_stack, splice.new_stack),
                        COALESCE(push_env.new_env, pop_env.new_env, env),
                        COALESCE(
                                print.ret, _to_json.ret, read_input.ret,
                                pr_str.ret, read_form.ret, read_form2.ret, tokenize2.ret, add_heap.ret, next_reader.ret,
                                eval2.ret, fn.ret, macro.ret,
                                get_env.ret, math_ops.ret, eq.ret, pop_env.ret, set_env.ret,
                                quote.ret, cons.ret, list.ret, _concat.ret, quasiquote.ret, catch.ret
                                ),
                        COALESCE(
                                print.calls, tokenize.calls, tokenize2.calls, pr_str.calls, read_form.calls, read_form2.calls,
                                eval.calls, eval2.calls, fn_apply.calls, macro_apply.calls,
                                push_env.calls, let.calls, let2.calls, def.calls, defmacro.calls, if.calls, if2.calls, _do.calls,
                                quasiquote.calls, try.calls, catch.calls),
                        COALESCE(
                                tokenize.throws, get_env.throws, read_form.throws, read_form2.throws, fn.throws, math_ops.throws, let2.throws,
                                let.throws, def.throws, defmacro.throws, macro.throws, if.throws, cons.throws, _concat.throws, try.throws,
                                invalid_opcode.throws),
                        COALESCE(print.new_output)
                ) rets (new_heap, new_stack, new_env, ret, calls, throws, new_output)
        CROSS JOIN LATERAL
                (
                SELECT  rets.new_heap,
                        COALESCE(
                                rets.new_stack,
                                CASE
                                WHEN rets.throws IS NOT NULL THEN
                                        stack || JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('throw', JSONB_BUILD_OBJECT('t', 'str', 'v', rets.throws)))
                                END,
                                JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE),
                                (stack - (-1)) || COALESCE(rets.calls, '[]'::JSONB)
                        ) AS new_stack,
                        rets.new_env,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT * FROM loop

REPL

Traditionally, Lisp programs run in a Read-Print-Eval loop, or REPL for short. So far, we have covered the read, print, and eval parts. What about the loop part?

We could easily slap another opcode into the query, but that would be unsportmanlike and against the spirit of Lisp. We have to code the REPL loop in Mal itself.

To run Mal programs in our interpreter, we need to get Mal programs in there, and we can't get Mal programs in there before we can write a REPL loop in Mal to run them. We're having a sort of chicken-and-egg problem here.

This is, of course, not new, so we need to put some bootstrap code on the stack manually. (As a side remark, that's exactly how "bootstrapping" came to mean "self-initialization": the program needs to lift itself by pulling by its own bootstraps, so to speak). But it has to be the bare minimum.

For this purpose, we'll create just one new opcode called #read-input. It will read a string from the initialization part of the query. We'll also put bootstrap code right on the stack, as a string with a Mal program. There will be only three opcodes on the stack: [["eval"], ["read-form"], ["tokenize"]], plus the query will dynamically add the arguments for tokenize.

How do we implement a REPL loop in Mal?

Normally, we would read forms in a loop, call eval, and print results. Mal, however, doesn't have loops out of the box. We could use recursion instead of loops, but there's a problem: we want all our evals to run in the same environment, so that later evals could see variables introduced by the previous evals that ran in the loop. Each recursion invocation creates its own environment, though. If we do it this way, eval environments get isolated, and further evals won't see the side effects of the previous ones.

Here's where macros get really useful.

On macros

In Lisps, macros are a way to generate Lisp programs dynamically, on the fly, and evaluate them right there and then. Other languages sometimes support this (to an extent), but usually don't treat this functionality as a first-class citizen.

What are some examples of that?

• In procedural extensions to database languages (PL/SQL, pl/pgsql, TSQL and others), you can compose a string with SQL or procedural code in it, and call it. This is called "dynamic SQL". Everyone does it, and everyone is embarrassed to admit it, because it feels kludgy.
• In non-procedural SQL, there's no concept of the sort.
• In JavaScript, there's eval. When you take into account the, how do I put it, lispness on JavaScript, it's not surprising at all.

  eval, though, is the pariah of the JavaScript world. It's there, but it's shunned, not allowed in the clean part of the house, and is not to be mentioned at the dinner table.

• In C#, you can create instances of the Expression class, which stores the AST of C# programs, by writing C# code right inside your source files, without converting it to a string first.

  You can then compile the expressions, which will turn them into executable delegates. Or you can use them in a pattern known as "fluent builders", which means using pieces of C# AST to build expressions in another language, such as SQL, MongoDB, Elasticsearch, and such like.

  Unlike other popular languages, building, evaluating, and using AST at runtime is not stigmatized in C#. That's in no small way due to the ability to generate language AST right in the language itself, at runtime, without an intermediate string representation. It's a very lispy thing to do.

It is sometimes said that as languages evolve, they inevitably converge to Lisp. This is, apparently, because their creators keep adding features that existed in Lisp for decades. C# is a shiny example of that.

The integrated ability to generate and use ASP at runtime was added in C# 3.0 (2007).

What about design time? There's an ability to have your editor run C# code every time you press a key in the editor. This code evaluates the AST of your program as you type it, and annotates it with squiggly lines: hints, warnings, and errors. It's called "Roslyn analyzers", and it made it to production in 2015.

How about build time? There's another feature of C# compilers, called "source generators". They, too, analyze the AST of the program you build, and create new ASTs, which they add into the build source tree. They were added in C# 9.0 (2020).

Even now, writing Roslyn analyzers and source code generators, and integrating them with your project, is not for the faint of heart. They require quite a lot of boilerplate.

JavaScript has Babel, and writing Babel code is not a pleasant experience at all. But at least it's there (since 2014).

Lisp had it all for decades, built right inside the language, as part of its core functionality. The reason for that is that Lisp programs are not strings (as they are in other languages), but lists, just like Lisp data. Lisp makes it really easy to have no distinction between code and data, and it's up to the programmer to draw the line.

Let me get on my soapbox again, just for a little while.

In the lambda calculus, all you have is an endless sea of functions pointlessly flowing into one another. To make them useful, you need to take one function by the tail and say: "From now on, you're special, and you mean something". (Usually, it's the identity function.) Only then, and not before, do they get a purpose in their life and start making sense.

Lisp is the same. Thank you for coming to my TED talk.

REPL in Mal

Here's the REPL code which we'll be using to bootstrap our interpreter:

((let* ()
    (do
      (def! reader (tokenize (read-input)))
      (def! read
        (fn* () (read-form reader)))
      (defmacro! repl
        (fn* ()
          (do
            (def! form (read))
            (if (= eof form)
              eof
              `(do
                (print (try* ~form (catch* exception exception)))
                (~repl)))))))))

Let's go through it piece by piece:

(def! reader (tokenize (read-input)))

(tokenize (read-input)) returns an instance of Reader, and we're assigning it to the variable reader.

(def! read
  (fn* () (read-form reader)))

Calling (read-form reader) returns the next form from this reader (and updates its internal state). We don't want to expose the reader to the outside. So we expose a generator: a function that closes over reader. The generator is called without parameters, and yet produces a new value each time.

We're saving the generator in the variable read. We can now call it like this: (read), which will evaluate to the next form from the input string.

(fn* ()
  (do
    (def! form (read))
    (if (= eof form)
      eof
      `(do
        (print (try* ~form (catch* exception exception)))
        (~repl)))))

This is a function that evaluates to an unparsed form (AST). The backtick ` on line 6 is a shorthand syntax for the special form quasiquote. It tells the evaluator that it should avoid evaluating the AST, unless specially instructed to do so. It is instructed to do so on lines 7 and 8, with the special form unquote (for which the tilde ~ is shorthand).

This function doesn't have any parameters, but closes over the variables repl and read, which come from the environment where the function was defined.

(defmacro! repl
  (fn* ()
    (do
      (def! form (read))
      (if (= eof form)
        eof
        `(do
          (print (try* ~form (catch* exception exception)))
          (~repl))))))

Here's where the magic is happening. Macro transforms a regular function into something called a "macro function". Calling a macro function is a two-stage process.

First, the evaluator calls the function that was provided as a source for the macro. The arguments to this function are pieces of unparsed AST that go after the macro call.

Instead of producing a value, the macro-backing function produces an AST. But, unlike the AST that comes from read-form, it can contain already evaluated functions, macros, and other objects: something that read-form can't produce. These values usually come from the environment that the function had closed over when it was created.

Second, the evaluator evaluates the new AST as it normally would. The evaluation happens in the same environment the macro function was called.

There's no way in Mal to define a macro object without putting it into the environment. It means that if we want our default REPL environment to be free from pollution, we should come up with something.

Here's how we do it:

((let* ()
    (do
      (def! reader …)
      (def! read …)
      (defmacro! repl …)))

We are wrapping the code that creates the variables read and repl into a let* form. This form creates a new environment, which repl closes over at the time of its creation.

Now, here's the trick. We can't create a macro without putting it into an environment. But defmacro!, besides creating a macro, also returns it. So the output of the chain (let* () (do … (defmacro! (…)))) will be a macro.

All that's left to do is immediately apply this parameterless macro. It can be just by wrapping the let* form into an additional pair of parentheses.

This way, all the forms that come from the input will be evaluated in the default environments. At the same time, it will be free from pollution. No variables will get created on the default environment except those that come from the evaluations of the user code.

Running the REPL

Let's try evaluate multiple expressions that come from the input:

(def! plus1 (fn* (x) (+ 1 x)))
(def! times2 (fn* (x) (* x 2)))
(def! result (plus1 (times2 3))

We'll only show the lines that do have output. In addition, we'll dump the list of symbols set in the default environment.

WITH    RECURSIVE
        constants AS
        (
        SELECT  ARRAY[
                'tokenize', 'print', 'to-json', 'read-input', 'read-form', 'throw', 'eval', 'list', 'cons', 'concat', 'pr-str',
                '+', '-', '*', '/', '='] AS builtin_functions,
                ARRAY['fn*', 'def!', 'let*', 'if', 'do', 'quote', 'quasiquote', 'defmacro!', 'try*'] AS special_forms,
                ARRAY['(', ')', '[', ']', '{', '}', '''', '`', '~', '~@'] AS symbols,
                TO_JSONB('('::TEXT) AS c_open_paren, TO_JSONB(')'::TEXT) AS c_closed_paren,
                TO_JSONB('['::TEXT) AS c_open_bracket, TO_JSONB(']'::TEXT) AS c_closed_bracket,
                TO_JSONB('{'::TEXT) AS c_open_curly, TO_JSONB('}'::TEXT) AS c_closed_curly,
                TO_JSONB('unquote'::TEXT) AS c_unquote, TO_JSONB('splice-unquote'::TEXT) AS c_splice_unquote,
                'null'::JSONB AS c_null, '{"t": "eof"}'::JSONB AS c_eof,
                '{"''": "quote", "`": "quasiquote", "~": "unquote", "~@": "splice-unquote"}'::JSONB AS c_reader_macros
        ),
        bootstrap (code, input) AS
        (
        SELECT  '
((let* ()
    (do
      (def! reader (tokenize (read-input)))
      (def! read
        (fn* () (read-form reader)))
      (defmacro! repl
        (fn* ()
          (do
            (def! form (read))
            (if (= eof form)
              eof
              `(do
                (print (try* ~form (catch* exception exception)))
                (~repl)))))))))
        ',
        '
(def! plus1 (fn* (x) (+ 1 x)))
(def! times2 (fn* (x) (* x 2)))
undefined-variable
(def! result (plus1 (times2 3)))
        '
        ),
        loop (heap, stack, env, output, step) AS
        (
        SELECT  '[[null, {"eof": {"t": "eof"}}]]'::JSONB,
                JSONB_INSERT('[["eval"], ["read-form"], ["tokenize"]]'::JSONB, '{-1, -1}', (SELECT JSONB_BUILD_OBJECT('t', 'str', 'v', TO_JSONB(code)) FROM bootstrap), TRUE),
                0, NULL::TEXT, 1
        FROM    constants
        UNION ALL
        SELECT  new.*, step + 1
        FROM    loop
        CROSS JOIN constants
        CROSS JOIN LATERAL (SELECT stack #> '{-1, 0}', JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]'), stack['-1'], stack #>> '{-2, 0}' = '#pop-env') q_call (callable, args, current_frame, can_tco)
        CROSS JOIN LATERAL
                (
                SELECT  CASE
                        WHEN JSONB_TYPEOF(callable) = 'string' THEN callable #>> '{}'
                        WHEN callable ->> 't' = 'fn*' THEN '#fn-apply'
                        WHEN callable ->> 't' = 'macro' THEN '#macro-apply'
                        END
                ) q_opcode (opcode)
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#tokenize', args[0])) AS calls,
                        CASE WHEN (args #>> '{0, t}' = 'str') IS DISTINCT FROM TRUE THEN 'Usage: (tokenize string)' END AS throws
                ) tokenize ON (opcode = 'tokenize')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN args[1] IS NOT NULL THEN JSONB_BUILD_OBJECT('t', 'reader', 'v', args[1]) END AS ret,
                        CASE WHEN args[1] IS NULL THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#add-heap', JSONB_BUILD_ARRAY(0, tokens))) END AS calls
                FROM    (
                        SELECT  COALESCE(JSONB_AGG(TO_JSONB(token) ORDER BY index), '[]'::JSONB)
                        FROM    REGEXP_MATCHES(args #>> '{0, v}', '[\s,]*(~@|[\[\]{}()''`~^@]|"(?:\\.|[^\\"])*"?|;.*|[^\s\[\]{}(''"`~^,;)]*)', 'gm') WITH ORDINALITY AS q (match, index)
                        CROSS JOIN LATERAL (SELECT match[1]) q_token (token)
                        WHERE   token > '' AND NOT token ^@ ';'
                        ) q_tokens (tokens)
                ) tokenize2 ON (opcode = '#tokenize')
        LEFT JOIN LATERAL (SELECT JSONB_BUILD_OBJECT('t', 'str', 'v', TO_JSONB(input)) AS ret FROM bootstrap) read_input ON (opcode = 'read-input')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN is_string THEN c_null END AS ret,
                        CASE WHEN is_string THEN args #>> '{0, v}' END new_output,
                        CASE WHEN NOT is_string THEN JSONB_BUILD_ARRAY(JSON_BUILD_ARRAY(opcode), JSONB_BUILD_ARRAY('pr-str', COALESCE(args[0], c_null))) END AS calls
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        LEFT JOIN LATERAL (SELECT TO_JSONB(args['0']::TEXT) AS ret) _to_json ON (opcode = 'to-json')
        LEFT JOIN LATERAL
                (
                SELECT  *,
                        CASE
                        WHEN type = 'array' AND JSONB_ARRAY_LENGTH(value) > 0 THEN NULL
                        ELSE JSONB_BUILD_OBJECT('t', 'str', 'v', TO_JSONB(
                                CASE
                                WHEN type IN ('number', 'boolean') THEN value::TEXT
                                WHEN type = 'string' THEN value #>> '{}'
                                WHEN type = 'null' THEN 'nil'
                                WHEN complex_type = 'str' THEN (value -> 'v')::TEXT
                                WHEN type = 'array' AND JSONB_ARRAY_LENGTH(value) = 0 THEN
                                (
                                SELECT  '(' || COALESCE(STRING_AGG(element #>> '{v}', ' ' ORDER BY index), '') || ')'
                                FROM    JSONB_ARRAY_ELEMENTS(rest) WITH ORDINALITY q (element, index)
                                )
                                WHEN complex_type IS NOT NULL THEN FORMAT('#<%s>', complex_type)
                                ELSE 'Unprintable value'
                                END))
                        END
                        AS ret,
                        CASE WHEN type = 'array' AND JSONB_ARRAY_LENGTH(value) > 0 THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('pr-str', value['0'])) END AS calls
                FROM    (SELECT args[0], JSONB_TYPEOF(args[0]), args[0] ->> 't', JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]')) q (value, type, complex_type, rest)
                ) pr_str ON (opcode = 'pr-str')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN catch_index IS NOT NULL THEN
                                JSONB_PATH_QUERY_ARRAY(stack, '$[0 to $index - 1]', JSONB_BUILD_OBJECT('index', catch_index)) ||
                                JSONB_BUILD_ARRAY(JSONB_SET(stack[catch_index], '{1}', 'true'::JSONB) || COALESCE(args[0], c_null))
                        ELSE JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('print', args[0]))
                        END
                        AS new_stack
                FROM    (SELECT) q
                LEFT JOIN LATERAL
                        (
                        SELECT  index::INT - 1
                        FROM    JSONB_ARRAY_ELEMENTS(stack) WITH ORDINALITY q_frames (frame, index)
                        WHERE   (frame ->> 0) = '#catch*'
                        ORDER BY
                                index DESC
                        LIMIT 1
                        ) q_catch_index (catch_index) ON TRUE
                ) throw ON (opcode = 'throw')
        LEFT JOIN LATERAL
                (
                SELECT  new_heap, TO_JSONB(JSONB_ARRAY_LENGTH(new_heap) - 1) ret
                FROM    (SELECT JSONB_INSERT(heap, '{-1}', args[0], TRUE)) AS q_heap (new_heap)
                ) add_heap ON (opcode = '#add-heap')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN value IS NULL THEN c_eof
                        WHEN value = 'nil' THEN c_null
                        WHEN value ~ '^[-+]?\d+' OR value IN ('true', 'false') THEN value::JSONB
                        WHEN value ^@ '"' THEN JSONB_BUILD_OBJECT('t', 'str', 'v', value::JSONB)
                        ELSE TO_JSONB(value)
                        END AS ret,
                        CASE WHEN value IS NOT NULL THEN JSONB_SET(heap, ARRAY[reader_ptr, '0'], TO_JSONB(position + 1)) END AS new_heap
                FROM    (SELECT args['0']) AS q_reader_ref (reader_ref)
                CROSS JOIN LATERAL (SELECT reader_ref ->> 'v') AS q_reader_ptr (reader_ptr)
                CROSS JOIN LATERAL (SELECT heap[reader_ptr]) q_reader (reader)
                CROSS JOIN LATERAL (SELECT CASE WHEN JSONB_TYPEOF(reader['0']) = 'number' THEN (reader['0'] #> '{}')::INT END) q_position (position)
                CROSS JOIN LATERAL (SELECT reader['1'] ->> position) AS q_value (value)
                ) next_reader ON (opcode = '#next-reader')
        LEFT JOIN LATERAL
                (
                SELECT  ret,
                        CASE WHEN ret IS NULL THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#read-form', reader_ref)) END AS calls,
                        CASE
                        WHEN (reader_ref ->> 't') = 'reader' IS DISTINCT FROM TRUE THEN 'Invalid reader reference'
                        WHEN ret #>> '{}' IN (')', ']', '}') THEN 'Unbalanced parens'
                        END AS throws
                FROM    (SELECT  args[0], args[1]) q_args (reader_ref, ret)
                ) read_form ON (opcode = 'read-form')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN calls IS NOT NULL THEN NULL
                        WHEN is_open_paren AND is_closed_paren THEN chunk
                        WHEN is_open_bracket AND is_closed_bracket THEN JSONB_BUILD_OBJECT('t', 'vec', 'v', chunk)
                        WHEN is_open_curly AND is_closed_curly AND JSONB_ARRAY_LENGTH(chunk) % 2 = 0 THEN
                                (
                                SELECT  JSONB_BUILD_OBJECT('t', 'map', 'v', COALESCE(JSONB_OBJECT_AGG(key #>> '{}', value), '{}'::JSONB))
                                FROM    JSONB_ARRAY_ELEMENTS(chunk) WITH ORDINALITY keys (key, index)
                                JOIN    JSONB_ARRAY_ELEMENTS(chunk) WITH ORDINALITY values (value, index)
                                ON      values.index = keys.index + 1
                                WHERE   keys.index % 2 = 1
                                )
                        WHEN NOT is_open THEN first_token
                        END AS ret,
                        calls,
                        CASE
                        WHEN (is_open AND is_eof) OR
                             (is_closed AND (
                                     (is_open_paren AND NOT is_closed_paren) OR
                                     (is_open_bracket AND NOT is_closed_bracket) OR
                                     (is_open_curly AND NOT is_closed_curly)
                                     )) THEN 'Unbalanced parens'
                        WHEN is_open_curly AND NOT is_closed_curly AND JSONB_ARRAY_LENGTH(chunk) % 2 = 1 AND JSONB_TYPEOF(chunk[-1]) <> 'string' THEN
                                FORMAT('Cannot use token "%s" as a map key', last_token)
                        END AS throws
                FROM    (SELECT  args[0] reader_ref, args[1] first_token, args[-1] last_token, JSONB_PATH_QUERY_ARRAY(args, '$[2 to last - 1]') chunk) q_tokens
                CROSS JOIN LATERAL
                        (
                        SELECT  COALESCE(first_token = c_open_paren, FALSE) is_open_paren,
                                COALESCE(last_token = c_closed_paren, FALSE) is_closed_paren,
                                COALESCE(first_token = c_open_bracket, FALSE) is_open_bracket,
                                COALESCE(last_token = c_closed_bracket, FALSE) is_closed_bracket,
                                COALESCE(first_token = c_open_curly, FALSE) is_open_curly,
                                COALESCE(last_token = c_closed_curly, FALSE) is_closed_curly,
                                COALESCE(last_token = c_eof, FALSE) is_eof,
                                c_reader_macros ->> (first_token #>> '{}') reader_macro_form
                        ) q_state
                CROSS JOIN LATERAL
                        (
                        SELECT  is_open_paren OR is_open_curly OR is_open_bracket AS is_open,
                                is_closed_paren OR is_closed_curly OR is_closed_bracket AS is_closed
                        ) q_state2
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                WHEN first_token IS NULL THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#next-reader', reader_ref))
                                WHEN is_open AND NOT (is_closed OR is_eof) THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#read-form', reader_ref))
                                WHEN reader_macro_form IS NOT NULL THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('list', reader_macro_form), JSONB_BUILD_ARRAY('#read-form', reader_ref))
                                END
                        ) q_calls (calls)
                ) read_form2 ON (opcode = '#read-form')
        LEFT JOIN LATERAL (SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', COALESCE(args[0], c_null))) AS calls) eval ON (opcode = 'eval')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN eval_calls IS NULL THEN eval_ast END AS ret,
                        eval_calls AS calls
                FROM    (SELECT args['0'], JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]')) q (eval_ast, eval_ret)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(eval_ast) = 'array', JSONB_TYPEOF(eval_ast) = 'string') q_array (ast_is_array, ast_is_symbol)
                CROSS JOIN LATERAL (SELECT CASE WHEN ast_is_array THEN JSONB_ARRAY_LENGTH(eval_ast) = 0 ELSE FALSE END, JSONB_ARRAY_LENGTH(eval_ret) = 0) q_empty (ast_array_empty, ret_empty)
                CROSS JOIN LATERAL
                        (
                        SELECT  COALESCE(eval_ast #>> '{0}' = ANY(special_forms), FALSE),
                                COALESCE(eval_ast #>> '{}' = ANY(builtin_functions), FALSE)
                        ) q_callable (is_special_form, is_builtin_function)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                WHEN ret_empty AND is_special_form THEN JSONB_BUILD_ARRAY(eval_ast)
                                WHEN ast_is_array AND NOT ast_array_empty THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#eval', eval_ast[0]))
                                WHEN ast_array_empty AND NOT ret_empty THEN JSONB_BUILD_ARRAY(eval_ret)
                                WHEN ast_is_symbol AND NOT is_builtin_function THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#get-env', eval_ast))
                                END
                        ) q_calls (eval_calls)
                ) eval2 ON (opcode = '#eval')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_OBJECT('t', 'fn*', 'v', JSONB_BUILD_ARRAY(env, args[0], COALESCE(args[1], c_null))) ret,
                        CASE
                        WHEN JSONB_TYPEOF(args[0]) <> 'array' OR EXISTS (SELECT FROM JSONB_ARRAY_ELEMENTS(args[0]) arg WHERE JSONB_TYPEOF(arg) <> 'string') THEN
                                'The first argument to fn* should be a list of symbols'
                        END throws
                ) fn ON (opcode = 'fn*')
        LEFT JOIN LATERAL
                (
                SELECT  ret,
                        CASE WHEN ret IS NULL THEN FORMAT('Error applying operator: (%s %s %s)', opcode, args -> 0, args -> 1) END AS throws
                FROM    (
                        SELECT  CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0) END::DOUBLE PRECISION,
                                CASE WHEN JSONB_TYPEOF(args['1']) = 'number' THEN (args ->> 1) END::DOUBLE PRECISION
                        ) q_args (arg1, arg2)
                LEFT JOIN LATERAL
                        (
                        SELECT  TO_JSONB(
                                CASE opcode
                                WHEN '+' THEN TO_JSONB(arg1 + arg2)
                                WHEN '-' THEN TO_JSONB(arg1 - arg2)
                                WHEN '*' THEN TO_JSONB(arg1 * arg2)
                                WHEN '/' THEN TO_JSONB(arg1 / arg2)
                                END
                                )
                        ) q_ret (ret) ON TRUE
                ) math_ops ON (opcode IN ('+', '-', '*', '/'))
        LEFT JOIN LATERAL
                (
                SELECT  TO_JSONB(COALESCE(args[0] = args[1], FALSE)) AS ret
                ) eq ON (opcode = '=')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN can_tco THEN next_frames ELSE JSONB_INSERT(next_frames, '{0}', JSONB_BUILD_ARRAY('#pop-env', env), FALSE) END AS calls
                FROM    (SELECT CASE WHEN callable ->> 't' = 'fn*' THEN (callable #>> '{v, 0}') END::INT) AS q_call_args (call_env)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_BUILD_ARRAY(
                                        JSONB_BUILD_ARRAY('#eval', callable #> '{v, 2}'),
                                        JSONB_BUILD_ARRAY('#fill-args', callable #> '{v, 1}', args),
                                        JSONB_BUILD_ARRAY('#push-env', call_env)
                                )
                        ) AS q_next_frames (next_frames)
                ) fn_apply ON (callable ->> 't' = 'fn*')
        LEFT JOIN LATERAL (SELECT JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval'), JSONB_SET(current_frame, '{0, t}', TO_JSONB('fn*'::TEXT))) AS calls) macro_apply ON (callable ->> 't' = 'macro')
        LEFT JOIN LATERAL
                (
                SELECT  value AS ret,
                        CASE WHEN value IS NULL THEN FORMAT('Variable %s not found', key) END AS throws
                FROM    (SELECT args ->> 0) AS q_key (key)
                LEFT JOIN LATERAL
                        (
                        WITH    RECURSIVE get (current_env) AS
                                (
                                SELECT  heap -> env
                                UNION ALL
                                SELECT  heap -> (current_env ->> 0)::INT
                                FROM    get
                                WHERE   current_env IS NOT NULL
                                )
                        SELECT  vals -> key
                        FROM    get
                        CROSS JOIN LATERAL (SELECT current_env -> 1) q (vals)
                        WHERE   vals ? key
                        LIMIT   1
                        ) AS env_value (value) ON TRUE
                ) get_env ON (opcode = '#get-env')
        LEFT JOIN LATERAL
                (
                SELECT  value AS ret,
                        JSONB_SET(heap, ARRAY[env::TEXT, 1::TEXT, key], value, TRUE) AS new_heap
                FROM    (SELECT args ->> 0, args['1']) q (key, value)
                ) set_env ON (opcode = '#set-env')
        LEFT JOIN LATERAL
                (
                SELECT  new_env,
                        CASE
                        WHEN new_env IS NULL THEN
                                JSONB_BUILD_ARRAY(
                                        current_frame,
                                        JSONB_BUILD_ARRAY('#add-heap', JSONB_BUILD_ARRAY(old_env, JSONB_BUILD_OBJECT()))
                                )
                        END AS calls
                FROM    (SELECT CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0)::INT END, CASE WHEN JSONB_TYPEOF(args['1']) = 'number' THEN (args ->> 1)::INT END) q_args (old_env, new_env)
                ) push_env ON (opcode = '#push-env')
        LEFT JOIN LATERAL (SELECT CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0)::INT END new_env, args['1'] ret) pop_env ON (opcode = '#pop-env')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_SET(heap, ARRAY[env::TEXT, 1::TEXT], env_data || new_data) AS new_heap
                FROM    (
                        SELECT  CASE WHEN opcode = '#fill-args' AND JSONB_TYPEOF(args['0']) = 'array' THEN args[0] END,
                                CASE WHEN opcode = '#fill-args' AND JSONB_TYPEOF(args['1'])  = 'array' THEN args[1] END
                        ) q_args (keys, values)
                CROSS JOIN LATERAL (SELECT heap[env::TEXT]['1']) q_env_data (env_data)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_OBJECT_AGG(key #>> '{}', COALESCE(value, c_null) ORDER BY index)
                        FROM    JSONB_ARRAY_ELEMENTS(keys) WITH ORDINALITY q_keys (key, index)
                        LEFT JOIN
                                JSONB_ARRAY_ELEMENTS(values) WITH ORDINALITY q_values (value, index)
                        USING   (index)
                        ) q_new_data (new_data)
                ) fill_args ON (opcode = '#fill-args')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN can_tco THEN next_frames
                        ELSE JSONB_INSERT(next_frames, '{0}', JSONB_BUILD_ARRAY('#pop-env', env), FALSE)
                        END AS calls,
                        CASE
                        WHEN NOT vars_is_array THEN 'The first argument to let* should be a list'
                        WHEN body IS NULL THEN 'There should be two arguments to let*'
                        END AS throws
                FROM    (SELECT args['0'] IS NOT NULL AND JSONB_TYPEOF(args['0']) = 'array', args['0'], args['1']) q_vars_array (vars_is_array, vars, body)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_BUILD_ARRAY(
                                        JSONB_BUILD_ARRAY('#let*', vars, body),
                                        JSONB_BUILD_ARRAY('#push-env', env)
                                )
                        ) q_next_frames (next_frames)
                ) let ON (opcode = 'let*')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN vars_is_array AND JSONB_ARRAY_LENGTH(vars) = 0 THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', body))
                        ELSE JSONB_BUILD_ARRAY(
                                JSONB_SET(current_frame, '{1}', JSONB_PATH_QUERY_ARRAY(vars, '$[2 to last]')),
                                JSONB_BUILD_ARRAY('#set-env', key ->> 'v'),
                                JSONB_BUILD_ARRAY('#eval', value)
                        )
                        END AS calls,
                        CASE WHEN JSONB_TYPEOF(key) <> 'string' THEN 'let*: variable names should be symbols' END AS throws
                FROM    (SELECT args[0], args[1]) q_vars_body (vars, body)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(vars) = 'array', vars[0], COALESCE(vars[1], c_null)) q_key_value (vars_is_array, key, value)
                ) let2 ON (opcode = '#let*')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#set-env', key), JSONB_BUILD_ARRAY('#eval', value)) AS calls,
                        CASE
                        WHEN (JSONB_TYPEOF(key) = 'string') IS DISTINCT FROM TRUE THEN 'def!: variable name should be a symbol'
                        WHEN value IS NULL THEN 'def!: value should be provided'
                        END AS throws
                FROM    (SELECT args[0], args[1]) q_vars (key, value)
                ) def ON (opcode = 'def!')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#set-env', key), JSONB_BUILD_ARRAY('#fn-to-macro'), JSONB_BUILD_ARRAY('#eval', value)) AS calls,
                        CASE
                        WHEN JSONB_TYPEOF(key) = 'string' IS DISTINCT FROM TRUE THEN 'defmacro!: variable name should be a symbol'
                        END AS throws
                FROM    (SELECT args[0], args[1]) q_vars (key, value)
                ) defmacro ON (opcode = 'defmacro!')
        LEFT JOIN LATERAL
                (
                SELECT  CASE args #>> '{0, t}'
                        WHEN 'macro' THEN args[0]
                        WHEN 'fn*' THEN JSONB_SET(args[0], '{t}', TO_JSONB('macro'::TEXT), FALSE)
                        END AS ret,
                        CASE
                        WHEN NOT COALESCE(args #>> '{0, t}' IN ('fn*', 'macro'), FALSE) THEN 'defmacro!: the second argument should evaluate to a function or a macro'
                        END AS throws
                ) macro ON (opcode = '#fn-to-macro')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(
                                JSONB_BUILD_ARRAY('#if2', yes, COALESCE(no, c_null)),
                                JSONB_BUILD_ARRAY('#eval', condition)
                        ) AS calls,
                        CASE WHEN condition IS NULL OR yes IS NULL THEN 'if should have at least two arguments' END throws
                FROM    (SELECT args[0], args[1], args[2]) q_args (condition, yes, no)
                ) if ON (opcode = 'if')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN result IS NULL OR result IN (c_null, 'false'::JSONB) THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', no))
                        ELSE JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', yes))
                        END AS calls
                FROM    (SELECT args[0], args[1], args[2]) q_args (yes, no, result)
                ) if2 ON (opcode = '#if2')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN total > 1 THEN
                                JSONB_BUILD_ARRAY(
                                        current_frame - 1,
                                        JSONB_BUILD_ARRAY('#nop'),
                                        eval_frame
                                )
                        ELSE    JSONB_BUILD_ARRAY(eval_frame)
                        END AS calls
                FROM    (SELECT args['0'], JSONB_ARRAY_LENGTH(args)) AS q_args (next, total)
                CROSS JOIN LATERAL (SELECT JSONB_BUILD_ARRAY('#eval', COALESCE(next, c_null))) AS q_eval_frame (eval_frame)
                ) _do ON (opcode = 'do')
        LEFT JOIN LATERAL (SELECT  COALESCE(args[0], c_null) AS ret) quote ON (opcode = 'quote')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN NOT is_array THEN COALESCE(value, '[]'::JSONB)
                        WHEN is_array AND JSONB_ARRAY_LENGTH(value) = 0 THEN JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]')
                        END ret,
                        CASE
                        WHEN maybe_special = c_unquote THEN JSONB_BUILD_ARRAY(patched_frame, JSONB_BUILD_ARRAY('eval', form[1]))
                        WHEN maybe_special = c_splice_unquote THEN JSONB_BUILD_ARRAY(patched_frame, JSONB_BUILD_ARRAY('#splice'), JSONB_BUILD_ARRAY('eval', form[1]))
                        ELSE JSONB_BUILD_ARRAY(patched_frame, JSONB_BUILD_ARRAY(opcode, form))
                        END calls
                FROM    (SELECT args[0], args[0][0], args[0][0][0], COALESCE(JSONB_TYPEOF(args[0]) = 'array', FALSE)) q_args (value, form, maybe_special, is_array)
                CROSS JOIN LATERAL (SELECT current_frame #- '{1, 0}') q_patched_frame (patched_frame)
                ) quasiquote ON (opcode = 'quasiquote')
        LEFT JOIN LATERAL (SELECT JSONB_SET(stack - (-1), '{-1}', stack['-2'] || args['0']) AS new_stack) splice ON (opcode = '#splice')
        LEFT JOIN LATERAL (SELECT args AS ret) list ON (opcode = 'list')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(head) || list AS ret,
                        CASE WHEN is_list IS DISTINCT FROM TRUE THEN 'Second argument to cons should be a list' END AS throws
                FROM    (SELECT args['0'], args['1'], JSONB_TYPEOF(args['1']) = 'array') AS q_args (head, list, is_list)
                ) cons ON (opcode = 'cons')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN NOT has_non_arrays THEN
                                (
                                SELECT  JSONB_AGG(element ORDER BY arg_index, element_index)
                                FROM    JSONB_ARRAY_ELEMENTS(args) WITH ORDINALITY q_args (arg, arg_index)
                                CROSS JOIN LATERAL
                                        JSONB_ARRAY_ELEMENTS(arg) WITH ORDINALITY q_element (element, element_index)
                                )
                        END AS ret,
                        CASE WHEN has_non_arrays THEN 'All arguments to concat should be lists' END AS throws
                FROM    (
                        SELECT  COALESCE(BOOL_OR(JSONB_TYPEOF(arg) <> 'array'), FALSE)
                        FROM    JSONB_ARRAY_ELEMENTS(args) arg
                        ) q (has_non_arrays)
                ) _concat ON (opcode = 'concat')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(
                                JSONB_BUILD_ARRAY('#catch*', FALSE, env, exception_symbol, handler_body),
                                JSONB_BUILD_ARRAY('#eval', args[0])
                                ) AS calls,
                        CASE WHEN NOT COALESCE(catch_symbol #>> '{}' = 'catch*', FALSE) OR NOT COALESCE(JSONB_TYPEOF(exception_symbol) = 'string', FALSE) THEN
                                'Usage: (try* form (catch* exception handler_form))'
                        END AS throws
                FROM    (SELECT args[1][0] AS catch_symbol, args[1][1] AS exception_symbol, args[1][2] AS handler_body) q_args
                ) try ON (opcode = 'try*')
        LEFT JOIN LATERAL
                (
                SELECT  CASE caught WHEN 'false'::JSONB THEN args[4] END ret,
                        CASE
                        WHEN caught = 'false'::JSONB THEN NULL
                        WHEN can_tco THEN next_frames
                        ELSE JSONB_INSERT(next_frames, '{0}', JSONB_BUILD_ARRAY('#pop-env', catch_env), FALSE)
                        END AS calls
                FROM    (SELECT args[0] caught, args[1] catch_env, args[2] exception_symbol, args[3] handler_body, args[4] value) q_args
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_BUILD_ARRAY(
                                        JSONB_BUILD_ARRAY('#eval', handler_body),
                                        JSONB_BUILD_ARRAY('#nop'),
                                        JSONB_BUILD_ARRAY('#set-env', exception_symbol, value),
                                        JSONB_BUILD_ARRAY('#push-env', catch_env)
                                )
                        ) q (next_frames)
                ) catch ON (opcode = '#catch*')
        LEFT JOIN LATERAL (SELECT) nop ON (opcode = '#nop')
        LEFT JOIN LATERAL (SELECT FORMAT('Invalid opcode: %s', callable) AS throws) invalid_opcode ON (opcode IS NULL)
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(add_heap.new_heap, next_reader.new_heap, fill_args.new_heap, set_env.new_heap, heap),
                        COALESCE(throw.new_stack, splice.new_stack),
                        COALESCE(push_env.new_env, pop_env.new_env, env),
                        COALESCE(
                                print.ret, _to_json.ret, read_input.ret,
                                pr_str.ret, read_form.ret, read_form2.ret, tokenize2.ret, add_heap.ret, next_reader.ret,
                                eval2.ret, fn.ret, macro.ret,
                                get_env.ret, math_ops.ret, eq.ret, pop_env.ret, set_env.ret,
                                quote.ret, cons.ret, list.ret, _concat.ret, quasiquote.ret, catch.ret
                                ),
                        COALESCE(
                                print.calls, tokenize.calls, tokenize2.calls, pr_str.calls, read_form.calls, read_form2.calls,
                                eval.calls, eval2.calls, fn_apply.calls, macro_apply.calls,
                                push_env.calls, let.calls, let2.calls, def.calls, defmacro.calls, if.calls, if2.calls, _do.calls,
                                quasiquote.calls, try.calls, catch.calls),
                        COALESCE(
                                tokenize.throws, get_env.throws, read_form.throws, read_form2.throws, fn.throws, math_ops.throws, let2.throws,
                                let.throws, def.throws, defmacro.throws, macro.throws, if.throws, cons.throws, _concat.throws, try.throws,
                                invalid_opcode.throws),
                        COALESCE(print.new_output)
                ) rets (new_heap, new_stack, new_env, ret, calls, throws, new_output)
        CROSS JOIN LATERAL
                (
                SELECT  rets.new_heap,
                        COALESCE(
                                rets.new_stack,
                                CASE
                                WHEN rets.throws IS NOT NULL THEN
                                        stack || JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('throw', JSONB_BUILD_OBJECT('t', 'str', 'v', rets.throws)))
                                END,
                                JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE),
                                (stack - (-1)) || COALESCE(rets.calls, '[]'::JSONB)
                        ) AS new_stack,
                        rets.new_env,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT  output, variables
FROM    loop
CROSS JOIN LATERAL (SELECT STRING_AGG(key, ', ') FROM JSONB_OBJECT_KEYS(heap[0][1]) key) q (variables)
WHERE   output IS NOT NULL

The read-print-eval sequence runs in a loop defined in Mal itself. In addition, the default environment is free from runtime pollution. All the side effects from the bootstrapping code have been swept under the rug, and the default environment remains nice and pristine.

New Year code

This implementation of Mal is, of course, far from being complete. Its default environment lacks library functions, and it's far from being able to self-host. However, we can still use it for pretty powerful computations.

As it turns out, 2026 is a happy number. It means that if we take all its digits, sum their squares, and repeat this procedure, eventually we will arrive at the number 1.

Let's write a Mal program that proves this fact. Before that, we'll need to add the definitions of the functions map and any into the REPL environment. Fortunately, by this moment, it's easy to do in Mal itself.

Here are the definitions of map and any:

(def! map
  (let* ()
    (do
      (def! _map
        (fn* (fn source result)
          (if (empty? source)
            result
            (_map fn (rest source) (concat result (list (fn (first source))))))))
      (fn* (fn source) (_map fn source ())))))
(def! any
  (fn* (predicate source)
    (if (empty? source)
      false
      (if (predicate (first source))
        true
        (any predicate (rest source))))))

And here's the code of the program that takes a number and tells if it's happy or not:

(do
  (def! square (fn* (x) (* x x)))
  (def! sum-square-digits
    (fn* (x)
      (+
        (square (% x 10))
        (if (> x 9)
          (sum-square-digits (floor (/ x 10)))
          0))))
  (def! is-happy-number
    (let* ()
      (do
        (def! _is-happy-number
          (fn* (x chain)
            (let* (next (sum-square-digits x) next-chain (concat chain (list next)))
              (if (= 1 next)
                (list true next-chain)
                (if (any (fn* (x) (= next x)) chain)
                  (list false next-chain)
                  (_is-happy-number next next-chain))))))
        (fn* (x) (_is-happy-number x (list x))))))
  (let* (number 2026
         result
         (is-happy-number number)
         happy
         (first result)
         chain
         (first (rest result)))
    (|| (pr-str number) " is " (if happy "a happy" "an unhappy") " number: " (pr-str chain))))

Let's run the program in our SQL implemenation of Mal:

WITH    RECURSIVE
        constants AS
        (
        SELECT  ARRAY[
                'tokenize', 'print', 'to-json', 'read-input', 'read-form', 'throw', 'eval', 'list', 'cons', 'concat', 'pr-str', '||',
                'first', 'rest',
                'empty?',
                '=', '+', '-', '*', '/', '%', '>', '<', 'floor', 'ceiling'] AS builtin_functions,
                ARRAY['fn*', 'def!', 'let*', 'if', 'do', 'quote', 'quasiquote', 'defmacro!', 'try*'] AS special_forms,
                ARRAY['(', ')', '[', ']', '{', '}', '''', '`', '~', '~@'] AS symbols,
                TO_JSONB('('::TEXT) AS c_open_paren, TO_JSONB(')'::TEXT) AS c_closed_paren,
                TO_JSONB('['::TEXT) AS c_open_bracket, TO_JSONB(']'::TEXT) AS c_closed_bracket,
                TO_JSONB('{'::TEXT) AS c_open_curly, TO_JSONB('}'::TEXT) AS c_closed_curly,
                TO_JSONB('unquote'::TEXT) AS c_unquote, TO_JSONB('splice-unquote'::TEXT) AS c_splice_unquote,
                'null'::JSONB AS c_null, '{"t": "eof"}'::JSONB AS c_eof,
                '{"''": "quote", "`": "quasiquote", "~": "unquote", "~@": "splice-unquote"}'::JSONB AS c_reader_macros
        ),
        bootstrap (code, input) AS
        (
        SELECT  '
(do
  (def! map
    (let* ()
      (do
        (def! _map
          (fn* (fn source result)
            (if (empty? source)
              result
              (_map fn (rest source) (concat result (list (fn (first source))))))))
        (fn* (fn source) (_map fn source ())))))
  (def! any
    (fn* (predicate source)
      (if (empty? source)
        false
        (if (predicate (first source))
          true
          (any predicate (rest source))))))
  ((let* ()
      (do
        (def! reader (tokenize (read-input)))
        (def! read
          (fn* () (read-form reader)))
        (defmacro! repl
          (fn* ()
            (do
              (def! form (read))
              (if (= eof form)
                eof
                `(do
                  (print(try* ~form (catch* exception exception)))
                  (~repl))))))))))
        ',
        '
(do
  (def! square (fn* (x) (* x x)))
  (def! sum-square-digits
    (fn* (x)
      (+
        (square (% x 10))
        (if (> x 9)
          (sum-square-digits (floor (/ x 10)))
          0))))
  (def! is-happy-number
    (let* ()
      (do
        (def! _is-happy-number
          (fn* (x chain)
            (let* (next (sum-square-digits x) next-chain (concat chain (list next)))
              (if (= 1 next)
                (list true next-chain)
                (if (any (fn* (x) (= next x)) chain)
                  (list false next-chain)
                  (_is-happy-number next next-chain))))))
        (fn* (x) (_is-happy-number x (list x))))))
  (let* (number 2026
         result
         (is-happy-number number)
         happy
         (first result)
         chain
         (first (rest result)))
    (|| (pr-str number) " is " (if happy "a happy" "an unhappy") " number: " (pr-str chain))))
        '
        ),
        loop (heap, stack, env, output, step) AS
        (
        SELECT  '[[null, {"eof": {"t": "eof"}}]]'::JSONB,
                JSONB_INSERT('[["eval"], ["read-form"], ["tokenize"]]'::JSONB, '{-1, -1}', (SELECT JSONB_BUILD_OBJECT('t', 'str', 'v', TO_JSONB(code)) FROM bootstrap), TRUE),
                0, NULL::TEXT, 1
        FROM    constants
        UNION ALL
        SELECT  new.*, step + 1
        FROM    loop
        CROSS JOIN constants
        CROSS JOIN LATERAL (SELECT stack #> '{-1, 0}', JSONB_PATH_QUERY_ARRAY(stack, '$[last][1 to last]'), stack['-1'], stack #>> '{-2, 0}' = '#pop-env') q_call (callable, args, current_frame, can_tco)
        CROSS JOIN LATERAL
                (
                SELECT  CASE
                        WHEN JSONB_TYPEOF(callable) = 'string' THEN callable #>> '{}'
                        WHEN callable ->> 't' = 'fn*' THEN '#fn-apply'
                        WHEN callable ->> 't' = 'macro' THEN '#macro-apply'
                        END
                ) q_opcode (opcode)
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#tokenize', args[0])) AS calls,
                        CASE WHEN (args #>> '{0, t}' = 'str') IS DISTINCT FROM TRUE THEN 'Usage: (tokenize string)' END AS throws
                ) tokenize ON (opcode = 'tokenize')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN args[1] IS NOT NULL THEN JSONB_BUILD_OBJECT('t', 'reader', 'v', args[1]) END AS ret,
                        CASE WHEN args[1] IS NULL THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#add-heap', JSONB_BUILD_ARRAY(0, tokens))) END AS calls
                FROM    (
                        SELECT  COALESCE(JSONB_AGG(TO_JSONB(token) ORDER BY index), '[]'::JSONB)
                        FROM    REGEXP_MATCHES(args #>> '{0, v}', '[\s,]*(~@|[\[\]{}()''`~^@]|"(?:\\.|[^\\"])*"?|;.*|[^\s\[\]{}(''"`~^,;)]*)', 'gm') WITH ORDINALITY AS q (match, index)
                        CROSS JOIN LATERAL (SELECT match[1]) q_token (token)
                        WHERE   token > '' AND NOT token ^@ ';'
                        ) q_tokens (tokens)
                ) tokenize2 ON (opcode = '#tokenize')
        LEFT JOIN LATERAL (SELECT JSONB_BUILD_OBJECT('t', 'str', 'v', TO_JSONB(input)) AS ret FROM bootstrap) read_input ON (opcode = 'read-input')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN is_string THEN c_null END AS ret,
                        CASE WHEN is_string THEN args #>> '{0, v}' END new_output,
                        CASE WHEN NOT is_string THEN JSONB_BUILD_ARRAY(JSON_BUILD_ARRAY(opcode), JSONB_BUILD_ARRAY('#pr-str', COALESCE(args[0], c_null))) END AS calls
                FROM    (SELECT COALESCE(args #>> '{0, t}' = 'str', FALSE)) q (is_string)
                ) print ON (opcode = 'print')
        LEFT JOIN LATERAL (SELECT TO_JSONB(args['0']::TEXT) AS ret) _to_json ON (opcode = 'to-json')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_OBJECT('t', 'str', 'v', COALESCE(STRING_AGG(arg ->> 'v', '' ORDER BY index), '')) AS ret
                FROM    JSONB_ARRAY_ELEMENTS(args) WITH ORDINALITY q_args (arg, index)
                ) str_concat ON (opcode = '||')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#pr-str', args[0])) calls
                ) pr_str ON (opcode = 'pr-str')
        LEFT JOIN LATERAL
                (
                SELECT  *,
                        CASE
                        WHEN type = 'array' AND JSONB_ARRAY_LENGTH(value) > 0 THEN NULL
                        ELSE JSONB_BUILD_OBJECT('t', 'str', 'v', TO_JSONB(
                                CASE
                                WHEN type IN ('number', 'boolean') THEN value::TEXT
                                WHEN type = 'string' THEN value #>> '{}'
                                WHEN type = 'null' THEN 'nil'
                                WHEN complex_type = 'str' THEN (value -> 'v')::TEXT
                                WHEN type = 'array' AND JSONB_ARRAY_LENGTH(value) = 0 THEN
                                (
                                SELECT  '(' || COALESCE(STRING_AGG(element #>> '{v}', ' ' ORDER BY index), '') || ')'
                                FROM    JSONB_ARRAY_ELEMENTS(rest) WITH ORDINALITY q (element, index)
                                )
                                WHEN complex_type IS NOT NULL THEN FORMAT('#<%s>', complex_type)
                                ELSE 'Unprintable value'
                                END))
                        END
                        AS ret,
                        CASE WHEN type = 'array' AND JSONB_ARRAY_LENGTH(value) > 0 THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#pr-str', value['0'])) END AS calls
                FROM    (SELECT args[0], JSONB_TYPEOF(args[0]), args[0] ->> 't', JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]')) q (value, type, complex_type, rest)
                ) pr_str2 ON (opcode = '#pr-str')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN catch_index IS NOT NULL THEN
                                JSONB_PATH_QUERY_ARRAY(stack, '$[0 to $index - 1]', JSONB_BUILD_OBJECT('index', catch_index)) ||
                                JSONB_BUILD_ARRAY(JSONB_SET(stack[catch_index], '{1}', 'true'::JSONB) || COALESCE(args[0], c_null))
                        ELSE JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('print', args[0]))
                        END
                        AS new_stack
                FROM    (SELECT) q
                LEFT JOIN LATERAL
                        (
                        SELECT  index::INT - 1
                        FROM    JSONB_ARRAY_ELEMENTS(stack) WITH ORDINALITY q_frames (frame, index)
                        WHERE   (frame ->> 0) = '#catch*'
                        ORDER BY
                                index DESC
                        LIMIT 1
                        ) q_catch_index (catch_index) ON TRUE
                ) throw ON (opcode = 'throw')
        LEFT JOIN LATERAL
                (
                SELECT  new_heap, TO_JSONB(JSONB_ARRAY_LENGTH(new_heap) - 1) ret
                FROM    (SELECT JSONB_INSERT(heap, '{-1}', args[0], TRUE)) AS q_heap (new_heap)
                ) add_heap ON (opcode = '#add-heap')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN value IS NULL THEN c_eof
                        WHEN value = 'nil' THEN c_null
                        WHEN value ~ '^[-+]?\d+' OR value IN ('true', 'false') THEN value::JSONB
                        WHEN value ^@ '"' THEN JSONB_BUILD_OBJECT('t', 'str', 'v', value::JSONB)
                        ELSE TO_JSONB(value)
                        END AS ret,
                        CASE WHEN value IS NOT NULL THEN JSONB_SET(heap, ARRAY[reader_ptr, '0'], TO_JSONB(position + 1)) END AS new_heap
                FROM    (SELECT args['0']) AS q_reader_ref (reader_ref)
                CROSS JOIN LATERAL (SELECT reader_ref ->> 'v') AS q_reader_ptr (reader_ptr)
                CROSS JOIN LATERAL (SELECT heap[reader_ptr]) q_reader (reader)
                CROSS JOIN LATERAL (SELECT CASE WHEN JSONB_TYPEOF(reader['0']) = 'number' THEN (reader['0'] #> '{}')::INT END) q_position (position)
                CROSS JOIN LATERAL (SELECT reader['1'] ->> position) AS q_value (value)
                ) next_reader ON (opcode = '#next-reader')
        LEFT JOIN LATERAL
                (
                SELECT  ret,
                        CASE WHEN ret IS NULL THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#read-form', reader_ref)) END AS calls,
                        CASE
                        WHEN (reader_ref ->> 't') = 'reader' IS DISTINCT FROM TRUE THEN 'Invalid reader reference'
                        WHEN ret #>> '{}' IN (')', ']', '}') THEN 'Unbalanced parens'
                        END AS throws
                FROM    (SELECT  args[0], args[1]) q_args (reader_ref, ret)
                ) read_form ON (opcode = 'read-form')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN calls IS NOT NULL THEN NULL
                        WHEN is_open_paren AND is_closed_paren THEN chunk
                        WHEN is_open_bracket AND is_closed_bracket THEN JSONB_BUILD_OBJECT('t', 'vec', 'v', chunk)
                        WHEN is_open_curly AND is_closed_curly AND JSONB_ARRAY_LENGTH(chunk) % 2 = 0 THEN
                                (
                                SELECT  JSONB_BUILD_OBJECT('t', 'map', 'v', COALESCE(JSONB_OBJECT_AGG(key #>> '{}', value), '{}'::JSONB))
                                FROM    JSONB_ARRAY_ELEMENTS(chunk) WITH ORDINALITY keys (key, index)
                                JOIN    JSONB_ARRAY_ELEMENTS(chunk) WITH ORDINALITY values (value, index)
                                ON      values.index = keys.index + 1
                                WHERE   keys.index % 2 = 1
                                )
                        WHEN NOT is_open THEN first_token
                        END AS ret,
                        calls,
                        CASE
                        WHEN (is_open AND is_eof) OR
                             (is_closed AND (
                                     (is_open_paren AND NOT is_closed_paren) OR
                                     (is_open_bracket AND NOT is_closed_bracket) OR
                                     (is_open_curly AND NOT is_closed_curly)
                                     )) THEN 'Unbalanced parens'
                        WHEN is_open_curly AND NOT is_closed_curly AND JSONB_ARRAY_LENGTH(chunk) % 2 = 1 AND JSONB_TYPEOF(chunk[-1]) <> 'string' THEN
                                FORMAT('Cannot use token "%s" as a map key', last_token)
                        END AS throws
                FROM    (SELECT  args[0] reader_ref, args[1] first_token, args[-1] last_token, JSONB_PATH_QUERY_ARRAY(args, '$[2 to last - 1]') chunk) q_tokens
                CROSS JOIN LATERAL
                        (
                        SELECT  COALESCE(first_token = c_open_paren, FALSE) is_open_paren,
                                COALESCE(last_token = c_closed_paren, FALSE) is_closed_paren,
                                COALESCE(first_token = c_open_bracket, FALSE) is_open_bracket,
                                COALESCE(last_token = c_closed_bracket, FALSE) is_closed_bracket,
                                COALESCE(first_token = c_open_curly, FALSE) is_open_curly,
                                COALESCE(last_token = c_closed_curly, FALSE) is_closed_curly,
                                COALESCE(last_token = c_eof, FALSE) is_eof,
                                c_reader_macros ->> (first_token #>> '{}') reader_macro_form
                        ) q_state
                CROSS JOIN LATERAL
                        (
                        SELECT  is_open_paren OR is_open_curly OR is_open_bracket AS is_open,
                                is_closed_paren OR is_closed_curly OR is_closed_bracket AS is_closed
                        ) q_state2
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                WHEN first_token IS NULL THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#next-reader', reader_ref))
                                WHEN is_open AND NOT (is_closed OR is_eof) THEN JSONB_BUILD_ARRAY(current_frame, JSONB_BUILD_ARRAY('#read-form', reader_ref))
                                WHEN reader_macro_form IS NOT NULL THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('list', reader_macro_form), JSONB_BUILD_ARRAY('#read-form', reader_ref))
                                END
                        ) q_calls (calls)
                ) read_form2 ON (opcode = '#read-form')
        LEFT JOIN LATERAL (SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', COALESCE(args[0], c_null))) AS calls) eval ON (opcode = 'eval')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN eval_calls IS NULL THEN eval_ast END AS ret,
                        eval_calls AS calls
                FROM    (SELECT args['0'], JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]')) q (eval_ast, eval_ret)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(eval_ast) = 'array', JSONB_TYPEOF(eval_ast) = 'string') q_array (ast_is_array, ast_is_symbol)
                CROSS JOIN LATERAL (SELECT CASE WHEN ast_is_array THEN JSONB_ARRAY_LENGTH(eval_ast) = 0 ELSE FALSE END, JSONB_ARRAY_LENGTH(eval_ret) = 0) q_empty (ast_array_empty, ret_empty)
                CROSS JOIN LATERAL
                        (
                        SELECT  COALESCE(eval_ast #>> '{0}' = ANY(special_forms), FALSE),
                                COALESCE(eval_ast #>> '{}' = ANY(builtin_functions), FALSE)
                        ) q_callable (is_special_form, is_builtin_function)
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE
                                WHEN ret_empty AND is_special_form THEN JSONB_BUILD_ARRAY(eval_ast)
                                WHEN ast_is_array AND NOT ast_array_empty THEN JSONB_BUILD_ARRAY(current_frame #- '{1, 0}', JSONB_BUILD_ARRAY('#eval', eval_ast[0]))
                                WHEN ast_array_empty AND NOT ret_empty THEN JSONB_BUILD_ARRAY(eval_ret)
                                WHEN ast_is_symbol AND NOT is_builtin_function THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#get-env', eval_ast))
                                END
                        ) q_calls (eval_calls)
                ) eval2 ON (opcode = '#eval')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_OBJECT('t', 'fn*', 'v', JSONB_BUILD_ARRAY(env, args[0], COALESCE(args[1], c_null))) ret,
                        CASE
                        WHEN JSONB_TYPEOF(args[0]) <> 'array' OR EXISTS (SELECT FROM JSONB_ARRAY_ELEMENTS(args[0]) arg WHERE JSONB_TYPEOF(arg) <> 'string') THEN
                                'The first argument to fn* should be a list of symbols'
                        END throws
                ) fn ON (opcode = 'fn*')
        LEFT JOIN LATERAL
                (
                SELECT  ret,
                        CASE WHEN ret IS NULL THEN FORMAT('Error applying operator: (%s %s %s)', opcode, args -> 0, args -> 1) END AS throws
                FROM    (
                        SELECT  CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0) END::DOUBLE PRECISION,
                                CASE WHEN JSONB_TYPEOF(args['1']) = 'number' THEN (args ->> 1) END::DOUBLE PRECISION
                        ) q_args (arg1, arg2)
                LEFT JOIN LATERAL
                        (
                        SELECT  TO_JSONB(
                                CASE opcode
                                WHEN '+' THEN TO_JSONB(arg1 + arg2)
                                WHEN '-' THEN TO_JSONB(arg1 - arg2)
                                WHEN '*' THEN TO_JSONB(arg1 * arg2)
                                WHEN '/' THEN TO_JSONB(arg1 / arg2)
                                WHEN '>' THEN TO_JSONB(arg1 > arg2)
                                WHEN '<' THEN TO_JSONB(arg1 < arg2)
                                WHEN '%' THEN TO_JSONB(arg1::INT % arg2::INT)
                                WHEN 'floor' THEN TO_JSONB(FLOOR(arg1))
                                WHEN 'ceiling' THEN TO_JSONB(CEILING(arg1))
                                END
                                )
                        ) q_ret (ret) ON TRUE
                ) math_ops ON (opcode IN ('+', '-', '*', '/', '%', '>', '<', 'floor', 'ceiling'))
        LEFT JOIN LATERAL
                (
                SELECT  TO_JSONB(COALESCE(args[0] = args[1], FALSE)) AS ret
                ) eq ON (opcode = '=')
        LEFT JOIN LATERAL
                (
                SELECT  CASE WHEN can_tco THEN next_frames ELSE JSONB_INSERT(next_frames, '{0}', JSONB_BUILD_ARRAY('#pop-env', env), FALSE) END AS calls
                FROM    (SELECT CASE WHEN callable ->> 't' = 'fn*' THEN (callable #>> '{v, 0}') END::INT) AS q_call_args (call_env)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_BUILD_ARRAY(
                                        JSONB_BUILD_ARRAY('#eval', callable #> '{v, 2}'),
                                        JSONB_BUILD_ARRAY('#fill-args', callable #> '{v, 1}', args),
                                        JSONB_BUILD_ARRAY('#push-env', call_env)
                                )
                        ) AS q_next_frames (next_frames)
                ) fn_apply ON (callable ->> 't' = 'fn*')
        LEFT JOIN LATERAL (SELECT JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval'), JSONB_SET(current_frame, '{0, t}', TO_JSONB('fn*'::TEXT))) AS calls) macro_apply ON (callable ->> 't' = 'macro')
        LEFT JOIN LATERAL
                (
                SELECT  value AS ret,
                        CASE WHEN value IS NULL THEN FORMAT('Variable %s not found', key) END AS throws
                FROM    (SELECT args ->> 0) AS q_key (key)
                LEFT JOIN LATERAL
                        (
                        WITH    RECURSIVE get (current_env) AS
                                (
                                SELECT  heap -> env
                                UNION ALL
                                SELECT  heap -> (current_env ->> 0)::INT
                                FROM    get
                                WHERE   current_env IS NOT NULL
                                )
                        SELECT  vals -> key
                        FROM    get
                        CROSS JOIN LATERAL (SELECT current_env -> 1) q (vals)
                        WHERE   vals ? key
                        LIMIT   1
                        ) AS env_value (value) ON TRUE
                ) get_env ON (opcode = '#get-env')
        LEFT JOIN LATERAL
                (
                SELECT  value AS ret,
                        JSONB_SET(heap, ARRAY[env::TEXT, 1::TEXT, key], value, TRUE) AS new_heap
                FROM    (SELECT args ->> 0, args['1']) q (key, value)
                ) set_env ON (opcode = '#set-env')
        LEFT JOIN LATERAL
                (
                SELECT  new_env,
                        CASE
                        WHEN new_env IS NULL THEN
                                JSONB_BUILD_ARRAY(
                                        current_frame,
                                        JSONB_BUILD_ARRAY('#add-heap', JSONB_BUILD_ARRAY(old_env, JSONB_BUILD_OBJECT()))
                                )
                        END AS calls
                FROM    (
                        SELECT  CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0)::DECIMAL::INT END,
                                CASE WHEN JSONB_TYPEOF(args['1']) = 'number' THEN (args ->> 1)::DECIMAL::INT END
                        ) q_args (old_env, new_env)
                ) push_env ON (opcode = '#push-env')
        LEFT JOIN LATERAL (SELECT CASE WHEN JSONB_TYPEOF(args['0']) = 'number' THEN (args ->> 0)::DECIMAL::INT END new_env, args['1'] ret) pop_env ON (opcode = '#pop-env')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_SET(heap, ARRAY[env::TEXT, 1::TEXT], env_data || new_data) AS new_heap
                FROM    (
                        SELECT  CASE WHEN opcode = '#fill-args' AND JSONB_TYPEOF(args['0']) = 'array' THEN args[0] END,
                                CASE WHEN opcode = '#fill-args' AND JSONB_TYPEOF(args['1'])  = 'array' THEN args[1] END
                        ) q_args (keys, values)
                CROSS JOIN LATERAL (SELECT heap[env::TEXT]['1']) q_env_data (env_data)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_OBJECT_AGG(key #>> '{}', COALESCE(value, c_null) ORDER BY index)
                        FROM    JSONB_ARRAY_ELEMENTS(keys) WITH ORDINALITY q_keys (key, index)
                        LEFT JOIN
                                JSONB_ARRAY_ELEMENTS(values) WITH ORDINALITY q_values (value, index)
                        USING   (index)
                        ) q_new_data (new_data)
                ) fill_args ON (opcode = '#fill-args')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN can_tco THEN next_frames
                        ELSE JSONB_INSERT(next_frames, '{0}', JSONB_BUILD_ARRAY('#pop-env', env), FALSE)
                        END AS calls,
                        CASE
                        WHEN NOT vars_is_array THEN 'The first argument to let* should be a list'
                        WHEN body IS NULL THEN 'There should be two arguments to let*'
                        END AS throws
                FROM    (SELECT args['0'] IS NOT NULL AND JSONB_TYPEOF(args['0']) = 'array', args['0'], args['1']) q_vars_array (vars_is_array, vars, body)
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_BUILD_ARRAY(
                                        JSONB_BUILD_ARRAY('#let*', vars, body),
                                        JSONB_BUILD_ARRAY('#push-env', env)
                                )
                        ) q_next_frames (next_frames)
                ) let ON (opcode = 'let*')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN vars_is_array AND JSONB_ARRAY_LENGTH(vars) = 0 THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', body))
                        ELSE JSONB_BUILD_ARRAY(
                                JSONB_SET(current_frame, '{1}', JSONB_PATH_QUERY_ARRAY(vars, '$[2 to last]')),
                                JSONB_BUILD_ARRAY('#set-env', key),
                                JSONB_BUILD_ARRAY('#eval', value)
                        )
                        END AS calls,
                        CASE WHEN JSONB_TYPEOF(key) <> 'string' THEN 'let*: variable names should be symbols' END AS throws
                FROM    (SELECT args[0], args[1]) q_vars_body (vars, body)
                CROSS JOIN LATERAL (SELECT JSONB_TYPEOF(vars) = 'array', vars[0], COALESCE(vars[1], c_null)) q_key_value (vars_is_array, key, value)
                ) let2 ON (opcode = '#let*')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#set-env', key), JSONB_BUILD_ARRAY('#eval', value)) AS calls,
                        CASE
                        WHEN (JSONB_TYPEOF(key) = 'string') IS DISTINCT FROM TRUE THEN 'def!: variable name should be a symbol'
                        WHEN value IS NULL THEN 'def!: value should be provided'
                        END AS throws
                FROM    (SELECT args[0], args[1]) q_vars (key, value)
                ) def ON (opcode = 'def!')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#set-env', key), JSONB_BUILD_ARRAY('#fn-to-macro'), JSONB_BUILD_ARRAY('#eval', value)) AS calls,
                        CASE
                        WHEN JSONB_TYPEOF(key) = 'string' IS DISTINCT FROM TRUE THEN 'defmacro!: variable name should be a symbol'
                        END AS throws
                FROM    (SELECT args[0], args[1]) q_vars (key, value)
                ) defmacro ON (opcode = 'defmacro!')
        LEFT JOIN LATERAL
                (
                SELECT  CASE args #>> '{0, t}'
                        WHEN 'macro' THEN args[0]
                        WHEN 'fn*' THEN JSONB_SET(args[0], '{t}', TO_JSONB('macro'::TEXT), FALSE)
                        END AS ret,
                        CASE
                        WHEN NOT COALESCE(args #>> '{0, t}' IN ('fn*', 'macro'), FALSE) THEN 'defmacro!: the second argument should evaluate to a function or a macro'
                        END AS throws
                ) macro ON (opcode = '#fn-to-macro')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(
                                JSONB_BUILD_ARRAY('#if2', yes, COALESCE(no, c_null)),
                                JSONB_BUILD_ARRAY('#eval', condition)
                        ) AS calls,
                        CASE WHEN condition IS NULL OR yes IS NULL THEN 'if should have at least two arguments' END throws
                FROM    (SELECT args[0], args[1], args[2]) q_args (condition, yes, no)
                ) if ON (opcode = 'if')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN result IS NULL OR result IN (c_null, 'false'::JSONB) THEN JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', no))
                        ELSE JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('#eval', yes))
                        END AS calls
                FROM    (SELECT args[0], args[1], args[2]) q_args (yes, no, result)
                ) if2 ON (opcode = '#if2')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN total > 1 THEN
                                JSONB_BUILD_ARRAY(
                                        current_frame - 1,
                                        JSONB_BUILD_ARRAY('#nop'),
                                        eval_frame
                                )
                        ELSE    JSONB_BUILD_ARRAY(eval_frame)
                        END AS calls
                FROM    (SELECT args['0'], JSONB_ARRAY_LENGTH(args)) AS q_args (next, total)
                CROSS JOIN LATERAL (SELECT JSONB_BUILD_ARRAY('#eval', COALESCE(next, c_null))) AS q_eval_frame (eval_frame)
                ) _do ON (opcode = 'do')
        LEFT JOIN LATERAL (SELECT  COALESCE(args[0], c_null) AS ret) quote ON (opcode = 'quote')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN NOT is_array THEN COALESCE(value, '[]'::JSONB)
                        WHEN is_array AND JSONB_ARRAY_LENGTH(value) = 0 THEN JSONB_PATH_QUERY_ARRAY(args, '$[1 to last]')
                        END ret,
                        CASE
                        WHEN maybe_special = c_unquote THEN JSONB_BUILD_ARRAY(patched_frame, JSONB_BUILD_ARRAY('eval', form[1]))
                        WHEN maybe_special = c_splice_unquote THEN JSONB_BUILD_ARRAY(patched_frame, JSONB_BUILD_ARRAY('#splice'), JSONB_BUILD_ARRAY('eval', form[1]))
                        ELSE JSONB_BUILD_ARRAY(patched_frame, JSONB_BUILD_ARRAY(opcode, form))
                        END calls
                FROM    (SELECT args[0], args[0][0], args[0][0][0], COALESCE(JSONB_TYPEOF(args[0]) = 'array', FALSE)) q_args (value, form, maybe_special, is_array)
                CROSS JOIN LATERAL (SELECT current_frame #- '{1, 0}') q_patched_frame (patched_frame)
                ) quasiquote ON (opcode = 'quasiquote')
        LEFT JOIN LATERAL (SELECT JSONB_SET(stack - (-1), '{-1}', stack['-2'] || args['0']) AS new_stack) splice ON (opcode = '#splice')
        LEFT JOIN LATERAL (SELECT args AS ret) list ON (opcode = 'list')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(head) || list AS ret,
                        CASE WHEN is_list IS DISTINCT FROM TRUE THEN 'Second argument to cons should be a list' END AS throws
                FROM    (SELECT args['0'], args['1'], JSONB_TYPEOF(args['1']) = 'array') AS q_args (head, list, is_list)
                ) cons ON (opcode = 'cons')
        LEFT JOIN LATERAL
                (
                SELECT  CASE
                        WHEN NOT has_non_arrays THEN
                                (
                                SELECT  JSONB_AGG(element ORDER BY arg_index, element_index)
                                FROM    JSONB_ARRAY_ELEMENTS(args) WITH ORDINALITY q_args (arg, arg_index)
                                CROSS JOIN LATERAL
                                        JSONB_ARRAY_ELEMENTS(arg) WITH ORDINALITY q_element (element, element_index)
                                )
                        END AS ret,
                        CASE WHEN has_non_arrays THEN 'All arguments to concat should be lists' END AS throws
                FROM    (
                        SELECT  COALESCE(BOOL_OR(JSONB_TYPEOF(arg) <> 'array'), FALSE)
                        FROM    JSONB_ARRAY_ELEMENTS(args) arg
                        ) q (has_non_arrays)
                ) _concat ON (opcode = 'concat')
        LEFT JOIN LATERAL
                (
                SELECT  JSONB_BUILD_ARRAY(
                                JSONB_BUILD_ARRAY('#catch*', FALSE, env, exception_symbol, handler_body),
                                JSONB_BUILD_ARRAY('#eval', args[0])
                                ) AS calls,
                        CASE WHEN NOT COALESCE(catch_symbol #>> '{}' = 'catch*', FALSE) OR NOT COALESCE(JSONB_TYPEOF(exception_symbol) = 'string', FALSE) THEN
                                'Usage: (try* form (catch* exception handler_form))'
                        END AS throws
                FROM    (SELECT args[1][0] AS catch_symbol, args[1][1] AS exception_symbol, args[1][2] AS handler_body) q_args
                ) try ON (opcode = 'try*')
        LEFT JOIN LATERAL
                (
                SELECT  CASE caught WHEN 'false'::JSONB THEN args[4] END ret,
                        CASE
                        WHEN caught = 'false'::JSONB THEN NULL
                        WHEN can_tco THEN next_frames
                        ELSE JSONB_INSERT(next_frames, '{0}', JSONB_BUILD_ARRAY('#pop-env', catch_env), FALSE)
                        END AS calls
                FROM    (SELECT args[0] caught, args[1] catch_env, args[2] exception_symbol, args[3] handler_body, args[4] value) q_args
                CROSS JOIN LATERAL
                        (
                        SELECT  JSONB_BUILD_ARRAY(
                                        JSONB_BUILD_ARRAY('#eval', handler_body),
                                        JSONB_BUILD_ARRAY('#nop'),
                                        JSONB_BUILD_ARRAY('#set-env', exception_symbol, value),
                                        JSONB_BUILD_ARRAY('#push-env', catch_env)
                                )
                        ) q (next_frames)
                ) catch ON (opcode = '#catch*')
        LEFT JOIN LATERAL (SELECT COALESCE(args[0][0], c_null) AS ret) first ON (opcode = 'first')
        LEFT JOIN LATERAL (SELECT JSONB_PATH_QUERY_ARRAY(args[0], '$[1 to last]') AS ret) rest ON (opcode = 'rest')
        LEFT JOIN LATERAL
                (
                SELECT  TO_JSONB(
                                CASE opcode
                                WHEN 'empty?' THEN NOT COALESCE(JSONB_TYPEOF(args[0]) = 'array' AND JSONB_ARRAY_LENGTH(args[0]) > 0, FALSE)
                                END
                        ) AS ret
                ) checks ON (opcode IN ('empty?'))
        LEFT JOIN LATERAL (SELECT) nop ON (opcode = '#nop')
        LEFT JOIN LATERAL (SELECT FORMAT('Invalid opcode: %s', callable) AS throws) invalid_opcode ON (opcode IS NULL)
        CROSS JOIN LATERAL
                (
                SELECT  COALESCE(add_heap.new_heap, next_reader.new_heap, fill_args.new_heap, set_env.new_heap, heap),
                        COALESCE(throw.new_stack, splice.new_stack),
                        COALESCE(push_env.new_env, pop_env.new_env, env),
                        COALESCE(
                                print.ret, _to_json.ret, read_input.ret,
                                str_concat.ret, pr_str2.ret, read_form.ret, read_form2.ret, tokenize2.ret, add_heap.ret, next_reader.ret,
                                eval2.ret, fn.ret, macro.ret,
                                get_env.ret, math_ops.ret, eq.ret, pop_env.ret, set_env.ret,
                                quote.ret, cons.ret, list.ret, _concat.ret, quasiquote.ret, catch.ret,
                                first.ret, rest.ret, checks.ret
                                ),
                        COALESCE(
                                print.calls, tokenize.calls, tokenize2.calls, pr_str.calls, pr_str2.calls, read_form.calls, read_form2.calls,
                                eval.calls, eval2.calls, fn_apply.calls, macro_apply.calls,
                                push_env.calls, let.calls, let2.calls, def.calls, defmacro.calls, if.calls, if2.calls, _do.calls,
                                quasiquote.calls, try.calls, catch.calls),
                        COALESCE(
                                tokenize.throws, get_env.throws, read_form.throws, read_form2.throws, fn.throws, math_ops.throws, let2.throws,
                                let.throws, def.throws, defmacro.throws, macro.throws, if.throws, cons.throws, _concat.throws, try.throws,
                                invalid_opcode.throws),
                        COALESCE(print.new_output)
                ) rets (new_heap, new_stack, new_env, ret, calls, throws, new_output)
        CROSS JOIN LATERAL
                (
                SELECT  rets.new_heap,
                        COALESCE(
                                rets.new_stack,
                                CASE
                                WHEN rets.throws IS NOT NULL THEN
                                        stack || JSONB_BUILD_ARRAY(JSONB_BUILD_ARRAY('throw', JSONB_BUILD_OBJECT('t', 'str', 'v', rets.throws)))
                                END,
                                JSONB_INSERT(stack - (-1), '{-1, -1}', rets.ret, TRUE),
                                (stack - (-1)) || COALESCE(rets.calls, '[]'::JSONB)
                        ) AS new_stack,
                        rets.new_env,
                        rets.new_output
                ) new
        WHERE   JSONB_ARRAY_LENGTH(stack) > 0
        )
SELECT  output, step
FROM    loop
WHERE   output IS NOT NULL

The program completes in 3685 cycles of our interpreter loop. And yes, 2026 is a happy number indeed!

I wish that your 2026 be as happy as this number!

You can find the queries in the GitHub repository: quassnoi/explain-extended-2026

Previous New Year posts:

• 2010: SQL graphics in Oracle, MySQL, SQL Server and PostgreSQL
• 2011: Drawing a clock in SQL
• 2012: Drawing snowflakes in SQL
• 2013: View of Earth from space in SQL
• 2014: Drawing fractals in SQL
• 2015: Composing music in SQL
• 2016: Conway’s Game of Life in SQL
• 2017: The Sultan’s Riddle in SQL
• 2018: Settlers of Catan in SQL
• 2019: GIF decoder in SQL
• 2020: A stereogram in SQL
• 2021: 3D picture of the coronavirus in SQL
• 2022: Quantum computer emulator in SQL
• 2023: Solving the Rubik’s Cube in SQL
• 2024: GPT in 500 lines of SQL
• 2025: Diffusion Model image generator in about 700 lines of pure SQL

The post Happy New Year: Lisp interpreter in SQL appeared first on EXPLAIN EXTENDED.

These days they say AI is taking over, and human-made art will soon go the way of the dodo. The same fate awaits SQL-made art, I'm afraid. But you can't stop the progress. If you can't beat'em, join'em. To make regular art, you need regular AI, and to make SQL art, you need SQL AI.

So today, in an effort to save SQL art from extinction, we will be implementing a program capable of creating realistic images of butterflies from scratch—in SQL, of course.

As always, a little bit of theory first.

These days, most AI products that generate pictures (like Kandinsky, Stable Diffusion, Midjorney, and similar) are backed by variations of a technique known as Diffusion Model. I will try my best to explain the intuition behind it, while aiming to keep the level of math and formulas at the bare necessary minimum.

To illustrate the non-SQL related portions of this post, I'll be using a ready-to-use, pretrained model that I found on HuggingFace. This model is called gnokit/ddpm-butterflies-64.

It's a DDPM model, with the UNet architecture as a backbone, trained to perform denoising in 1000 steps with the linear noise schedule from 0.0001 to 0.02. I'll explain later what all these words mean.

It's been trained on the Smithsonian Butterflies dataset. It can unconditionally generate 64×64×3 images of butterflies that don't exist in nature. "Unconditionally" here means there is no prompt or anything. You just run it and get a random butterfly.

With that out of the way, let's begin. First of all, what the hell is DDPM?

The Denoising Diffusion Probabilistic Model

There is a popular quote attributed to Michelangelo. When asked about how he made his statues, he responded: "I saw the angel trapped in the marble and I carved until I set him free". Translated to the language of math, it means that he saw a block of marble in front of him, the angel's image in his mind, and was able to calculate, so to speak, their difference: how much marble he should have chiseled away at any given place until the former became the latter.

Just like this:

Making statues and painting pictures is something Michelangelo could do, and I can't. But this never stopped me from giving advice. Everything I can do, I do by making a series of small, simple steps, each one bringing my work closer to the completion. When you learn a craft, you have to learn two things: first, what do to on every one of the small, simple steps; second, which step leads to which next one.

Learning how to make art (or anything else, for that matter) works the same way: break everything into many small pieces, and learn how to complete every piece. When you see Bob Ross painting lessons on TV, it's a series of simple brush strokes, going from a half-incomplete picture to a slightly less incomplete one.

Turns out, computers can learn how to paint in the same way.

When you teach a computer how to paint, you show it pairs of images of stuff. The first image in the pair is the original photo, and the second one is the same photo with some noise added. The amount of noise varies from very little (almost clear picture) to very large (almost completely random pixels). You also tell the computer how much noise you have added.

During the training phase, the computer gets lots of these pairs and feeds them into the special function. This function is called the "backbone" of the model (or, quite often, just the "model").

On input, the function receives two values: a vector with the pixels of the dirty image, and the amount of noise (a scalar number).

On output, it returns an image that is supposed to be as close to the original image as possible. Or, what is essentially the same, the vector with the noise that you need to subtract from the dirty image to get back to the original.

The output of the function needs to have the same shape as the input. If the first input parameter to the function is a 64 (height) × 64 (width) × 3 (RGB) vector that represents a dirty image, so is the function output, which represents the noise it "thinks" was added to the original.

Here's the shape of the function:

backbone(image: vector, step: int) -> vector
# Takes the dirty image and the amount of noise added to it. The amount of noise is encoded in an integer
# Returns the noise 

The function doesn't receive the original image. It has to guess what it was just from the dirty image, and the amount of noise. Note that the amount of noise is encoded as an integer (I'll explain later why)

How does the backbone predict the noise?

The backbone function is composed of a series of simple operations. These operations are matrix multiplications and additions (lots of them), exponentiations, logarithms, and simple trig. All these operations (and, hence, their composition) are differentiable.

The matrix multiplications that happen inside the functions use a lot of constants that are baked into the function (i.e. don't come from its inputs). During the training phase, these constants can change between invocations — and changing them is exactly what training does —but once the training is done, these constants are set in stone. They are called "parameters" of the model. Their values can change (only during the training), but their total number and shapes (whether they are scalars, vectors, or matrices) are set at design time. All of this together: the code of the function, and the shapes of its internal parameters, is called the "architecture" of the model.

Before the training commences, these parameters are set to random values (literally, random). On the first invocation, the function, of course, does a very poor job of predicting the added noise. The training process looks at its output (rather underwhelming, at first) and compares it to the real vector of noise that has been added. Unlike the backbone, the training process knows both the original image and the added noise.

"Comparing" noises, in this case, means calling a special function called the "loss function". The loss function returns a single number (a scalar). The smaller the difference between the predicted and the actual noise vectors, the lower the return value of the loss function, with 0 meaning the ideal match.

The exact shape of the loss function is not an integral part of the backbone (it's only being used during the training phase). The authors of the original DDPM paper found out that Mean Squared Error (the sum of squares of the differences between predicted and actual noise values for each pixel and channel) works quite well as a loss function.

Once the first pass of the function has been performed and the loss value has been calculated, the actual learning process happens.

As a developer, you compose your backbone function by chaining methods from your framework (take this vector; multiply it by this matrix of weight constants; add a vector of bias constants; take the exponent, and so on).

The framework that is doing the training tracks all the operations that the function performs, records their order, and calculates the partial derivatives with respect to all the parameters that were used in this function. It uses the technique known as automatic differentiation. Most popular frameworks used for machine learning (PyTorch, TensorFlow and others) have built-in utilities that let the developer just compose the functions, and the framework will take care of the differentiation.

The return value of the loss function, at a cursory glance, looks like a single scalar number. But the variable where it's stored also carries an auxiliary data structure for the framework's internal use. This structure keeps the whole graph of the calculations performed on the inputs and the parameters, which eventually led to the number. It also stores cached numeric values of the gradient of the loss function, or, in other words, the value of its partial derivatives with respect to all the parameters that have been part of this graph. The parameters in a DDPM usually number in the millions or even billions. One puny number carries the whole history of how this number came to be.

The loss function is also tracked by the automatic differentiation framework. It means it has to be differentiable (and simple enough) as well. The calculation of the loss is composed of those of the backbone (which is differentiable), and those of the mean squared error (which is differentiable too). Making a small enough change to the parameters results in a small change in the value of the loss function. So what happens if we change the values of all the parameters in the direction opposite that of the gradient? The value of the loss will, quite mechanically, become smaller, which is what we want.

Once the loss has been calculated, the framework looks at the gradient of the loss function, and adjusts all the parameters ever so slightly opposite the gradient. It does it so that the backbone, had it been called again with the same input and new parameters, would return a smaller loss. This process is called backpropagation.

Note that I said that the loss would be smaller for the same input. But the inputs don't repeat during the real training. On every step, the loss function is fixed to work better with the old data, but doesn't get to enjoy it: on the next step, it will get new data. The results will still be underwhelming, but this time, hopefully, less so.

This process is repeated until the loss function settles on its minimum. It's not guaranteed to happen, though:

1. The loss, after a certain number of steps, might stop decreasing and start growing. It happens when the model becomes so good at reproducing the pictures from its initial training data, that it can't learn how to paint anything else. This is called overfitting.
2. The training process takes a long time and we run out of money to rent GPUs.
3. Finally, we might have picked the wrong math for our function. If that's the case, we can keep training it until the cows come home, and it still won't converge to an acceptable level of loss.

But if we manage to come up with a good model architecture (i.e. we are composing just the right sequence of linear algebra, exponents, and trig), the model will converge fast on a small enough loss value. It means it can take a noisy picture of a butterfly, and predict how to undo the noise quite well.

Why is it called "diffusion"?

The name comes from the similarly named concept in physics. If you carefully put a drop of red dye in water, the dye will start diffusing. The molecules of the dye will become subject to the Brownian motion (also known as random walk). The drop will keep its original round shape for a brief moment, then it will become slightly less round, then a formless cloud, and so on, until all the traces of the original shape are lost and water would acquire a uniform pinkish color. It's a gradual process. Each blurry shape of the dye cloud comes from a less blurry one that was there a moment ago, by adding a little bit of chaos to it. This process is known as the Markov chain.

The original idea of adding the noise to the clear image was similar to that. The noise would be added gradually, in small increments, through a Markov chain. Each subsequent noisy image would have a dependency on the previous, slightly less noisy one. The model would learn to undo the noise added on each step.

How many steps should be there and how much noise should we add on each step? It's up to the developer. We could add the same amount of noise, or make it grow linearly, or even come up with a function. The authors of the original DDPM paper decided to use 1000 steps, and make the amount of noise grow linearly with each step: from 0.0001 on the first step (least noisy image) to 0.02 on the last one (most noisy).

"Adding" is a little bit of a misnomer here: as we are sequentially "adding" amounts of noise, we are actually multiplying one's complements of these amounts (what's left of the original image). If we do the math, we'll see that on step 1000 the share of the original image will be about 4e-05. At this point, we can safely ignore it and say that the 1000's step on this Markov's chain is indistinguishable from noise.

These figures: the total number of steps, and the amounts of noise added on each step, are called the "noise schedule" of the model. If you remember, our backbone function accepts the noise level encoded in an integer. This integer is just the number of the step on the Markov chain. From this number, and from the noise schedule, we can always figure out the exact amount of noise.

This sounds promising in theory, but let's recall how training works. The model takes the noisy image on input and returns its best estimation of the non-noisy image on the output. If we added noise to the images in small increments, we would have to give the model every intermediate image during the training phase.

It means that to train the model to undo small amounts of noise added 250 times over (which, the way the math works, would result in about 50% noise in total), we would have to run 250 passes over 250 images, each one having just a tiny bit of noise added to the image from the previous step.

This is bad for several reasons:

1. It's very expensive computationally.
2. With that much math, there's a good chance that rounding errors will pile up to the point where all the original signal drowns in rounding errors.
3. It has to do with how the automatic differentiation works well towards decreasing the loss function. It needs to track the path of the original input to the model (which is taken for granted) through the math operations over the constants ("parameters"). The constants are being adjusted along their gradients and, once the learning is done with, are set in stone.

   Noise is not the parameters, so it has to be the input. We will have 700 instances of noise added to one single instance of the original image. Since to the model, all the numbers are the same and their "meaning" only emerges, if ever, during the training phase, it will have a much harder time discerning the original image from all that noise.

4. The sequence of the functions inside the model will now explicitly depend on its input. While models can have loops over the same parameters inside of them, the number of iterations in these loops has to be fixed at design time. This particular problem can be fixed by hardcoding the number of the loops to 1000 and substituting missing images with zeros or something, but then again, it will take more time.

With all this, it seems like the Markov chain approach isn't exactly going to work out of the box. But the authors of the original DDPM architecture have made a very astute observation that allowed them to fix it and save the day.

Reparameterization

If you generate noise just by putting a random value sampled from the Gaussian distribution (with the mean 0 and the variance 1) to every pixel of the noise vector, the end result will also come out as if sampled from the standard Gaussian distribution , with 0 for the mean and identity for the covariation matrix. The Gaussian distribution is also known as the normal distribution, or the bell curve distribution.

Now, you have two options for generating a dirty image. You can go by the book and add a certain small amount of standard Gaussian noise to the image in many sequential steps. Or you can make a shortcut of sorts. You can take the same image, sample the noise just once, and add a bigger amount of that noise to the clean image in one fell swoop, this bigger amount being defined by a simple enough formula.

One nice property of the standard bell curve noise is that the end results of both these operations will be identical from the statistical point of view. They will have the same distribution. It means that the results of both these operations, repeated many times over, will look like equally noisy images of butterflies or whatever it was in the training set. There's no way to tell the results of these two processes apart using statistical methods. If the result of the sequential process was good enough to usefully train the model, so will be the result of the shortcut process.

From the math perspective, adding an amount of noise to the vector of image pixel intensities twice looks like this:

Here, and are two different random variables sampled from the same standard Gaussian distribution. The expression in the square brackets is a sum of two independent, normally distributed variables, and can be replaced with another random variable. It will also be normally distributed, with a mean equal to the sum of the terms' means, and the variance equal to the sum of terms' variances. The means are zero and will remain so. Variances depend on the scale, so those will change. We can replace the expression in the brackets:

and the final result for the image with noise added twice will look like this:

It's easy to see that the formula for every next step will be exactly the same, just with more factors in the product. So instead of adding a certain amount of noise step by step, you can calculate the total amount of noise using the formula above, and add it all at once. The results will be statistically indistinguishable.

This technique is known as the reparameterization trick. In hindsight, it might look like a simple, even an obvious thing. But it took a great bit of ingenuity to notice it and put it to work.

In practice, it means that you don't have to go through all the steps to generate an image with a bigger amount of noise, and don't have to feed the intermediate results to the model either. You can just feed the number of steps to the formula and find the right amount of noise at once.

Another nice effect of this trick is that if you generate an image on step from a Gaussian sample, you can use the same sample to generate an image on step . The difference between these two images (that is, the noise that would have been added on step ) would also have the same standard Gaussian distribution, whose parameters are easily calculatable.

We can pretend to have a Markov chain without actually computing all of its steps.

Where are the small steps?

Putting all the above together, we have a model that takes a noisy image and the amount of noise added to it and does a semi-decent job of predicting what the original image would be.

The model can now, so to say, see the images trapped inside white noise and set them free.

I can hear the model saying: "I see a piece of marble, and you say that the angel is about halfway inside. This is good enough. I see the vague shape of the wings, face, and everything else. I know a thing or two about angels. I have a good idea of how much marble I need to chisel away at every point to set the angel free. Give me a chisel and a mallet, and step aside".

Now, we've trained the model to work separately with each step, including those close to the end. As we remember, at step 1000 the dirty image is virtually indistinguishable from white noise. By the end of the training process, the loss level is supposed to be small and uniform, regardless of the step (if that's not the case, we don't have a good model to begin with). The model should do a decent enough job of reconstructing the original image from any step.

So, technically, we don't even need to begin with an image that once was a butterfly. We can just give the model chemically pure white noise. Then we would tell it to work its way back from step 1000, and it would paint us a butterfly.

Right?

Well, let's see how it works in practice.

We'll take the backbone function of the trained model, feed it some noise, and ask it to reconstruct the image from step 1000.

Here's what we get:

Is it a butterfly?

Well, it's definitely… something. It's not white noise. If you squint hard enough, you can see some kind of pattern, a very vague outline of the wings. But even its own grandmother wouldn't call it a butterfly.

So the model made a prediction about how to cancel the noise all the way through to the original image. The results of this prediction don't look good. Maybe we didn't need to go all the way through at once? After all, why did we learn on all these different noise levels separately?

Now, it looks like it's time for a little heart-to-heart with the model.

Listen, Michelangelo. I know you mean well. You think you know exactly how to set the angel free. Remove the noise all the way to what you think is the butterfly. Let me be straight with you: you can't. But hear me out now.

What if you didn't try to do this all at once? Instead of chiseling away all this noise, how about you just start in the right direction but don't work all your way through?

You're at step 1000 now. Don't go right to zero. Start removing the noise, but stop at the same amount step 999 would have. It would still be mostly noise, but this time a little bit closer to the butterfly. Then just run the backbone again with the new data, see if the noise estimation will correct itself a little bit, and then we maybe will do a better job next time.

How about that, buddy? Baby steps, remember.

Ah, it's a computer, it doesn't understand what we say anyway. So we will explain it in pseudocode:

# Algorithm to generate an image using the Denoising Diffusion Probabilistic Model:
image = random(height, width, channels) # -> vector Random pixels
for step in reverse(range(0, 1000, -1)): # from 999 to 0
    noise = backbone(image: vector, step: int) # Predict noise
    image = subtract_some_noise(image, noise, step) # Subtract predicted noise
    image = image + add_noise(step) if step > 0 else 0 # Add some noise (yes, add). More on that later
return image # 0% noise, realistic photo

And this is how you get the baby steps. And most AI image generation software actually works this way. "Knowing" how pictures become noise, it starts from pure noise and tries to gradually cancel it to get a picture. This noise was not generated from the Markov chain and had never been added to a picture in the first place. But the model still tries to gradually cancel it in steps. And, in so doing, arrive at the picture that this noise would have been added to, had it existed in reality.

Crazy as it sounds, it works.

Backbone

There are several popular architectures for the backbone of the model, the function that is doing the heavy lift. The one that the butterfly model is using is called U-Net. We'll go through its internals in a short while. But first, let's look at some general principles.

Where do we start with the backbone at all?

In the Middle Ages, there were artists who knew elephants existed but never saw them. They made their artistic impressions of elephants from their verbal descriptions. Here are some of them:

A verbal description of an elephant resembles a grocery list of its characteristic features: "long trunk with bell mouth at the end," "big ears," "tusks," and so on. It's somewhat of a vector. You check every dimension, you get yourself an elephant. The animals in the pictures above fit these descriptions. We can recognize them as elephants, albeit funny-looking ones. What is it exactly that is so funny about them?

The grocery lists off of which the artists worked were incomplete. They did mention big ears and long trunks, but omitted the shapes of the legs, the body proportions, and other important things. The artists were applying these features on top of the images of the animals that had already existed in their minds. The beasts in the pictures are, essentially, goats and lions, whose ears were made bigger and noses longer as to match the incomplete description. This, however, was enough to make us see what they meant.

The working hypothesis is that if we somehow make a complete list of all the features of an elephant, or a butterfly, or anything else, we can make a photo-realistic picture of it. How do we go about it?

Latent space

Generative AI models work with images of certain fixed dimensions. When we see, say, a 64 × 64 × 3 image (height, width, and colors), we can treat it as a vector with 12288 dimensions. Every dimension of this vector will represent the brightness of a certain pixel. For simplicity, we'll assume that the values are from 0 (black) to 1 (the brightest color your monitor can handle).

We need a function that would map this vector to a vector of "butterflyness," so to speak, the grocery list of all things that, taken together, make a butterfly. It's, supposedly, quite a long list. If maybe one or two things are missing from it, it would be a "funny butterfly," or maybe a "cartoon butterfly," but still a butterfly alright. The items on this list should be encoded with continuous numbers, and the mapping should be differentiable.

The space of such vectors would be called a "latent space."

The main idea behind machine learning is that humans are not even supposed to try to describe these features analytically. The mapping would, as they put it, "emerge" during the training phase, the basic mechanics of which we learned in the previous section.

A quick refresher. The most common way of mapping one vector to another is the linear mapping. Mathematically, it's a matrix multiplication (by a matrix of constants) that changes the dimensions, followed by a vector addition (with a constant term) that translates the result. The constants in the factor matrix are traditionally called weights, and the constants in the summand vector are traditionally called biases. The linear mapping is a differentiable operation that lends itself easily to the procedure of machine learning.

You can stack linear mappings, but however many you stack, the end result will still be a linear mapping, which can be reformulated with a single matrix of weights and a single vector of biases, so stacking them makes no real difference.

Very soon people understood that linear mappings are a powerful, but not omnipotent, tool and have one very nasty limitation.

Vectors live in linear spaces, and every linear space can be cut in half by a hyperplane. A 2D space can be cut in half by a line, a 3D space by a plane, and so on. Turns out that if the feature vector that you're trying to learn is linearly inseparable, that is, you can't draw a straight line, or a plane, or a hyperplane in such a way that all the "butterflies" would stay on one side of it and all "non-butterflies" on the other, then linear mappings wouldn't work.

However, if you combine as few as two linear mappings, putting a non-linear function after each one of them, then, at least in theory, this combination can model just about everything we care about. The only limitation is the number of the dimensions in the vector space. You might be able to model any function, but you might need vectors with one million, or one billion, or any other number of dimensions for that. You might be more successful if you stack more than one linear mapping, interspersing them with non-linear functions (called "activators," or "activation functions," in machine learning parlance). Of course, the activation function should be differentiable as well.

Such a sandwich of linear mappings and activators is called a "neural network." A linear mapping plus activation is called a "layer." It's a basic building unit of almost all things AI. Everything is done by stacking layers of linear mappings and activation functions in clever ways.

Our intuition is to make two stacks of NN layers. The first stack would map the pixels of the image into a vector in the latent space. Some dimensions of this vector would be features that make a butterfly a butterfly (whatever it might be), and the rest of the dimensions would be responsible for everything else. The second stack would bump the butterfly-related components of the vector all the way up to eleven, to the Platonic ideal of butterflyness, leaving everything else as is or not even caring about it, and then map the result back to the image. This would make it the original image, just without noise. The difference would be noise, and we would return it to the loop that cancels the noise (it's called the schedule runner).

Again, that's just general intuition. We don't know what the "latent space of butterflyness" looks like or will look like, we don't know what its components do or will represent. We can't (generally) come up with them on purpose, or isolate their meanings in human-understandable terms.

All we can do is give the neural network enough dimensions to work with and use whatever little intuition we have in connecting these dimensions the way that we hope will help it converge. The exact semantics of all these features, dimensions, and latent spaces will emerge when (and if) the training converges on a small enough value of the loss function, and we still won't know what they are and how exactly they work.

Sounds promising, ain't it?

How do I design all these neural stacks?

Is it a general question? If yes, I'm sorry, I don't know. Really. Sorry.

The answer to this question would be the same as to "How do I compose a melody?" or "How do I write a short story?": simple, but utterly useless. Here it is. You write stories by putting the letters in order, left to right, minding your commas. You compose a melody by stacking lenghts of chords and pauses between them. They have to make sense, be captivating, and be pleasant to the eye or the ear. The same thing about machine learning. It has to be computationally feasible; it has to converge at the end of the training phase; and it has to do its job. That's it. It's an art. I don't know how to do that.

However, same as with composing and creative writing, there's a talent component to it, and there's a craft component to it. Inspiration and perspiration, and all this jazz. As for the craft part, there are some tricks and patterns that have empirically proven to be useful over the years. We'll now consider some of these tricks.

Convolution

It's a special kind of NN layer, particularly useful when working with images.

The input and the output vectors are laid out as 2D grids (or, in other words, matrices). Each pixel of the output vector is computed by a Frobenius inner product of the input matrix and some constant parameters—not of the whole input vector, though, but of its smaller sliding subgrid. In machine learning parlance, it is said that this layer is not fully connected, because not every component of the input vector affects every component of the output vector.

Mathematically, it's equivalent to a fully connected layer with some constants hardcoded to be 0 and not change during the training phase.

Convolution is particularly useful for isolating different features of images, and convolutions are being used in graphics processing even outside the domain of machine learning. When you open your favorite graphics editor and do things like blurring, sharpening, edge detection, magic wand selection, and so on, under the hood it runs convolutions.

This page has a nice interactive demonstration of convolutions and how you can use them in image processing. The parameters in the matrix multiplied with the sliding window are learned during the training phase, but its shape and dimensions are fixed at design time.

Having these layers instead of fully connected layers helps the model to isolate visual features while bringing down the amount of number crunching.

Residual connections

We make a fully connected layer (or several layers), and then, in a separate operation, we add its input to the output.

Before that, our neural network looked like a sandwich, and now it looks more like a taco. Mathematically, this operation is equivalent to adding a hidden layer with hardcoded 0's and 1's at certain places in weights and biases, and the identity function for the activator, so it's still a sandwich, i.e. a good old stack of linear maps and activations. Still, it's much easier to reason about it this way.

This method has been empirically found to help with exploding and vanishing gradients. If the neural model comprises a lot of stacked layers, its partial derivatives in respect to individual parameters may become either very small or very large. If it happens, the model as a whole becomes either too little sensitive to parameter changes (and, thus, never converges) or too sensitive, in which case even a small change to the parameter value throws the loss function too far away from a local minimum. Adding residual connections inside the network helps with that problem.

Normalization

Most neural networks start their training being initialized to random numbers. During the training itself, features emerge, and they are being encoded with different dimensions inside the latent spaces. But in the beginning, it's just a goo of numbers, like in a pupa from which a butterfly would then emerge (see what I did here?). All these numbers have the same distribution, and their numerical values are on the same scale.

If these numbers happen to encode different features and will come to live in different dimensions, some of these numbers might happen to become higher, and some lower. Household values of mass are measured in the range of 10e-3 through 10e3 kilograms, those of time in the range of 1 through 10e8 seconds, and those of, I don't know, electric capacitance in the range of 10e-15 through 10e-6 farad. It's been noted (again, empirically) that models that happen to keep their parameters in the same range train better and converge faster. We humans have the same problem; that's why we have invented all these "kilo-," "mega-," "micro-," and other prefixes that help us keep our numbers in palatable ranges.

Because of this, it helps if, every now and then, there are layers whose only purpose in life is bringing the numbers flowing through all the math to the same general range. They are just being multiplied by a certain number: usually, the reciprocal of their average, root mean, or something like this. It can be done for all the numbers in the network or separately for different chunks of them. Different techniques for that are called "layer normalization," "instance normalization," "group normalization," etc. (see Group Normalization, Wu, He, arXiv:1803.08494), which are different implementations of the same general idea.

Attention

This layer is generally considered one of the most (if not the most) important things that have contributed into the AI boom of today. It's been introduced in the famous paper Attention Is All You Need, Vaswani et al., arXiv:1706.03762.

The intuition behind this layer is that it splits a hidden layer into several subvectors and tries to encode the level of influence each pair of the subvectors exerts on each other. The assumption is that if we model the function this way, it will help the model understand how different features (that will emerge during the training and of which we don't know in advance) correlate with each other. It's supposed to mimic the way attention in humans and other animals works.

The values of correlations are taken from something known as a "differentiable key-value store." It's not a real key-value store; it's just a catchy name for a particular way to connect vectors in a neural network. Here's the intuition behind it, the way I understand it.

Let's consider a regular key-value store, where both keys and values are floating-point numbers and cannot be null. The store returns the stored value () if the query () equals the key (). If not, it returns 0, for the lack of a magic value signifying a miss. It can be modeled by the expression , where is the Kronecker delta function.

The Kronecker delta is not a differentiable function and cannot be directly used in the neural network, so we have to approximate it using differentiable functions. A good enough approximation of the Kronecker delta is our good friend, the normal distribution, scaled to , with the mean around and low enough variance. If is "close enough" to , then the expression will evaluate to a value "close enough" to , and to zero otherwise. and are vectors, and in vector spaces, "close" means "has large cosine similarity." The cosine similarity is the value of the dot product of two vectors, scaled down to the range of . So the expression for the "differentiable key-value store" would become , which then gets scaled down. The exact nature of the downscaling process varies between implementations, but usually, the softmax function is being used for that.

The values of keys, values, and queries are not learned parameters (or, rather, not just learned parameters). They change their values on every invocation (or, the way they usually put it, "forward pass") of the neural network even outside the training phase. This is the closest thing that the neural network has to short-term memory.

Now that we are done with the basics, let's move to higher-level building blocks.

U-Net

Here's the diagram of the U-Net architecture:

This architecture consists of three pieces:

1. The encoder (the left-hand side of the U), which maps the image from a vector of RGB pixels to a vector in the latent space.
2. The middle part, which runs attention and residual connections on the latent space
3. The decoder (the right-hand side of the U), which maps the vector in the latent space to the vector of noise pixels in RGB.

The left-hand and the right-hand sides of the U are traditionally being called "encoder" and "decoder," but it's a little bit of a misnomer, because they have residual connections bypassing the latent space layer. A pure encoder and a pure decoder would have been able to work independently of each other, only communicating through the latent space.

The backbone of the model that we are exploring looks almost like the one on the diagram, with a couple of differences (which I will mention separately). Here's what's going on in the implementation of the U-Net architecture that drives the model:

1. On the encoder, there are four big submodels. Each one of them comprises two blocks known as ResNets.
2. Every ResNet consists of two convolutional layers (with a 3×3 kernel) and a residual connection. The activation function is the sigmoid.
3. Before the convolutional layers, the image is group-normalized with 32 groups.
4. After the first convolutional layer, the image is mixed with a vector encoding information about the current noise schedule step (more about that later).
5. The image starts out with three channels (RGB) and not one, as in the picture. In every one of the four decoder's blocks, it's split into more channels (64, 128, 256, and 512, accordingly), and later, the channels are recombined in the encoder's blocks.
6. After each ResNet, a downsampling is performed. Instead of using Max Pooling (like in the picture), our model uses strided convolution. It works like a regular convolution, but the sliding subgrid moves two cells at a time across the grid. The output grid, as it is easy to see, comes out twice as small.
7. In the bottom two blocks on the left side of the U, after each ResNet, an attention block is added (not shown on the picture).
8. The bottom side of the U is where the heavy lifting is happening. It's two ResNets in a sequence, with an attention block between them. The working theory is that at this layer, our vectors are in the latent space and record different aspects of the butterflyness of the original image more or less orthogonally in their dimensions.
9. The right side of the U (the decoder) mirrors the left side, but with a little twist: inputs and outputs are twice as large, and there are three ResNets instead of two. It's because for the encoder, we concatenate the results from the bottom layer with the results that came straight from the appropriate parts of the decoder. This is sort of a residual layer on steroids, and its presence has been shown to help with the stability of the training process.
10. There is also an attention block between the two ResNets in the middle part. (Attention blocks are not shown on the diagram.)

Step encoding

We remember that our backbone receives two parameters: the image itself and the number of step in the noise schedule. Where does the step number go?

Theoretically, we could have just multiplied the original image by it or added it. However, those would have been linear operations, and those get lost really easily during training (the first normalization block on the way would have killed them both). We need some way to make it more robust and, ideally, spread it along more dimensions so that more parameters would be influenced by its value.

How do we go about it?

A single number from 0 to 999 can be encoded with a 10-bit value that easily fits into a single floating-point variable and even leaves some extra space. For the most use scenarios, that would be fine. Usually, people try to pack as much data as possible into a single variable. That's what compression algorithms are for.

But now, for a change, we need to solve the opposite task: how to encode this value using a lot of variables? The more, the merrier. Our input vectors have the number 64 among their dimensions, so different values of 64 float variables would be a good start. And remember, no cheating. No stuffing with zeros or something like that. All these variables have to be different and meaningful; otherwise they will get cancelled by the first normalization block or bias vector on the way.

One thing we can do is make each digit go into a separate dimension, so, say, for step 576, 5 would go to the first dimension of the vector, 7 into the second one, and 6 into the third one. Three dimensions covered, sixty-one more to go. Can we do better?

Well, we can do the binary. 999 is 1111100111 in binary; that's 10 bits, so ten dimensions are covered, fifty-four more to go. And this, mind you, is not a good encoding for neural network purposes. Ones and zeros are way too discrete; we like things that are more smooth and differentiable. Can we do better?

It might be hard to see how we can go to a base less than binary (?), while making it less discrete (???), but amazingly enough, there is a way.

Let's look at binary encodings of some integer numbers, starting from zero:


0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 1
0 0 0 0 0 0 0 0 1 0
0 0 0 0 0 0 0 0 1 1
0 0 0 0 0 0 0 1 0 0
0 0 0 0 0 0 0 1 0 1
0 0 0 0 0 0 0 1 1 0
0 0 0 0 0 0 0 1 1 1
0 0 0 0 0 0 1 0 0 0


etc.

We can see that the last digit changes with every number, the one before last every two numbers, the one before it every four numbers, and so on. Every number has a very strict and regular periodic pattern.

What else has a periodic pattern? Why, the trigonometric functions have. We could represent any natural number using this expression:

and then use terms of this expression for the components of the vector (ditching the factors). This would boil down to just using 0s and 1s of the binary representation, so we would have run out of 1s by the eleventh dimension.

But let's look closer at this expression. We want to get rid of discreteness and make everything smooth, right? And we want to have meaningful numbers?

The first thing we could do is get rid of that ceiling operation inside the argument to the cosine. This would kill the exact mapping to the original number. But we don't need the number's exact value; we just need it encoded in a vector. The exact number would actually only confuse the neural network.

And while we're at that, let's look at the denominators inside the argument to the cosine. They are 1, 2, 4, 8, and so on, so the powers of two. Two is the smallest natural number that has meaningful powers, but now that we don't care about natural anymore, why can't we go lower than two but higher than one? Real numbers would work just as well for the base, and this way we would have more meaningful digits.

This is exactly how the encoding of the step numbers works. Here's the expression for it:

Here's what some of the dimensions of this vector look like for the first 100 timesteps:

All the numbers are smooth, periodic, and all of them make good use of the dimensions space.

This technique is known as sinusoidal position embedding, and is closely related to the Fourier Transform, an enormously useful technique for just about any field of signal processing.

SQL implementation

I've implemented the U-Net function just as a series of lateral cross-joins, without any recursive CTE or other tricks. This makes the nature of neural networks more obvious.

As usual, I'm using vanilla Postgres 17, without any extensions, and I'm not writing any procedural code either. To avoid excessive copy-pasting, I'm wrapping the basic blocks (like linear mappings, convolutions, sigmoid activations, and attention blocks) into pure SQL functions, which also don't have any procedural code.

Here's how it looks. Click on the title to expand the source; refresh the page to collapse it again. (I know it sucks; sorry about that.)

CREATE OR REPLACE FUNCTION EMB(_step INT)
RETURNS REAL[]
AS
$$

SELECT  ARRAY_AGG(COS(value)) || ARRAY_AGG(SIN(value)) AS t_emb
FROM    GENERATE_SERIES(0, 31) AS dim
CROSS JOIN LATERAL
        (
        SELECT  _step * EXP(-LN(10000) * dim / 32.0) AS value
        ) q;

$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION CONV2D3X3(_input REAL[][][], _weight REAL[][][][], _bias REAL[], _stride INT = 1)
RETURNS REAL[]
AS
$$

SELECT  ARRAY_AGG(value)
FROM    (
        SELECT  ARRAY_LENGTH(_input, 3) AS image_width,
                ARRAY_LENGTH(_input, 2) AS image_height
        ) q1
CROSS JOIN LATERAL
        GENERATE_SERIES(0, ARRAY_LENGTH(_weight, 1) - 1) AS c_out
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(value) AS value
        FROM    GENERATE_SERIES(0, image_height - 3 + 2, _stride) ky
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(value) AS value
                FROM    GENERATE_SERIES(0, image_width - 3 + 2, _stride) kx
                CROSS JOIN LATERAL
                        (
                        SELECT  SUM(
                                        COALESCE(_input[c_in + 1][ky - 1 + 1][kx - 1 + 1], 0) * _weight[c_out + 1][c_in + 1][1][1] +
                                        COALESCE(_input[c_in + 1][ky - 1 + 1][kx     + 1], 0) * _weight[c_out + 1][c_in + 1][1][2] +
                                        COALESCE(_input[c_in + 1][ky - 1 + 1][kx + 1 + 1], 0) * _weight[c_out + 1][c_in + 1][1][3] +
                                        COALESCE(_input[c_in + 1][ky     + 1][kx - 1 + 1], 0) * _weight[c_out + 1][c_in + 1][2][1] +
                                        COALESCE(_input[c_in + 1][ky     + 1][kx     + 1], 0) * _weight[c_out + 1][c_in + 1][2][2] +
                                        COALESCE(_input[c_in + 1][ky     + 1][kx + 1 + 1], 0) * _weight[c_out + 1][c_in + 1][2][3] +
                                        COALESCE(_input[c_in + 1][ky + 1 + 1][kx - 1 + 1], 0) * _weight[c_out + 1][c_in + 1][3][1] +
                                        COALESCE(_input[c_in + 1][ky + 1 + 1][kx     + 1], 0) * _weight[c_out + 1][c_in + 1][3][2] +
                                        COALESCE(_input[c_in + 1][ky + 1 + 1][kx + 1 + 1], 0) * _weight[c_out + 1][c_in + 1][3][3] +
                                        0
                                )
                                + _bias[c_out + 1]
                                AS value
                        FROM    GENERATE_SERIES(0, ARRAY_LENGTH(_weight, 2) - 1) AS c_in
                        ) q
                ) q
        ) q2
$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION CONV2D1X1(_input REAL[][][], _weight REAL[][][][], _bias REAL[], _stride INT = 1)
RETURNS REAL[]
AS
$$

SELECT  ARRAY_AGG(value)
FROM    (
        SELECT  ARRAY_LENGTH(_input, 3) AS image_width,
                ARRAY_LENGTH(_input, 2) AS image_height
        ) q1
CROSS JOIN LATERAL
        GENERATE_SERIES(0, ARRAY_LENGTH(_weight, 1) - 1) AS c_out
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(value) AS value
        FROM    GENERATE_SERIES(0, image_height - 1, _stride) ky
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(value) AS value
                FROM    GENERATE_SERIES(0, image_width - 1, _stride) kx
                CROSS JOIN LATERAL
                        (
                        SELECT  SUM(
                                        COALESCE(_input[c_in + 1][ky + 1][kx + 1], 0) * _weight[c_out + 1][c_in + 1][1][1]
                                )
                                + _bias[c_out + 1]
                                AS value
                        FROM    GENERATE_SERIES(0, ARRAY_LENGTH(_weight, 2) - 1) AS c_in
                        ) q
                ) q
        ) q2
$$

LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION LINEAR_1_2(_input REAL[], _weight REAL[][], _bias REAL[])
RETURNS REAL[]
AS
$$

SELECT  ARRAY_AGG(value)
FROM    GENERATE_SUBSCRIPTS(_weight, 1) l1
CROSS JOIN LATERAL
        (
        SELECT  SUM(_input[l2] * _weight[l1][l2]) + _bias[l1] AS value
        FROM    GENERATE_SUBSCRIPTS(_weight, 2) l2
        ) q

$$

LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION LINEAR_2_2(_input REAL[][], _weight REAL[][], _bias REAL[])
RETURNS REAL[][]
AS
$$

SELECT  ARRAY_AGG(value)
FROM    GENERATE_SUBSCRIPTS(_input, 1) i1
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(value) AS value
        FROM    GENERATE_SUBSCRIPTS(_weight, 1) w1
        CROSS JOIN LATERAL
                (
                SELECT  SUM(_input[i1][l2] * _weight[w1][l2]) + _bias[w1] AS value
                FROM    GENERATE_SUBSCRIPTS(_weight, 2) l2
                )
        ) q

$$

LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;


CREATE OR REPLACE FUNCTION PLUS_3_1(_a REAL[][][], _b REAL[])
RETURNS REAL[]
AS
$$

SELECT  ARRAY_AGG(v2)
FROM    GENERATE_SUBSCRIPTS(_a, 1) l1
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v3) AS v2
        FROM    GENERATE_SUBSCRIPTS(_a, 2) l2
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(_a[l1][l2][l3] + _b[l1]) AS v3
                FROM    GENERATE_SUBSCRIPTS(_a, 2) l3
                )
        ) q

$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION PLUS_3_2(_a REAL[][][], _b REAL[][])
RETURNS REAL[]
AS
$$

SELECT  ARRAY_AGG(v2)
FROM    GENERATE_SUBSCRIPTS(_a, 1) l1
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v3) AS v2
        FROM    GENERATE_SUBSCRIPTS(_a, 2) l2
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(_a[l1][l2][l3] + _b[l1][l2]) AS v3
                FROM    GENERATE_SUBSCRIPTS(_a, 2) l3
                )
        ) q

$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION PLUS_3_3(_a REAL[][][], _b REAL[][])
RETURNS REAL[]
AS
$$

SELECT  ARRAY_AGG(v2)
FROM    GENERATE_SUBSCRIPTS(_a, 1) l1
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v3) AS v2
        FROM    GENERATE_SUBSCRIPTS(_a, 2) l2
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(_a[l1][l2][l3] + _b[l1][l2][l3]) AS v3
                FROM    GENERATE_SUBSCRIPTS(_a, 2) l3
                )
        ) q

$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION SILU(_v REAL[])
RETURNS REAL[]
AS
$$
SELECT  ARRAY_AGG(v * EXP(v) / (1 + EXP(v)))
FROM    UNNEST(_v) AS v
$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION SILU2(_v REAL[][])
RETURNS REAL[][]
AS
$$
SELECT  ARRAY_AGG(v2)
FROM    GENERATE_SUBSCRIPTS(_v, 1) l1
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v * EXP(v) / (1 + EXP(v))) v2
        FROM    GENERATE_SUBSCRIPTS(_v, 1) l2
        CROSS JOIN LATERAL
                (
                SELECT  _v[l1][l2] AS v
                ) q
        ) q
$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION SILU3(_v REAL[][][])
RETURNS REAL[][][]
AS
$$
SELECT  ARRAY_AGG(v2)
FROM    GENERATE_SUBSCRIPTS(_v, 1) l1
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v3) AS v2
        FROM    GENERATE_SUBSCRIPTS(_v, 2) l2
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(v * EXP(v) / (1 + EXP(v))) v3
                FROM    GENERATE_SUBSCRIPTS(_v, 3) l3
                CROSS JOIN LATERAL
                        (
                        SELECT  _v[l1][l2][l3] AS v
                        ) q
                ) q
        ) q
$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION GROUP_NORM(_image REAL[][][], _weight REAL[], _bias REAL[], _groups INT)
RETURNS REAL[][]
AS
$$

SELECT  value
FROM    (
        SELECT  ARRAY_LENGTH(_image, 1) / _groups AS group_size
        ) c
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(var_denominator) AS var_denominator,
                ARRAY_AGG(mean) AS mean
        FROM    GENERATE_SERIES(0, ARRAY_LENGTH(_image, 1) -1, group_size) AS group_number
        CROSS JOIN LATERAL
                (
                SELECT  SQRT(VAR_SAMP(value) + 1e-05) AS var_denominator,
                        AVG(value) AS mean
                FROM    GENERATE_SERIES(0, group_size - 1) AS group_position
                CROSS JOIN LATERAL
                        GENERATE_SUBSCRIPTS(_image, 2) iy
                CROSS JOIN LATERAL
                        GENERATE_SUBSCRIPTS(_image, 3) ix
                CROSS JOIN LATERAL
                        (
                        SELECT  _image[group_number + group_position + 1][iy][ix] AS value
                        )
                ) q
        ) q1
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(l1) AS value
        FROM    GENERATE_SERIES(0, ARRAY_LENGTH(_image, 1) - 1) AS channel
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(l2) AS l1
                FROM    GENERATE_SUBSCRIPTS(_image, 2) iy
                CROSS JOIN LATERAL
                        (
                        SELECT  ARRAY_AGG(l3) AS l2
                        FROM    GENERATE_SUBSCRIPTS(_image, 3) ix
                        CROSS JOIN LATERAL
                                (
                                SELECT  (_image[channel + 1][iy][ix] - mean[channel / group_size + 1]) / var_denominator[channel / group_size + 1] *
                                        _weight[channel + 1] + _bias[channel + 1] AS l3
                                )
                        ) q
                ) q
        ) q2


$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION RESNET
        (
        _input REAL[],
        _t_emb3 REAL[],
        _norm1_weight REAL[],
        _norm1_bias REAL[],
        _conv1_weight REAL[],
        _conv1_bias REAL[],
        _time_emb_proj_weight REAL[],
        _time_emb_proj_bias REAL[],
        _norm2_weight REAL[],
        _norm2_bias REAL[],
        _conv2_weight REAL[],
        _conv2_bias REAL[],
        _conv_shortcut_weight REAL[],
        _conv_shortcut_bias REAL[]
        )
RETURNS REAL[]
AS
$$

SELECT  hs9
FROM    (
        SELECT
        ) d
CROSS JOIN LATERAL
        GROUP_NORM(_input, _norm1_weight, _norm1_bias, 32) hs2
CROSS JOIN LATERAL
        SILU3(hs2) hs3
CROSS JOIN LATERAL
        CONV2D3X3(hs3, _conv1_weight, _conv1_bias) hs4
CROSS JOIN LATERAL
        LINEAR_1_2(_t_emb3, _time_emb_proj_weight, _time_emb_proj_bias) t_emb4
CROSS JOIN LATERAL
        PLUS_3_1(hs4, t_emb4) hs5
CROSS JOIN LATERAL
        GROUP_NORM(hs5, _norm2_weight, _norm2_bias, 32) hs6
CROSS JOIN LATERAL
        SILU3(hs6) AS hs7
CROSS JOIN LATERAL
        CONV2D3X3(hs7, _conv2_weight, _conv2_bias) hs8
CROSS JOIN LATERAL
        (SELECT COALESCE(CONV2D1X1(_input, _conv_shortcut_weight, _conv_shortcut_bias), _input) input2) input2
CROSS JOIN LATERAL
        PLUS_3_3(input2, hs8) hs9

$$
LANGUAGE SQL
IMMUTABLE
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION RESNET
        (
        _input REAL[],
        _t_emb3 REAL[],
        _prefix TEXT
        )
RETURNS REAL[]
AS
$$
SELECT  RESNET(
                _input,
                _t_emb3,
                norm1.weight,
                norm1.bias,
                conv1.weight,
                conv1.bias,
                time_emb_proj.weight,
                time_emb_proj.bias,
                norm2.weight,
                norm2.bias,
                conv2.weight,
                conv2.bias,
                conv_shortcut.weight,
                conv_shortcut.bias
        ) rn2
FROM    (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.norm1'
        ) norm1 (weight, bias)
CROSS JOIN
        (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.conv1'
        ) conv1 (weight, bias)
CROSS JOIN
        (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.time_emb_proj'
        ) time_emb_proj (weight, bias)
CROSS JOIN
        (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.norm2'
        ) norm2 (weight, bias)
CROSS JOIN
        (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.conv2'
        ) conv2 (weight, bias)
LEFT JOIN
        (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.conv_shortcut'
        ) conv_shortcut (weight, bias)
ON      TRUE
$$
LANGUAGE SQL
STRICT
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION SCALED_DOT_PRODUCTION_ATTENTION
        (
        _query REAL[],
        _key REAL[],
        _value REAL[],
        _head_dim INT
        )
RETURNS REAL[]
AS
$$

SELECT  scaled_reshaped
FROM    (
        SELECT  ARRAY_LENGTH(_query, 2) / _head_dim AS heads,
                1 / SQRT(_head_dim) AS scale_factor
        ) c
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v2) AS attn_weight
        FROM    GENERATE_SERIES(0, heads - 1) head
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(v3) AS v2
                FROM    GENERATE_SUBSCRIPTS(_query, 1) qy
                CROSS JOIN LATERAL
                        (
                        SELECT  ARRAY_AGG(vexp) AS vexps
                        FROM    GENERATE_SUBSCRIPTS(_key, 1) ky
                        CROSS JOIN LATERAL
                                (
                                SELECT  EXP(SUM(_query[qy][head * _head_dim + x + 1] * _key[ky][head * _head_dim + x + 1]) * scale_factor) AS vexp
                                FROM    GENERATE_SERIES(0, _head_dim - 1) x
                                ) l4
                        ) l3
                CROSS JOIN LATERAL
                        (
                        SELECT  SUM(vexp) AS denominator
                        FROM    UNNEST(vexps) vexp
                        ) q
                CROSS JOIN LATERAL
                        (
                        SELECT  ARRAY_AGG(vexp / denominator) AS v3
                        FROM    UNNEST(vexps) vexp
                        ) q2
                ) l2
        ) hs1
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v2) AS scaled_reshaped
        FROM    GENERATE_SUBSCRIPTS(attn_weight, 2) y
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(v3) AS v2
                FROM    GENERATE_SERIES(0, heads - 1) head
                CROSS JOIN LATERAL
                        GENERATE_SERIES(0, _head_dim - 1 + head - head) hc
                CROSS JOIN LATERAL
                        (
                        SELECT  SUM(attn_weight[head + 1][y][ax] * _value[ax][head * _head_dim + hc + 1]) AS v3
                        FROM    GENERATE_SUBSCRIPTS(attn_weight, 3) ax
                        ) l3
                ) l2
        ) hs2
$$
LANGUAGE SQL
IMMUTABLE
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION ATTN
        (
        _input REAL[],
        _group_norm_weight REAL[],
        _group_norm_bias REAL[],
        _to_q_weight REAL[],
        _to_q_bias REAL[],
        _to_k_weight REAL[],
        _to_k_bias REAL[],
        _to_v_weight REAL[],
        _to_v_bias REAL[],
        _to_out_weight REAL[],
        _to_out_bias REAL[]
        )
RETURNS REAL[]
AS
$$

SELECT  attn
FROM    (
        SELECT  ARRAY_LENGTH(_input, 1) AS channel,
                ARRAY_LENGTH(_input, 2) AS height,
                ARRAY_LENGTH(_input, 3) AS width
        ) c
CROSS JOIN LATERAL
        GROUP_NORM(_input, _group_norm_weight, _group_norm_bias, 32) hs2
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v2) AS hs3
        FROM    GENERATE_SUBSCRIPTS(hs2, 2) l2
        CROSS JOIN LATERAL
                GENERATE_SUBSCRIPTS(hs2, 3 + (l2 - l2)) l3
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(hs2[l1][l2][l3]) v2
                FROM    GENERATE_SUBSCRIPTS(hs2, 1) l1
                )
        ) hs3
CROSS JOIN LATERAL
        LINEAR_2_2(hs3, _to_q_weight, _to_q_bias) query
CROSS JOIN LATERAL
        LINEAR_2_2(hs3, _to_k_weight, _to_k_bias) key
CROSS JOIN LATERAL
        LINEAR_2_2(hs3, _to_v_weight, _to_v_bias) value
CROSS JOIN LATERAL
        SCALED_DOT_PRODUCTION_ATTENTION(query, key, value, 8) scaled_reshaped
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v2) AS hs4
        FROM    GENERATE_SUBSCRIPTS(_to_out_weight, 1) oy
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(v3) AS v2
                FROM    GENERATE_SERIES(0, height - 1) ch
                CROSS JOIN LATERAL
                        (
                        SELECT  ARRAY_AGG(v4) AS v3
                        FROM    GENERATE_SERIES(0, width - 1) cw
                        CROSS JOIN LATERAL
                                (
                                SELECT  ch * height + cw + 1 AS scy
                                ) q
                        CROSS JOIN LATERAL
                                (
                                SELECT  SUM(scaled_reshaped[scy][x] * _to_out_weight[oy][x]) + _to_out_bias[oy] AS v4
                                FROM    GENERATE_SUBSCRIPTS(scaled_reshaped, 2) x
                                ) l4
                        ) l3
                ) l2 
        ) hs4
CROSS JOIN LATERAL
        PLUS_3_3(hs4, _input) AS attn

$$
LANGUAGE SQL
IMMUTABLE
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION ATTN
        (
        _input REAL[],
        _prefix TEXT
        )
RETURNS REAL[]
AS
$$
SELECT  ATTN(
                _input,
                group_norm.weight,
                group_norm.bias,
                to_q.weight,
                to_q.bias,
                to_k.weight,
                to_k.bias,
                to_v.weight,
                to_v.bias,
                to_out.weight,
                to_out.bias
        ) rn2
FROM    (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.group_norm'
        ) group_norm (weight, bias)
CROSS JOIN
        (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.to_q'
        ) to_q (weight, bias)
CROSS JOIN
        (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.to_k'
        ) to_k (weight, bias)
CROSS JOIN
        (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.to_v'
        ) to_v (weight, bias)
CROSS JOIN
        (
        SELECT  weight::TEXT::REAL[], bias::TEXT::REAL[]
        FROM    parameters
        WHERE   key = _prefix || '.to_out.0'
        ) to_out (weight, bias)
$$
LANGUAGE SQL
STRICT
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION UPSAMPLE_NEAREST2D_3_SCALE_2(_image REAL[][][])
RETURNS REAL[][][]
AS
$$

SELECT  ARRAY_AGG(v1)
FROM    GENERATE_SUBSCRIPTS(_image, 1) l1
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v2) AS v1
        FROM    GENERATE_SUBSCRIPTS(_image, 2) l2
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, 1 + l2 - l2)
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(_image[l1][l2][l3]) AS v2
                FROM    GENERATE_SUBSCRIPTS(_image, 3) l3
                CROSS JOIN LATERAL
                        GENERATE_SERIES(0, 1 + l3 - l3)
                ) l3
        ) l2

$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

Here's the implementation of the whole U-Net function:

CREATE OR REPLACE FUNCTION UNET(_image REAL[][][], _step INT)
RETURNS REAL[][][]
AS
$$

SELECT  out
FROM    EMB(_step) t_emb
CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'time_embedding.linear_1') t_embp1 (weight, bias)
CROSS JOIN LATERAL LINEAR_1_2(t_emb, t_embp1.weight, t_embp1.bias) t_emb2
CROSS JOIN SILU(t_emb2) t_emb3
CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'time_embedding.linear_2') t_embp2 (weight, bias)
CROSS JOIN LATERAL LINEAR_1_2(t_emb3, t_embp2.weight, t_embp2.bias) t_emb4
CROSS JOIN SILU(t_emb4) t_emb5
CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'conv_in') conv_in (weight, bias)
CROSS JOIN LATERAL CONV2D3X3(_image, conv_in.weight, conv_in.bias) input1

-- Down 1
        
CROSS JOIN LATERAL RESNET(input1, t_emb5, 'down_blocks.0.resnets.0') db0rn0
CROSS JOIN LATERAL RESNET(db0rn0, t_emb5, 'down_blocks.0.resnets.1') db0rn1
CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'down_blocks.0.downsamplers.0.conv') db0ds (weight, bias)
CROSS JOIN LATERAL CONV2D3X3(db0rn1, db0ds.weight, db0ds.bias, 2) AS db0

-- Down 2

CROSS JOIN LATERAL RESNET(db0, t_emb5, 'down_blocks.1.resnets.0') db1rn0
CROSS JOIN LATERAL RESNET(db1rn0, t_emb5, 'down_blocks.1.resnets.1') db1rn1
CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'down_blocks.1.downsamplers.0.conv') db1ds (weight, bias)
CROSS JOIN LATERAL CONV2D3X3(db1rn1, db1ds.weight, db1ds.bias, 2) AS db1

-- Down 3 with Attention

CROSS JOIN LATERAL RESNET(db1, t_emb5, 'down_blocks.2.resnets.0') db2rn0
CROSS JOIN LATERAL ATTN(db2rn0, 'down_blocks.2.attentions.0') db2att0
CROSS JOIN LATERAL RESNET(db2att0, t_emb5, 'down_blocks.2.resnets.1') db2rn1
CROSS JOIN LATERAL ATTN(db2rn1, 'down_blocks.2.attentions.1') db2att1
CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'down_blocks.2.downsamplers.0.conv') db2ds (weight, bias)
CROSS JOIN LATERAL CONV2D3X3(db2att1, db2ds.weight, db2ds.bias, 2) AS db2

-- Down 4 with Attention

CROSS JOIN LATERAL RESNET(db2, t_emb5, 'down_blocks.3.resnets.0') db3rn0
CROSS JOIN LATERAL ATTN(db3rn0, 'down_blocks.3.attentions.0') db3att0
CROSS JOIN LATERAL RESNET(db3att0, t_emb5, 'down_blocks.3.resnets.1') db3rn1
CROSS JOIN LATERAL ATTN(db3rn1, 'down_blocks.3.attentions.1') db3

-- Mid

CROSS JOIN LATERAL RESNET(db3, t_emb5, 'mid_block.resnets.0') mbrn0
CROSS JOIN LATERAL ATTN(mbrn0, 'mid_block.attentions.0') mbatt0
CROSS JOIN LATERAL RESNET(mbatt0, t_emb5, 'mid_block.resnets.1') mb

-- Up 1 with Attention

CROSS JOIN LATERAL RESNET(mb || db3, t_emb5, 'up_blocks.0.resnets.0') ub0rn0
CROSS JOIN LATERAL ATTN(ub0rn0, 'up_blocks.0.attentions.0') ub0att0
CROSS JOIN LATERAL RESNET(ub0att0 || db3att0, t_emb5, 'up_blocks.0.resnets.1') ub0rn1
CROSS JOIN LATERAL ATTN(ub0rn1, 'up_blocks.0.attentions.1') ub0att1
CROSS JOIN LATERAL RESNET(ub0att1 || db2, t_emb5, 'up_blocks.0.resnets.2') ub0rn2
CROSS JOIN LATERAL ATTN(ub0rn2, 'up_blocks.0.attentions.2') ub0att2
CROSS JOIN UPSAMPLE_NEAREST2D_3_SCALE_2(ub0att2) ub0us1
CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'up_blocks.0.upsamplers.0.conv') ub0us (weight, bias)
CROSS JOIN CONV2D3X3(ub0us1, ub0us.weight, ub0us.bias) ub0

-- Up 2 with Attention

CROSS JOIN LATERAL RESNET(ub0 || db2att1, t_emb5, 'up_blocks.1.resnets.0') ub1rn0
CROSS JOIN LATERAL ATTN(ub1rn0, 'up_blocks.1.attentions.0') ub1att0
CROSS JOIN LATERAL RESNET(ub1att0 || db2att0, t_emb5, 'up_blocks.1.resnets.1') ub1rn1
CROSS JOIN LATERAL ATTN(ub1rn1, 'up_blocks.1.attentions.1') ub1att1
CROSS JOIN LATERAL RESNET(ub1att1 || db1, t_emb5, 'up_blocks.1.resnets.2') ub1rn2
CROSS JOIN LATERAL ATTN(ub1rn2, 'up_blocks.1.attentions.2') ub1att2
CROSS JOIN UPSAMPLE_NEAREST2D_3_SCALE_2(ub1att2) ub1us1
CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'up_blocks.1.upsamplers.0.conv') ub1us (weight, bias)
CROSS JOIN CONV2D3X3(ub1us1, ub1us.weight, ub1us.bias) ub1

-- Up 3

CROSS JOIN LATERAL RESNET(ub1 || db1rn1, t_emb5, 'up_blocks.2.resnets.0') ub2rn0
CROSS JOIN LATERAL RESNET(ub2rn0 || db1rn0, t_emb5, 'up_blocks.2.resnets.1') ub2rn1
CROSS JOIN LATERAL RESNET(ub2rn1 || db0, t_emb5, 'up_blocks.2.resnets.2') ub2rn2
CROSS JOIN UPSAMPLE_NEAREST2D_3_SCALE_2(ub2rn2) ub2us1
CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'up_blocks.2.upsamplers.0.conv') ub2us (weight, bias)
CROSS JOIN CONV2D3X3(ub2us1, ub2us.weight, ub2us.bias) ub2

-- Up 4

CROSS JOIN LATERAL RESNET(ub2 || db0rn1, t_emb5, 'up_blocks.3.resnets.0') ub3rn0
CROSS JOIN LATERAL RESNET(ub3rn0 || db0rn0, t_emb5, 'up_blocks.3.resnets.1') ub3rn1
CROSS JOIN LATERAL RESNET(ub3rn1 || input1, t_emb5, 'up_blocks.3.resnets.2') ub3

-- Decode

CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'conv_norm_out') conv_norm_out (weight, bias)
CROSS JOIN LATERAL GROUP_NORM(ub3, conv_norm_out.weight, conv_norm_out.bias, 32) conv_norm_out1
CROSS JOIN LATERAL SILU3(conv_norm_out1) conv_norm_out2
CROSS JOIN (SELECT weight::TEXT::REAL[], bias::TEXT::REAL[] FROM parameters WHERE key = 'conv_out') conv_out (weight, bias)
CROSS JOIN LATERAL CONV2D3X3(conv_norm_out2, conv_out.weight, conv_out.bias) out

$$
LANGUAGE SQL
STRICT
LEAKPROOF
PARALLEL SAFE;

The model parameters are stored in PostgreSQL arrays in a table. At the bottom of this post, you will find a link to the GitHub repository with the source code of all the functions and a script to download the model from HuggingFace and upload it to the database.

Let's try running a single pass of the U-Net on a random input and see where it gets us. This query generates a 3×64×64 array of random pixels and pipes it through U-Net. The result is also an array of the same shape. It's supposed to represent an estimation of noise: a vector, which, when subtracted from the input, would yield having some features of a butterfly.

For brevity, I'm only returning the first two items from each of the resulting array's dimensions, and also its general shape.

Here's the query:

SELECT SETSEED(0.20241231);

WITH    initial_image (input) AS MATERIALIZED
        (
        SELECT  ARRAY_AGG(v)
        FROM    GENERATE_SERIES(1, 3) l1
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(v) v
                FROM    GENERATE_SERIES(1, 64) l2
                CROSS JOIN LATERAL
                        (
                        SELECT  ARRAY_AGG(RANDOM()::REAL) v
                        FROM    GENERATE_SERIES(1, 64) l3
                        ) l3
                ) l3
        )
SELECT  unet[1:2][1:2][1:2], ARRAY_DIMS(unet)
FROM    UNET((SELECT input FROM initial_image), 999) unet

It works and takes a whopping 24 minutes on my machine. Just a single pass through U-Net.

Doing this 1000 times in a row would take 24000 minutes, or a little under 17 days. And New Year's Eve is only hours away.

Shit.

Can we fix it somehow?

Skipping the steps

Earlier, we told our model not to make hasty assumptions about its predictions and instead do the reconstruction of the images in smaller steps. Instead of cancelling predicted noise all at once, that is, going from "sample" to "sample minus the noise" for every pixel, we would treat it as a general direction. Mathematically, it would mean that we should add a certain percentage of the original pixel value with a certain percentage of the denoised pixel. We do just enough of that so as to end up at the level of noise that would have been added on the previous step. Then we would run the model again and make a more correct prediction.

What if we want to skip some steps? Instead of cancelling just a little bit of noise and ending up at the previous step, we would cancel more of it and end up several steps earlier. Of course, that would throw all the math off, and we would not arrive at the same distribution of images that going through every step separately would end us. However, for the purposes of making ASCII SQL art in a terminal, it could be just enough.

The model that we are using is trained to evaluate the noise vector. It could have been made to evaluate the original image or the mean of the distribution of the previous image, conditioned on the subsequent one. Mathematically, it's all the same, and closed-form expressions exist to convert between all three of these values.

However, the trained model that we are using is not exactly following the path outlined in the paper. Its backbone function returns the estimation of noise, and from this estimation, on each step, the image from the previous step is sampled. The closed-form expression for this operation, per the paper, is this:

But the model doesn't use this expression. It first calculates the estimation of the original image, then clamps it to the range of , and then samples a point from the distribution on the target step, conditioned on the clamped estimation of the original image:

Now, let me digress a little while it's still fresh in my memory. Remember the pseudocode that demonstrated the use of the noise scheduler? There was a mysterious line saying "add noise." The last term in the expression, , is doing just that. is sampled from the standard Gaussian distribution and is calculated from the noise schedule. Now, why would we want to add back noise that we were working so hard to remove?

The expression in the brackets has the same mean as the distribution of images that would be generated by the Markov chain, but lower variance. In practice it means that had we used this expression, the model would be less creative than it could be. The set of results it would be likely to generate would be narrower and closer to the model's "idea" of an idealized, generic butterfly.

Here's what we get if we don't add this noise:

Those all are butterflies alright, and they're all even technically different. But, I mean, come on. According to the model, this is what an "average" butterfly looks like, and if we sample from the distribution that doesn't vary a lot, all we get are samples from the closest vicinity of that "average."

By increasing the variance and sampling from the theoretic distribution, we make sure that the model can generate a wider range of images that we humans would still recognize as a butterfly.

Ok, back to skipping the steps. The expression above can be generalized to produce the estimation of the image on an arbitrary step:

The expression for the sigma doesn't match the theoretical value, but it's good enough for our purposes.

Experiments show that making just four steps on the following schedule: , is enough to generate realistic enough images of butterflies:

To implement the scheduler, we'll need to create several more SQL functions:

CREATE OR REPLACE FUNCTION PREDICT_PREVIOUS(_image REAL[][][], _noise REAL[][][], _t INT, _t2 INT)
RETURNS REAL[][][]
AS
$$
WITH    alphas_p AS
        (
        SELECT  EXP(SUM(LN(1 - ((0.02 - 0.0001) * step / 999 + 0.0001))) OVER (ORDER BY step)) AS alphas_p,
                step
        FROM    GENERATE_SERIES(0, 999) step
        )
SELECT  v1
FROM    (
        SELECT  alphas_p AS alphas_p_t
        FROM    alphas_p
        WHERE   step = _t
        ) q_t
CROSS JOIN LATERAL
        (
        SELECT  alphas_p AS alphas_p_t2
        FROM    alphas_p
        WHERE   step = _t2
        ) q_t2
CROSS JOIN LATERAL
        (
        SELECT  SQRT(alphas_p_t2) * (1 - alphas_p_t / alphas_p_t2) / (1 - alphas_p_t) AS x0_c,
                SQRT(alphas_p_t / alphas_p_t2) * (1 - alphas_p_t2) / (1 - alphas_p_t) AS xt2_c
        )
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v2) AS v1
        FROM    GENERATE_SUBSCRIPTS(_image, 1) l1
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(v3) AS v2
                FROM    GENERATE_SUBSCRIPTS(_image, 2) l2
                CROSS JOIN LATERAL
                        (
                        SELECT  ARRAY_AGG(image_t2) AS v3
                        FROM    GENERATE_SUBSCRIPTS(_image, 3) l3
                        CROSS JOIN LATERAL
                                (
                                        SELECT  _image[l1][l2][l3] AS image,
                                                _noise[l1][l2][l3] AS noise
                                ) q
                        CROSS JOIN LATERAL
                                (
                                SELECT  GREATEST(LEAST((image - noise * SQRT(1 - alphas_p_t)) / SQRT(alphas_p_t), 1), -1) AS x0
                                ) x0
                        CROSS JOIN LATERAL
                                (
                                SELECT  x0_c * x0 + xt2_c * image AS image_t2
                                ) image_t2
                        ) l3
                ) l2
        ) l1
$$
LANGUAGE SQL
IMMUTABLE
STRICT
LEAKPROOF
PARALLEL SAFE;

CREATE OR REPLACE FUNCTION GENERATE_NOISE()
RETURNS REAL[][][]
AS
$$
WITH    initial_image (input) AS MATERIALIZED
        (
        SELECT  ARRAY_AGG(v)
        FROM    GENERATE_SERIES(1, 3) l1
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG(v) v
                FROM    GENERATE_SERIES(1, 64) l2
                CROSS JOIN LATERAL
                        (
                        -- Introduce dependency, otherwise GENERATE_SERIES(1, 64) will be cached and generate duplicates
                        -- The distribution is uniform, but has the same mean and variance as the standard normal distribution
                        SELECT  ARRAY_AGG((RANDOM()::REAL * 2 - 1) * 1.73 + l1 + l2 + l3 - l1 - l2 - l3) v
                        FROM    GENERATE_SERIES(1, 64) l3
                        ) l3
                ) l3
        )
SELECT  input
FROM    initial_image
$$
LANGUAGE SQL;

CREATE OR REPLACE FUNCTION GENERATE_CREATIVE_NOISE(_t INT, _t2 INT)
RETURNS REAL[][][]
AS
$$
WITH    alphas_p AS
        (
        SELECT  EXP(SUM(LN(1 - ((0.02 - 0.0001) * step / 999 + 0.0001))) OVER (ORDER BY step)) AS alphas_p,
                step
        FROM    GENERATE_SERIES(0, 999) step
        )
SELECT  ARRAY_AGG(v)
FROM    (
        SELECT  alphas_p AS alphas_p_t
        FROM    alphas_p
        WHERE   step = _t
        ) q_t
CROSS JOIN LATERAL
        (
        SELECT  alphas_p AS alphas_p_t2
        FROM    alphas_p
        WHERE   step = _t2
        ) q_t2
CROSS JOIN LATERAL
        (
        SELECT  SQRT((1 - alphas_p_t2)  / (1 - alphas_p_t) * (1 - alphas_p_t / alphas_p_t2)) AS creative_noise_c
        ) q1
CROSS JOIN LATERAL
        GENERATE_SERIES(1, 3) l1
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(v) v
        FROM    GENERATE_SERIES(1, 64) l2
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG((RANDOM()::REAL * 2 - 1) * 1.73 * creative_noise_c + l1 + l2 + l3 - l1 - l2 - l3) v
                FROM    GENERATE_SERIES(1, 64) l3
                ) l3
        ) l3
$$
LANGUAGE SQL;

The image

Let's run the scheduler!

The query below completes in 101 minutes and generates an image of a butterfly:

WITH    initial_image (input) AS MATERIALIZED
        (
        SELECT  GENERATE_NOISE()
        )
SELECT  v2
FROM    (SELECT)
CROSS JOIN LATERAL UNET((SELECT input FROM initial_image), 500) unet1
CROSS JOIN LATERAL PREDICT_PREVIOUS((SELECT input FROM initial_image), unet1, 500, 250) step1
CROSS JOIN LATERAL GENERATE_CREATIVE_NOISE(500, 250) noise1
CROSS JOIN PLUS_3_3(step1, noise1) image2
CROSS JOIN LATERAL UNET(image2, 250) unet2
CROSS JOIN LATERAL PREDICT_PREVIOUS(image2, unet2, 250, 125) step2
CROSS JOIN LATERAL GENERATE_CREATIVE_NOISE(250, 125) noise2
CROSS JOIN PLUS_3_3(step2, noise2) image3
CROSS JOIN LATERAL UNET(image3, 125) unet3
CROSS JOIN LATERAL PREDICT_PREVIOUS(image3, unet3, 125, 60) step3
CROSS JOIN LATERAL GENERATE_CREATIVE_NOISE(125, 60) noise3
CROSS JOIN PLUS_3_3(step3, noise3) image4
CROSS JOIN LATERAL UNET(image4, 60) unet4
CROSS JOIN LATERAL PREDICT_PREVIOUS(image4, unet4, 60, 0) butterfly
CROSS JOIN LATERAL
        (
        SELECT  STRING_TO_ARRAY(' `.-'':_,^=;><+!rc*/z?sLTv)J7(|Fi{C}fI31tluneoZ5Yxjya]2ESwqkP6h9d4VpOGbUAKXHm8RD#Bg0MNWQ%&@', NULL) AS codes
        )
CROSS JOIN LATERAL
        (
        SELECT  v2
        FROM    GENERATE_SUBSCRIPTS(butterfly, 2) y
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(code, '') v2
                FROM    GENERATE_SUBSCRIPTS(butterfly, 3) x
                CROSS JOIN LATERAL
                        (
                        SELECT  FLOOR(LEAST(GREATEST(AVG(-v) * 2 - 1 + 1, 0), 1) * (ARRAY_LENGTH(codes, 1) - 1))::INT + 1 AS brightness
                        FROM    UNNEST(butterfly[1:3][y:y][x:x]) v
                        )
                CROSS JOIN LATERAL
                        (
                        SELECT  codes[brightness] AS code
                        )
                )
        )

May the New Year bring you bright colors and make you spread your wings like a butterfly!

You can find the queries and the installation code in the GitHub repository: quassnoi/explain-extended-2025

Previous New Year posts:

• 2010: SQL graphics in Oracle, MySQL, SQL Server and PostgreSQL
• 2011: Drawing a clock in SQL
• 2012: Drawing snowflakes in SQL
• 2013: View of Earth from space in SQL
• 2014: Drawing fractals in SQL
• 2015: Composing music in SQL
• 2016: Conway’s Game of Life in SQL
• 2017: The Sultan’s Riddle in SQL
• 2018: Settlers of Catan in SQL
• 2019: GIF decoder in SQL
• 2020: A stereogram in SQL
• 2021: 3D picture of the coronavirus in SQL
• 2022: Quantum computer emulator in SQL
• 2023: Solving the Rubik’s Cube in SQL
• 2024: GPT in 500 lines of SQL

The post Happy New Year: Diffusion Model image generator in about 700 lines of pure SQL appeared first on EXPLAIN EXTENDED.

This year, the talk of the town was AI and how it can do everything for you.

I like it when someone or something does everything for me. To this end, I decided to ask ChatGPT to write my New Year's post:

"Hey ChatGPT. Can you implement a large language model in SQL?"
"No, SQL is not suitable for implementing large language models. SQL is a language for managing and querying data in relational databases. Implementing a language model requires complex algorithms, large datasets, and often deep learning frameworks, which are beyond the capabilities of SQL."

It just proves that if you want something done right, you have to do it yourself.

Encouraged by this optimistic forecast, today we will implement a large language model in SQL.

Theory

While writing this post, I used the wonderful article GPT in 60 Lines of NumPy by Jay Mody. This article explains the inner workings of a GPT model much better than I can hope to do. Still, a little recap is in order.

What is a generative large language model from a technical perspective?

A generative LLM is a function. It takes a text string as input (called "prompt" in AI parlance), and returns an array of strings and numbers. Here's what the signature of this function looks like:

llm(prompt: str) -> list[tuple[str, float]]

This function is deterministic. It does a lot of math under the hood, but all this math is hardwired. If you call it repeatedly with the same input, it will always return the same output.

It may come as a surprise to anyone who's been using ChatGPT and similar products because they can give different answers to the same question. Yet, it's true. We will shortly see how it works.

What are the values this function returns?

Something like this:

llm("I wish you a happy New")

0       (' Year', 0.967553)
1       (' Years', 0.018199688)
2       (' year', 0.003573329)
3       (' York', 0.003114716)
4       (' New', 0.0009022804)
…
50252   (' carbohyd', 2.3950911e-15)
50253   (' volunte', 2.2590102e-15)
50254   ('pmwiki', 1.369229e-15)
50255   (' proport', 1.1198108e-15)
50256   (' cumbers', 7.568147e-17)

It returns an array of tuples. Each tuple consists of a word (or, rather, a string) and a number. The number is the probability that this word will continue the prompt. The model "thinks" that the phrase "I wish you a happy New" will be followed by the character sequence " Year" with a probability of 96.7%, " Years" of 1.8% and so on.

The word "think" above is quoted because, of course, the model doesn't really think. It mechanically returns arrays of words and numbers according to some hardwired internal logic.

If it's that dumb and deterministic, how can it generate different texts?

Large language models are used in text applications (chatbots, content generators, code assistants etc). These applications repeatedly call the model and select the word suggested by it (with some degree of randomness). The next suggested word is added to the prompt and the model is called again. This continues in a loop until enough words are generated.

The accrued sequence of words will look like a text in a human language, complete with grammar, syntax and even what appears to be intelligence and reasoning. In this aspect, it is not unlike a Markov chain which works on the same principle.

The internals of a large language model are wired up so that the next suggested word will be a natural continuation of the prompt, complete with its grammar, semantics and sentiment. Equipping a function with such a logic became possible through a series of scientific breakthroughs (and programming drudgery) that have resulted in the development of the family of algorithms known as GPT, or Generative Pre-trained Transformer.

What does "Generative Pre-trained Transformer" mean?

"Generative" means that it generates text (by adding continuations to the prompt recursively, as we saw earlier).

"Transformer" means that it uses a particular type of neural network, first developed by Google and described in this paper.

"Pre-trained" is a little bit historical. Initially, the ability for the model to continue text was thought of as just a prerequisite for a more specialized task: inference (finding logical connections between phrases), classification (for instance, guessing the number of stars in a hotel rating from the text of the review), machine translation and so on. It was thought that these two parts should have been trained separately, the language part being just a pre-training for a "real" task that would follow.

As the original GPT paper puts it:

We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task.

It was not until later that people realized that, with a model large enough, the second step was often not necessary. A Transformer model, trained to do nothing else than generate texts, turned out to be able to follow human language instructions that were contained in these texts, with no additional training ("fine-tuning" in AI parlance) required.

With that out of the way, let's focus on the implementation.

Generation

Here is what happens when we try to generate text from the prompt using GPT2:

def generate(prompt: str) -> str:
  # Transforms a string into a list of tokens.
  tokens = tokenize(prompt) # tokenize(prompt: str) -> list[int]

  while True:

    # Runs the algorithm.
    # Returns tokens' probabilities: a list of 50257 floats, adding up to 1.
    candidates = gpt2(tokens) # gpt2(tokens: list[int]) -> list[float]

    # Selects the next token from the list of candidates
    next_token = select_next_token(candidates)
    # select_next_token(candidates: list[float]) -> int

    # Append it to the list of tokens
    tokens.append(next_token)

    # Decide if we want to stop generating.
    # It can be token counter, timeout, stopword or something else.
    if should_stop_generating():
      break

  # Transform the list of tokens into a string
  completion = detokenize(tokens) # detokenize(tokens: list[int]) -> str
  return completion

Let's implement all these pieces one by one in SQL.

Tokenizer

Before a text can be fed to a neural network, it needs to be converted into a list of numbers. Of course, that's barely news: that's what text encodings like Unicode do. Plain Unicode, however, doesn't really work well with neural networks.

Neural networks, at their core, do a lot of matrix multiplications and capture whatever predictive powers they have in the coefficients of these matrixes. Some of these matrixes have one row per every possible value in the "alphabet"; others have one row per "character".

Here, the words "alphabet" and "character" don't have the usual meaning. In Unicode, the "alphabet" is 149186 characters long (this is how many different Unicode points there are at the time of this writing), and a "character" can be something like this: ﷽ (yes, that's a single Unicode point number 65021, encoding a whole phrase in Arabic that is particularly important for the Muslims). Note that the very same phrase could have been written in usual Arabic letters. It means that the same text can have many encodings.

As an illustration, let's take the word "PostgreSQL". If we were to encode it (convert to an array of numbers) using Unicode, we would get 10 numbers that could potentially be from 1 to 149186. It means that our neural network would need to store a matrix with 149186 rows in it and perform a number of calculations on 10 rows from this matrix. Some of these rows (corresponding to the letters of the English alphabet) would be used a lot and pack a lot of information; others, like poop emoji and obscure symbols from dead languages, would hardly be used at all, but still take up space.

Naturally, we want to keep both these numbers, the "alphabet" length and the "character" count, as low as possible. Ideally, all the "characters" in our alphabet should be distributed uniformly, and we still want our encoding to be as powerful as Unicode.

The way we can do that, intuitively, is to assign unique numbers to sequences of words that occur often in the texts we work with. In Unicode, the same religious phrase in Arabic can be encoded using either a single code point, or letter by letter. Since we are rolling our own encoding, we can do the same for the words and phrases that are important for the model (i.e. show up often in texts).

For instance, we could have separate numbers for "Post", "greSQL" and "ing". This way, the words "PostgreSQL" and "Posting" would both have a length of 2 in our representation. And of course, we would still maintain separate code points for shorter sequences and individual bytes. Even if we come across gibberish or a text in a foreign language, it would still be encodable, albeit longer.

GPT2 uses a variation of the algorithm called Byte pair encoding to do precisely that. Its tokenizer uses a dictionary of 50257 code points (in AI parlance, "tokens") that correspond to different byte sequences in UTF-8 (plus the "end of text" as a separate token).

This dictionary was built by statistical analysis performed like this:

1. Start with a simple encoding of 256 tokens: one token per byte.
2. Take a large corpus of texts (preferably the one the model will be trained on).
3. Encode it.
4. Calculate which pair of tokens is the most frequent. Let's assume it's 0x20 0x74 (space followed by the lowercase "t").
5. Assign the next available value (257) to this pair of bytes.
6. Repeat the steps 3-5, now paying attention to the byte sequences. If a sequence of bytes can be encoded with a complex token, use the complex token. If there are ambiguities (say, "abc" can, at some point, be encoded as "a" + "bc" or "ab" + "c"), use the one with the lowest number (because it was added earlier and hence is more frequent). Do this recursively until all sequences that can collapse into a single token will collapse into a single token.
7. Perform the collapse 50000 times over.

The number 50000 was chosen more or less arbitrarily by the developers. Other models keep the number of tokens in a similar range (from 30k to 100k).

At every iteration of this algorithm, a new token that is a concatenation of two previous ones will be added to the dictionary. Ultimately, we will end up with 50256 tokens. Add a fixed-number token for "end-of-text", and we're done.

The GPT2 version of BTE has another layer of encoding: the token dictionary maps tokens to strings and not arrays of bytes. Mapping from bytes to string characters is defined in this function. We will save the dictionary it produces in the table encoder.

Let's see how we can implement the tokenizer in SQL.

The tokenizer is an integral part of GPT2, and the token dictionary can be downloaded from OpenAI's website along with the rest of the model. We will need to import it into the table tokenizer. At the bottom of this post, you will find a link to the code repository. Its code will automate populating database tables needed for the model.

In a recursive CTE, we will split this word into tokens (starting with single bytes) and merge the best adjacent pairs, until there is nothing left to merge. The merging itself happens in a nested recursive CTE.

For the demo, I will use the word "Mississippilessly". Each record in the resultset shows the best pair to collapse found so far, and also the progress through the query.

WITH    RECURSIVE
        bpe AS
        (
        SELECT  (n + 1)::BIGINT AS position, character, TRUE AS continue, 1 AS step,
                NULL::INT AS token, NULL::TEXT AS combined
        FROM    CONVERT_TO('Mississippilessly', 'UTF-8') AS bytes
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, LENGTH(bytes) - 1) AS n
        JOIN    encoder
        ON      byte = GET_BYTE(bytes, n)
        UNION ALL
        (
        WITH    RECURSIVE
                base AS
                (
                SELECT  *
                FROM    bpe
                WHERE   continue
                ),
                bn AS
                (
                SELECT  ROW_NUMBER() OVER (ORDER BY position) AS position,
                        continue,
                        character,
                        character || LEAD(character) OVER (ORDER BY position) AS cluster
                FROM    base
                ),
                top_rank AS
                (
                SELECT  tokenizer.*
                FROM    bn
                CROSS JOIN LATERAL
                        (
                        SELECT  *
                        FROM    tokenizer
                        WHERE   tokenizer.cluster = bn.cluster
                        LIMIT   1
                        ) tokenizer
                ORDER BY
                        token
                LIMIT   1
                ),
                breaks AS
                (
                SELECT  0::BIGINT AS position, 1 AS length
                UNION ALL
                SELECT  bn.position,
                        CASE WHEN token IS NULL THEN 1 ELSE 2 END
                FROM    breaks
                JOIN    bn
                ON      bn.position = breaks.position + length
                LEFT JOIN
                        top_rank
                USING   (cluster)
                )
        SELECT  position, character, token IS NOT NULL,
                (SELECT step + 1 FROM base LIMIT 1), token, top_rank.cluster
        FROM    breaks
        LEFT JOIN
                top_rank
        ON      1 = 1
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(character, '' ORDER BY position) AS character
                FROM    bn
                WHERE   bn.position >= breaks.position
                        AND bn.position < breaks.position + length
                ) bn
        WHERE   position > 0
        )
        )
SELECT  step, MAX(token) AS token, MAX(combined) AS combined, ARRAY_AGG(character ORDER BY position)
FROM    bpe
WHERE   continue
GROUP BY
        step
ORDER BY
        step

On each step, the BPE algorithm finds the best pair of tokens to merge and merges them (you can see the merged pair and its rank in the output). This procedure brings down the token space size from Unicode's 150k to 50k, and the number of tokens (in this particular word) from 17 to 5. Both are great improvements.

When working with multiple words, the tokenizer first splits the text into separate words using this regexp and merges the tokens inside each word separately. Unfortunately, PostgreSQL doesn't support Unicode character properties in regexps, so I had to tweak it a little bit (probably killing proper Unicode support in the process). Here's how it looks in SQL:

WITH    input AS
        (
        SELECT  'PostgreSQL is great' AS prompt
        ),
        clusters AS
        (
        SELECT  part_position, bpe.*
        FROM    input
        CROSS JOIN LATERAL
                REGEXP_MATCHES(prompt, '''s|''t|''re|''ve|''m|''ll|''d| ?\w+| ?\d+| ?[^\s\w\d]+|\s+(?!\S)|\s+', 'g') WITH ORDINALITY AS rm (part, part_position)
        CROSS JOIN LATERAL
                (
                WITH    RECURSIVE
                        bpe AS
                        (
                        SELECT  (n + 1)::BIGINT AS position, character, TRUE AS continue
                        FROM    CONVERT_TO(part[1], 'UTF-8') AS bytes
                        CROSS JOIN LATERAL
                                GENERATE_SERIES(0, LENGTH(bytes) - 1) AS n
                        JOIN    encoder
                        ON      byte = GET_BYTE(bytes, n)
                        UNION ALL
                        (
                        WITH    RECURSIVE
                                base AS
                                (
                                SELECT  *
                                FROM    bpe
                                WHERE   continue
                                ),
                                bn AS
                                (
                                SELECT  ROW_NUMBER() OVER (ORDER BY position) AS position,
                                        continue,
                                        character,
                                        character || LEAD(character) OVER (ORDER BY position) AS cluster
                                FROM    base
                                ),
                                top_rank AS
                                (
                                SELECT  tokenizer.*
                                FROM    bn
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  *
                                        FROM    tokenizer
                                        WHERE   tokenizer.cluster = bn.cluster
                                        LIMIT   1
                                        ) tokenizer
                                ORDER BY
                                        token
                                LIMIT   1
                                ),
                                breaks AS
                                (
                                SELECT  0::BIGINT AS position, 1 AS length
                                UNION ALL
                                SELECT  bn.position,
                                        CASE WHEN token IS NULL THEN 1 ELSE 2 END
                                FROM    breaks
                                JOIN    bn
                                ON      bn.position = breaks.position + length
                                LEFT JOIN
                                        top_rank
                                USING   (cluster)
                                )
                        SELECT  position, character, token IS NOT NULL
                        FROM    breaks
                        LEFT JOIN
                                top_rank
                        ON      1 = 1
                        CROSS JOIN LATERAL
                                (
                                SELECT  STRING_AGG(character, '' ORDER BY position) AS character
                                FROM    bn
                                WHERE   bn.position >= breaks.position
                                        AND bn.position < breaks.position + length
                                ) bn
                        WHERE   position > 0
                        )
                        )
                SELECT  position, character AS cluster
                FROM    bpe
                WHERE   NOT continue
                ) bpe
        ),
        tokens AS
        (
        SELECT  token, cluster
        FROM    clusters
        JOIN    tokenizer
        USING   (cluster)
        )
SELECT  *
FROM    tokens

The weird character Ġ is the whitespace.

This query tokenizes the prompt and converts it into an array of numbers. This way, the prompt is ready for its journey through the layers of the model.

Embeddings

The tokens represent parts of the human languages (about 0.75 words per token, in general), so any model that is trying to succeed at text completion should somehow encode the relationships between these parts. Even in isolation, the parts of the speech have sets of orthogonal properties.

Let's take the word "subpoena" (which happens to have a whole token in itself in the GPT2 tokenizer). Is it a noun? Yes, very much so. Is it a verb? Well, sort of. Is it an adjective? Not that much, but it can be if you squint hard enough. Is it legalese? Hell yes. And so on.

All these properties are orthogonal, i.e. independent of each other. A word can be a legalese noun but not an adjective or a verb. In English, any combination thereof can happen.

Things with orthogonal properties are best encoded using vectors. Instead of having a single property (like a token number), we can have many. And it helps if we can wiggle them as we want. For instance, for a word to continue the phrase "A court decision cited by the lawyer mentions the …" we would probably want something that's heavy on the legalese dimension and at the same time heavy on being a noun. We don't really care if it has a side hustle being an adjective, a verb, or a flower.

In math, mapping narrower values into wider spaces (such as token IDs to vectors) is called an embedding. This is exactly what we are doing here.

How do we decide which properties these vectors represent? We don't. We just provide enough vector space for every token and hope that the model during its training phase will populate these dimensions with something meaningful. GPT2 uses 768 dimensions for its vectors. There is no telling in advance (and, actually, even in the retrospective) what property of the word will, say, the dimension 247 encode. Surely it would encode something, but it's not easy to tell what it is.

What properties of each token do we want to embed in the vector space? Anything that has any bearing on what the next token would be.

Token id? Of course. Different tokens mean different things.

Position of the token in the text? Yes, please. "Blue violet" and "violet blue" are not the same thing.

Relationships of tokens to each other? Sure! That's, probably, the most important part of the job, and the Attention block of the Transformer architecture was the first one to get it right.

Tokens and positions are easy to embed. Let's say we have the phrase "PostgreSQL is great", which, as we already know, maps to four tokens: [6307, 47701, 318, 1049].

Among other parameters of GPT2, there are two matrixes called WTE (word token embedding) and WPE (word position embedding). As the names suggest, the former stores embeddings of the tokens, and the latter stores embeddings of the positions. The actual values of these embeddings have been populated ("learned") during the training of GPT2. As far as we are concerned, they are constants that live in the database tables wte and wpe.

WTE is 50257×768 and WPE is 1024×768. The latter means that the maximum number of tokens that we can use in a prompt to GPT2 is 1024. If we provide more tokens in the prompt, we just won't be able to pull positional embeddings for them. It's an architectural aspect ("hyperparameter" in AI parlance) of the model that is set at design time and cannot be changed by training. When people talk about the "context window" of an LLM, they mean this number.

We have the token 6307 at place 0, 47701 at 1, 318 at 2, and 1049 at 3. For each of these tokens and positions, we have two vectors: one from WTE and another one from WPE. We need to add them together. The four resulting vectors will be the inputs for the next part of the algorithm: the feed-forward neural network with the attention mechanism.

For the SQL part, we will use pgvector, a PostgreSQL extension.

A little disclaimer: normally, I write code for my New Year posts in vanilla SQL, sometimes with pure SQL functions as helpers. It would be perfectly possible to do it for this post as well by defining vector operations on arrays, at the cost of some performance decrease (it was done in version 1 and worked, albeit slowly). With the advent of the AI and growing importance of vector databases, pgvector or its equivalent will definitely make it into the core of PostgreSQL within two or three releases. I just decided to ride the wave of the future.

Here's how we do that in SQL:

WITH    embeddings AS
        (
        SELECT  place, values
        FROM    UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        )
SELECT  place, (values::REAL[])[0:5]
FROM    embeddings

(To keep the output short, this query only shows the first 5 dimensions for each vector)

Attention

The part that really makes the Transformer architecture tick is the self-attention mechanism. It was first described in the 2017 paper "Attention is all you need" by Vasmani et al., probably the most famous AI paper, whose name has since become a snowclone (a cliché for naming other papers).

So far, we have several vectors that, hopefully, encode some syntactic and semantic properties of the words in our prompt. We need these properties to somehow transfer to the last vector. A little spoiler alert: at the end of the day, it will be the last vector that will store the embedding for the continuation word.

In a phrase like "I looked at the violet and saw that it was not the usual …", the ellipsis has to be something you see (and this notion has to jump from "saw"), something that's a property of a violet (jumping from "violet" to "it" and then to the ellipsis), and something that is "unusual" (jumping from "not" and "usual" and flipping the sign in the dimensions responsible for the usualness). The analogy in the real world would be a person reading a book in a foreign language that they kind of have a basic command of, but don't quite know very well. They would need to consciously trace their way from one word to another, and if they don't pay attention to the crucial part of the phrase, their understanding would be wrong.

To enable this transfer of meaning from one token to another, we need to allow the vectors of all the tokens to influence each other. If we want to populate the word "it" with some concrete semantics, how much of the semantics should come from the previous vectors in the prompt, and how much should remain from the word "it" itself?

To solve this problem, the model uses 12 sets of matrixes called Q (query), K (key) and V (value). Each of them has 64 columns. They are obtained from the vector embeddings through a 768×2304 linear transformation c_attn, whose weights and biases are stored in the tables c_attn_w and c_attn_b.

The result of c_attn is a matrix with n_token rows and 2304 columns (3×12×64). It consists of 12 Q matrixes, 12 K matrixes and 12 V matrixes stacked horizontally, in this order.

Each set of Q, K and V is called a "head". They are used to perform the step known as "multi-headed causal self-attention", by calculating the attention function.

Here's the formula for the attention function:

,

where softmax is the weight normalization function. It's defined like this:

is a constant matrix called a "causal mask". It is defined like this:

Softmax turns negative infinities into zeros.

Why do we need masking?

The prompt in our previous examples had 4 tokens, and the first thing the model did was calculate the 4 embeddings for these 4 tokens. As the model progresses, these vectors will undergo a lot of calculations, but for the most part, they will be independent and parallel. Changes in one vector will not affect the other vectors, as if they had not existed. The self-attention block is the only place in the whole model where the vectors affect each other.

Once the model is done with the math, the candidates for the next token will be decided solely from the last embedding. All the information flow should be directed towards this last vector and not from it. The transient values of the last embedding should not affect the transient values of the previous embeddings during the forward pass of the model.

That's why we "mask" the latter embeddings so that they don't influence the earlier embeddings through this particular channel. Hence the word "causal" in "multi-headed causal self-attention".

Why are the matrixes called "query", "key" and "value"?

To be honest, I'm not sure it's even a good analogy. But I'll still do my take on the intuition behind it.

In machine learning, generally, calculations should not involve variable-length loops or statement branching. Everything should be done through the composition of simple analytic functions (additions, multiplications, powers, logarithms and trig). It allows backpropagation, which relies on technologies like automatic differentiation, to work efficiently.

The mathematical model of the key-value store is the expression

, but it's not a smooth, differentiable function and it will not work well with backpropagation. To make it work, we would need to turn it into a smooth function that would be close to when is close to , and close to otherwise.

The Gaussian distribution ("bell curve"), scaled to , with the expectation of and a small enough standard deviation would do perfectly for this purpose:

, where is an arbitrary parameter, defining how sharp the bell curve is.

In a vector space with many enough dimensions, if we take a fixed vector and several vectors that randomly and uniformly deviate from on every dimension, their dot products will naturally form the bell curve. So, in the vector space, the concept of a "differentiable key-value store" can be modeled by the expression , which is what we are using in our attention function.

Again, this analogy is far-fetched. It's best not to pay too much attention (no pun intended) to these concepts of attention, meaning flow, hash tables and so on. Just think of them as an inspiration for a math trick that has been put to the test and proved to work really well.

Let's illustrate this step:

WITH    embeddings AS
        (
        SELECT  place, values
        FROM    UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        ),
        c_attn_w AS
        (
        SELECT  *
        FROM    c_attn_w
        WHERE   block = 0
        ),
        c_attn_b AS
        (
        SELECT  *
        FROM    c_attn_b
        WHERE   block = 0
        ),
        ln_1_g AS
        (
        SELECT  *
        FROM    ln_1_g
        WHERE   block = 0
        ),
        ln_1_b AS
        (
        SELECT  *
        FROM    ln_1_b
        WHERE   block = 0
        ),
        mha_norm AS
        (
        SELECT  place, mm.values + c_attn_b.values AS values
        FROM    (
                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                FROM    (
                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                        FROM    (
                                SELECT  place, norm.values
                                FROM    embeddings
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  AVG(value) AS mean,
                                                VAR_POP(value) AS variance
                                        FROM    UNNEST(values::REAL[]) value
                                        ) agg
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
                                        ) norm
                                ) agg
                        CROSS JOIN
                                ln_1_b
                        CROSS JOIN
                                ln_1_g
                        ) layer_norm
                CROSS JOIN
                        c_attn_w
                GROUP BY
                        place
                ) mm
        CROSS JOIN
                c_attn_b
        ),
        head AS
        (
        SELECT  place,
                (values::REAL[])[1:64]::VECTOR(64) AS q,
                (values::REAL[])[1 + 768:64 + 768]::VECTOR(64) AS k,
                (values::REAL[])[1 + 1536:64 + 1536]::VECTOR(64) AS v
        FROM    mha_norm
        ),
        sm_input AS
        (
        SELECT  h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
        FROM    head h1
        CROSS JOIN
                head h2
        ),
        sm_diff AS
        (
        SELECT  x, y, value - MAX(value) OVER (PARTITION BY x) AS diff
        FROM    sm_input
        ),
        sm_exp AS
        (
        SELECT  x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
        FROM    sm_diff
        ),
        softmax AS
        (
        SELECT  x, y AS place, e / SUM(e) OVER (PARTITION BY x) AS value
        FROM    sm_exp
        ),
        attention AS
        (
        SELECT  place, (ARRAY_AGG(value ORDER BY ordinality))[:3] AS values
        FROM    (
                SELECT  x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * head.v) AS values
                FROM    softmax
                JOIN    head
                USING   (place)
                GROUP BY
                        x
                ) q
        CROSS JOIN LATERAL
                UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
        GROUP BY
                place
        )
SELECT  place,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((q::REAL[])[:3]) AS n) AS q,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((k::REAL[])[:3]) AS n) AS k,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((v::REAL[])[:3]) AS n) AS v,
        matrix,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:3]) AS n) AS attention
FROM    head
JOIN    attention
USING   (place)
JOIN    (
        SELECT  x AS place, STRING_AGG(CASE WHEN value > 0 THEN TO_CHAR(value, '0.00') ELSE '    0' END, ' ' ORDER BY place) AS matrix
        FROM    softmax
        GROUP BY
                x
        ) softmax_grouped
USING   (place)

Here is what we did:

1. Before calculating the attention function, we normalized the vectors by applying the linear transformation . The matrix and the vector are called "scale" and "shift", accordingly. They are learned parameters of the model, which are stored in the tables ln_1_g and ln_1_b
2. We are only showing the first head of the first layer of the algorithm. After we multiplied the vectors by the learned coefficients from c_attn_w and c_attn_b ("weight" and "bias"), we sliced the resulting 2304-vectors, taking 64-vectors starting at the positions 0, 768 and 1536. They correspond to the vectors Q, K and V for the first head.
3. EXP in PostgreSQL fails on really small numbers, that's why we shortcut to zero if the argument to EXP is less than -745.13.
4. We are only showing the first three elements for each vector. The attention matrix we show in full.

As we can see, the first value vector got copied to the output as is (as it will do in every other layer of the algorithm). It means that once the model has been trained, the output embedding for the first token will be only defined by the value of the first token. In general, during the recursive inference phase, where tokens only get added to the prompt, only the last embedding in the output will ever change compared to the previous iteration. This is what the causal mask does.

Looking a bit forward: the attention block is the only place in the entire algorithm where tokens can influence each other during the forward pass. Since we have disabled the ability of later tokens to influence the previous ones in this step, all the calculations done on the previous tokens can be reused between the forward passes of the model.

Remember, the model operates by appending tokens to the prompt. If our original (tokenized) prompt is "Post greSQL Ġis Ġgreat" and the next one will be (for instance) "Post greSQL Ġis Ġgreat Ġfor", all the results of the calculations made on the first four tokens can be reused for the new prompt; they will never change, regardless of what is appended to them.

Jay Mody's illustrative article doesn't make use of this fact (and neither do we, for the sake of simplicity), but the original GPT2 implementation does.

Once all the heads are done, we will end up with 12 matrixes, each 64 columns wide and n_tokens rows tall. To map it back to the dimension of embedding vectors (768), we just need to stack these matrixes horizontally.

The final step of multi-headed attention involves projecting the values through a learned linear transformation of the same dimension. Its weights and biases are stored in the tables c_proj_w and c_proj_b.

Here's what the code for a complete multi-headed attention step in the first layer looks like:

WITH    embeddings AS
        (
        SELECT  place, values
        FROM    UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        ),
        c_proj_w AS
        (
        SELECT  *
        FROM    c_proj_w
        WHERE   block = 0
        ),
        c_proj_b AS
        (
        SELECT  *
        FROM    c_proj_b
        WHERE   block = 0
        ),
        mlp_c_fc_w AS
        (
        SELECT  *
        FROM    mlp_c_fc_w
        WHERE   block = 0
        ),
        mlp_c_fc_b AS
        (
        SELECT  *
        FROM    mlp_c_fc_b
        WHERE   block = 0
        ),
        mlp_c_proj_w AS
        (
        SELECT  *
        FROM    mlp_c_proj_w
        WHERE   block = 0
        ),
        mlp_c_proj_b AS
        (
        SELECT  *
        FROM    mlp_c_proj_b
        WHERE   block = 0
        ),
        c_attn_w AS
        (
        SELECT  *
        FROM    c_attn_w
        WHERE   block = 0
        ),
        c_attn_b AS
        (
        SELECT  *
        FROM    c_attn_b
        WHERE   block = 0
        ),
        ln_1_g AS
        (
        SELECT  *
        FROM    ln_1_g
        WHERE   block = 0
        ),
        ln_1_b AS
        (
        SELECT  *
        FROM    ln_1_b
        WHERE   block = 0
        ),
        mha_norm AS
        (
        SELECT  place, mm.values + c_attn_b.values AS values
        FROM    (
                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                FROM    (
                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                        FROM    (
                                SELECT  place, norm.values
                                FROM    embeddings
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  AVG(value) AS mean,
                                                VAR_POP(value) AS variance
                                        FROM    UNNEST(values::REAL[]) value
                                        ) agg
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
                                        ) norm
                                ) agg
                        CROSS JOIN
                                ln_1_b
                        CROSS JOIN
                                ln_1_g
                        ) layer_norm
                CROSS JOIN
                        c_attn_w
                GROUP BY
                        place
                ) mm
        CROSS JOIN
                c_attn_b
        ),
        heads AS
        (
        SELECT  place, head,
                (values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
                (values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS k,
                (values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
        FROM    mha_norm
        CROSS JOIN
                GENERATE_SERIES(0, 11) head
        ),
        sm_input AS
        (
        SELECT  head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
        FROM    heads h1
        JOIN    heads h2
        USING   (head)
        ),
        sm_diff AS
        (
        SELECT  head, x, y, value - MAX(value) OVER (PARTITION BY head, x) AS diff
        FROM    sm_input
        ),
        sm_exp AS
        (
        SELECT  head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
        FROM    sm_diff
        ),
        softmax AS
        (
        SELECT  head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS value
        FROM    sm_exp
        ),
        attention AS
        (
        SELECT  place, ARRAY_AGG(value ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
        FROM    (
                SELECT  head, x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * heads.v) AS values
                FROM    softmax
                JOIN    heads
                USING   (head, place)
                GROUP BY
                        head, x
                ) q
        CROSS JOIN LATERAL
                UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
        GROUP BY
                place
        ),
        mha AS
        (
        SELECT  place, w.values + c_proj_b.values AS values
        FROM    (
                SELECT  attention.place, ARRAY_AGG(INNER_PRODUCT(attention.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
                FROM    attention
                CROSS JOIN
                        c_proj_w
                GROUP BY
                        attention.place
                ) w
        CROSS JOIN
                c_proj_b
        )
SELECT  place,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:10]) AS n) AS q
FROM    mha

Before the results of multi-headed attention are passed to the next step, the original inputs are added to them. This trick was described in the original transformer paper. It's supposed to help with vanishing and exploding gradients.

It's a common problem during training: sometimes the gradients of the parameters turn out too big or too small. Changing them on the training iteration either has very little effect on the loss function (and so the model converges very slowly), or, on the opposite, has such a big effect that even a small change throws the loss function too far away from its local minimum, negating the training efforts.

Feedforward

This is what the deep neural networks do. The larger part of the model parameters is actually used at this step.

This step is a multi-layer perceptron with three layers (768, 3072, 768), using the Gaussian Error Linear Unit (GELU) as an activation function:

This function has been observed to yield good results in deep neural networks. It can be analytically approximated like this:

The learned linear transformation parameters for layer connections are called c_fc (768 → 3072) and c_proj (3072 → 768). The values for the first layer are first normalized using the coefficients in the learned parameter ln_2. After the feedforward step is completed, its input is again added to the output. This, too, is a part of the original transformer design.

The whole feedforward step looks like this:

And here's how we do this in SQL:

WITH    embeddings AS
        (
        SELECT  place, values
        FROM    UNNEST(ARRAY[6307, 47701, 318, 1049]) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        ),
        c_proj_w AS
        (
        SELECT  *
        FROM    c_proj_w
        WHERE   block = 0
        ),
        c_proj_b AS
        (
        SELECT  *
        FROM    c_proj_b
        WHERE   block = 0
        ),
        mlp_c_fc_w AS
        (
        SELECT  *
        FROM    mlp_c_fc_w
        WHERE   block = 0
        ),
        mlp_c_fc_b AS
        (
        SELECT  *
        FROM    mlp_c_fc_b
        WHERE   block = 0
        ),
        mlp_c_proj_w AS
        (
        SELECT  *
        FROM    mlp_c_proj_w
        WHERE   block = 0
        ),
        mlp_c_proj_b AS
        (
        SELECT  *
        FROM    mlp_c_proj_b
        WHERE   block = 0
        ),
        c_attn_w AS
        (
        SELECT  *
        FROM    c_attn_w
        WHERE   block = 0
        ),
        c_attn_b AS
        (
        SELECT  *
        FROM    c_attn_b
        WHERE   block = 0
        ),
        ln_1_g AS
        (
        SELECT  *
        FROM    ln_1_g
        WHERE   block = 0
        ),
        ln_1_b AS
        (
        SELECT  *
        FROM    ln_1_b
        WHERE   block = 0
        ),
        ln_2_b AS
        (
        SELECT  *
        FROM    ln_2_b
        WHERE   block = 0
        ),
        ln_2_g AS
        (
        SELECT  *
        FROM    ln_2_g
        WHERE   block = 0
        ),
        mha_norm AS
        (
        SELECT  place, mm.values + c_attn_b.values AS values
        FROM    (
                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                FROM    (
                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                        FROM    (
                                SELECT  place, norm.values
                                FROM    embeddings
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  AVG(value) AS mean,
                                                VAR_POP(value) AS variance
                                        FROM    UNNEST(values::REAL[]) value
                                        ) agg
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
                                        ) norm
                                ) agg
                        CROSS JOIN
                                ln_1_b
                        CROSS JOIN
                                ln_1_g
                        ) layer_norm
                CROSS JOIN
                        c_attn_w
                GROUP BY
                        place
                ) mm
        CROSS JOIN
                c_attn_b
        ),
        heads AS
        (
        SELECT  place, head,
                (values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
                (values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS k,
                (values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
        FROM    mha_norm
        CROSS JOIN
                GENERATE_SERIES(0, 11) head
        ),
        sm_input AS
        (
        SELECT  head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
        FROM    heads h1
        JOIN    heads h2
        USING   (head)
        ),
        sm_diff AS
        (
        SELECT  head, x, y, value - MAX(value) OVER (PARTITION BY head, x) AS diff
        FROM    sm_input
        ),
        sm_exp AS
        (
        SELECT  head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
        FROM    sm_diff
        ),
        softmax AS
        (
        SELECT  head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS value
        FROM    sm_exp
        ),
        attention AS
        (
        SELECT  place, ARRAY_AGG(value ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
        FROM    (
                SELECT  head, x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * heads.v) AS values
                FROM    softmax
                JOIN    heads
                USING   (head, place)
                GROUP BY
                        head, x
                ) q
        CROSS JOIN LATERAL
                UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
        GROUP BY
                place
        ),
        mha AS
        (
        SELECT  place, w.values + c_proj_b.values + embeddings.values AS values
        FROM    (
                SELECT  attention.place, ARRAY_AGG(INNER_PRODUCT(attention.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
                FROM    attention
                CROSS JOIN
                        c_proj_w
                GROUP BY
                        attention.place
                ) w
        CROSS JOIN
                c_proj_b
        JOIN    embeddings
        USING   (place)
        ),
        ffn_norm AS
        (
        SELECT  place, agg.values * ln_2_g.values + ln_2_b.values AS values
        FROM    (
                SELECT  place, norm.values
                FROM    mha
                CROSS JOIN LATERAL
                        (
                        SELECT  AVG(value) AS mean,
                                VAR_POP(value) AS variance
                        FROM    UNNEST(values::REAL[]) value
                        ) agg
                CROSS JOIN LATERAL
                        (
                        SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
                        ) norm
                ) agg
        CROSS JOIN
                ln_2_b
        CROSS JOIN
                ln_2_g
        ),
        ffn_a AS
        (
        SELECT  gelu.place, gelu.values
        FROM    (
                SELECT  place, w.values + mlp_c_fc_b.values AS values
                FROM    (
                        SELECT  ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
                        FROM    ffn_norm
                        CROSS JOIN
                                mlp_c_fc_w
                        GROUP BY
                                ffn_norm.place
                        ) w
                CROSS JOIN
                        mlp_c_fc_b
                ) v
        CROSS JOIN LATERAL
                (
                SELECT  place, ARRAY_AGG(0.5 * value * (1 + TANH(0.797884560802 * (value + 0.044715 * value*value*value))) ORDER BY ordinality)::VECTOR(3072) AS values
                FROM    UNNEST(values::REAL[]) WITH ORDINALITY n (value, ordinality)
                GROUP BY
                        place
                ) gelu
        ),
        ffn AS
        (
        SELECT  place, w.values + mlp_c_proj_b.values + mha.values AS values
        FROM    (
                SELECT  ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
                FROM    ffn_a
                CROSS JOIN
                        mlp_c_proj_w
                GROUP BY
                        ffn_a.place
                ) w
        CROSS JOIN
                mlp_c_proj_b
        JOIN    mha
        USING   (place)
        )
SELECT  place,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:10]) AS n) AS q
FROM    ffn

This output is what comes out of the first block of GPT2.

Blocks

What we saw in the previous steps is repeated in layers (called "blocks"). The blocks are set up in a pipeline so that the output of a previous block goes straight to the next one. Each block has its own set of learned parameters.

In SQL, we would need to connect the blocks using a recursive CTE.

Once the final block produces the values, we need to normalize it using the learned parameter ln_f.

Here's what the model ultimately looks like:

And here's how it looks in SQL:

WITH    RECURSIVE
        initial AS
        (
        SELECT  ARRAY[6307, 47701, 318, 1049] AS input
        ),
        hparams AS
        (
        SELECT  12 AS n_block
        ),
        embeddings AS
        (
        SELECT  place, values
        FROM    initial
        CROSS JOIN
                hparams
        CROSS JOIN LATERAL
                UNNEST(input) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        ),
        transform AS
        (
        SELECT  0 AS block, place, values
        FROM    embeddings
        UNION ALL
        (
        WITH    previous AS
                (
                SELECT  *
                FROM    transform
                )
        SELECT  block + 1 AS block, transformed_layer.*
        FROM    hparams
        CROSS JOIN LATERAL
                (
                SELECT  block
                FROM    previous
                WHERE   block < 12
                LIMIT   1
                ) q
        CROSS JOIN LATERAL
                (
                WITH    ln_2_b AS
                        (
                        SELECT  *
                        FROM    ln_2_b
                        WHERE   block = q.block
                        ),
                        ln_2_g AS
                        (
                        SELECT  *
                        FROM    ln_2_g
                        WHERE   block = q.block
                        ),
                        c_proj_w AS
                        (
                        SELECT  *
                        FROM    c_proj_w
                        WHERE   block = q.block
                        ),
                        c_proj_b AS
                        (
                        SELECT  *
                        FROM    c_proj_b
                        WHERE   block = q.block
                        ),
                        mlp_c_fc_w AS
                        (
                        SELECT  *
                        FROM    mlp_c_fc_w
                        WHERE   block = q.block
                        ),
                        mlp_c_fc_b AS
                        (
                        SELECT  *
                        FROM    mlp_c_fc_b
                        WHERE   block = q.block
                        ),
                        mlp_c_proj_w AS
                        (
                        SELECT  *
                        FROM    mlp_c_proj_w
                        WHERE   block = q.block
                        ),
                        mlp_c_proj_b AS
                        (
                        SELECT  *
                        FROM    mlp_c_proj_b
                        WHERE   block = q.block
                        ),
                        c_attn_w AS
                        (
                        SELECT  *
                        FROM    c_attn_w
                        WHERE   block = q.block
                        ),
                        c_attn_b AS
                        (
                        SELECT  *
                        FROM    c_attn_b
                        WHERE   block = q.block
                        ),
                        ln_1_g AS
                        (
                        SELECT  *
                        FROM    ln_1_g
                        WHERE   block = q.block
                        ),
                        ln_1_b AS
                        (
                        SELECT  *
                        FROM    ln_1_b
                        WHERE   block = q.block
                        ),
                        mha_norm AS
                        (
                        SELECT  place, mm.values + c_attn_b.values AS values
                        FROM    (
                                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                                FROM    (
                                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                                        FROM    (
                                                SELECT  place, norm.values
                                                FROM    previous
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  AVG(value) AS mean,
                                                                VAR_POP(value) AS variance
                                                        FROM    UNNEST(values::REAL[]) value
                                                        ) agg
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
                                                        ) norm
                                                ) agg
                                        CROSS JOIN
                                                ln_1_b
                                        CROSS JOIN
                                                ln_1_g
                                        ) layer_norm
                                CROSS JOIN
                                        c_attn_w
                                GROUP BY
                                        place
                                ) mm
                        CROSS JOIN
                                c_attn_b
                        ),
                        heads AS
                        (
                        SELECT  place, head,
                                (values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
                                (values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS k,
                                (values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
                        FROM    mha_norm
                        CROSS JOIN
                                GENERATE_SERIES(0, 11) head
                        ),
                        sm_input AS
                        (
                        SELECT  head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
                        FROM    heads h1
                        JOIN    heads h2
                        USING   (head)
                        ),
                        sm_diff AS
                        (
                        SELECT  head, x, y, value - MAX(value) OVER (PARTITION BY head, x) AS diff
                        FROM    sm_input
                        ),
                        sm_exp AS
                        (
                        SELECT  head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
                        FROM    sm_diff
                        ),
                        softmax AS
                        (
                        SELECT  head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS value
                        FROM    sm_exp
                        ),
                        attention AS
                        (
                        SELECT  place, ARRAY_AGG(value ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
                        FROM    (
                                SELECT  head, x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * heads.v) AS values
                                FROM    softmax
                                JOIN    heads
                                USING   (head, place)
                                GROUP BY
                                        head, x
                                ) q
                        CROSS JOIN LATERAL
                                UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
                        GROUP BY
                                place
                        ),
                        mha AS
                        (
                        SELECT  place, w.values + c_proj_b.values + previous.values AS values
                        FROM    (
                                SELECT  attention.place, ARRAY_AGG(INNER_PRODUCT(attention.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
                                FROM    attention
                                CROSS JOIN
                                        c_proj_w
                                GROUP BY
                                        attention.place
                                ) w
                        CROSS JOIN
                                c_proj_b
                        JOIN    previous
                        USING   (place)
                        ),
                        ffn_norm AS
                        (
                        SELECT  place, agg.values * ln_2_g.values + ln_2_b.values AS values
                        FROM    (
                                SELECT  place, norm.values
                                FROM    mha
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  AVG(value) AS mean,
                                                VAR_POP(value) AS variance
                                        FROM    UNNEST(values::REAL[]) value
                                        ) agg
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
                                        ) norm
                                ) agg
                        CROSS JOIN
                                ln_2_b
                        CROSS JOIN
                                ln_2_g
                        ),
                        ffn_a AS
                        (
                        SELECT  gelu.place, gelu.values
                        FROM    (
                                SELECT  place, w.values + mlp_c_fc_b.values AS values
                                FROM    (
                                        SELECT  ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
                                        FROM    ffn_norm
                                        CROSS JOIN
                                                mlp_c_fc_w
                                        GROUP BY
                                                ffn_norm.place
                                        ) w
                                CROSS JOIN
                                        mlp_c_fc_b
                                ) v
                        CROSS JOIN LATERAL
                                (
                                SELECT  place, ARRAY_AGG(0.5 * value * (1 + TANH(0.797884560802 * (value + 0.044715 * value*value*value))) ORDER BY ordinality)::VECTOR(3072) AS values
                                FROM    UNNEST(values::REAL[]) WITH ORDINALITY n (value, ordinality)
                                GROUP BY
                                        place
                                ) gelu
                        ),
                        ffn AS
                        (
                        SELECT  place, w.values + mlp_c_proj_b.values + mha.values AS values
                        FROM    (
                                SELECT  ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
                                FROM    ffn_a
                                CROSS JOIN
                                        mlp_c_proj_w
                                GROUP BY
                                        ffn_a.place
                                ) w
                        CROSS JOIN
                                mlp_c_proj_b
                        JOIN    mha
                        USING   (place)
                        )
                SELECT  *
                FROM    ffn
                ) transformed_layer
        )
        ),
        block_output AS
        (
        SELECT  *
        FROM    hparams
        JOIN    transform
        ON      transform.block = n_block
        ),
        ln_f AS
        (
        SELECT  place, norm.values * ln_f_g.values + ln_f_b.values AS values
        FROM    block_output
        CROSS JOIN LATERAL
                (
                SELECT  AVG(value) AS mean,
                        VAR_POP(value) AS variance
                FROM    UNNEST(values::REAL[]) AS n(value)
                ) agg
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n (value, ordinality)
                ) norm
        CROSS JOIN
                ln_f_b
        CROSS JOIN
                ln_f_g
        )
SELECT  place,
        (SELECT STRING_AGG(TO_CHAR(n, 'S0.000'), ' ') || ' …' FROM UNNEST((values::REAL[])[:10]) AS n) AS q
FROM    ln_f

This is the output of the model.

The fourth vector is the actual embedding of the next token predicted by the model. We just need to map it back to the tokens.

Tokens

We have an embedding (a 768-vector) which, according to the model, captures the semantics and the grammar of the most likely continuation of the prompt. Now we need to map it back to the token.

One of the first steps the model makes is mapping the tokens to their embeddings. It is done through the 50257×768 matrix wte. We will need to use the same matrix to map the embedding back to the token.

The problem is that the exact reverse mapping is not possible: the embedding will not (likely) be equal to any of the rows in the matrix. So we will need to find the "closest" token to the embedding.

Since the dimensions of embeddings capture (as we hope) some semantic and grammatical aspects of the token, we need them to match as closely as possible. One way to consolidate the closeness of each dimension would be to just calculate the dot product of the two embeddings. The higher the dot product, the closer the token is to the prediction.

To do this, we will multiply the embedding by the matrix wte. The result will be a single-column matrix, 50257 rows tall. Each value in this result will be the dot product of the predicted embedding and the token embedding. The higher this number, the more likely it is for the token to continue the prompt.

To pick the next token, we will need to convert the similarities to probabilities. To do this, we will use our good friend softmax (the same function that we used to normalize attention weights).

Why use softmax for probabilities?

Softmax has the nice property of satisfying Luce's choice axiom. It means that the relative probabilities of two options don't depend on the presence or probability of other options. If A is twice as probable as B, then the presence or absence of other options will not change this ratio (although it of course can change the absolute values).

The vector of dot products ("logit" in AI parlance) contains arbitrary scores that don't have an intrinsic scale. If A has a larger score than B, we know that it's more likely, but that's about it. We can tweak the inputs to softmax as we please, as long as they keep their order (i.e. larger scores stay larger).

One common way to do that is to normalize the scores by subtracting the greatest value from the set from them (so that the biggest score becomes 0 and the rest become negative numbers). Then we take some fixed number (let's say five or ten) top scores. Finally, we multiply each score by a constant before feeding it to softmax.

The number of top scores that we take is usually called and the multiplication constant (or, rather, its reverse) is called "temperature" (). The higher the temperature, the more smoothed out the probabilities, and the bigger the chance that the next picked token will not be just the first one.

The formula for tokens' probabilities is , where is the set of scores.

Why is it called "temperature"?

The softmax function has another name: Boltzmann distribution. It's extensively used in physics. Among other things, it serves as a base for the barometric formula, which tells how density or air varies with altitude.

Intuitively, hot air rises. It spreads further away from the Earth. When air is hot, it's more likely for an air molecule to bounce off its neighbors and jump at an otherwise impossible height. Compared to colder temperatures, air density increases at higher altitudes and drops at sea level.

See how air behaves at different temperatures:

Courtesy of Dominic Ford, Bouncing Balls and the Boltzmann Distribution

By analogy, a large "temperature" increases the probability of second-choice tokens being selected (at the expense of the first-choice tokens, of course). The inference becomes less predictable and more "creative".

Let's put this all into SQL. The prompt was "PostgreSQL is great". Here are the top 5 tokens that, according to the model, are most likely to continue this phrase, and their probabilities at different temperatures:

WITH    RECURSIVE
        initial AS
        (
        SELECT  ARRAY[6307, 47701, 318, 1049] AS input
        ),
        hparams AS
        (
        SELECT  12 AS n_block,
                5 AS top_n,
                ARRAY_LENGTH(input, 1) AS n_seq
        FROM    initial
        ),
        embeddings AS
        (
        SELECT  place, values
        FROM    initial
        CROSS JOIN
                hparams
        CROSS JOIN LATERAL
                UNNEST(input) WITH ORDINALITY AS tokens (token, ordinality)
        CROSS JOIN LATERAL
                (
                SELECT  ordinality - 1 AS place
                ) o
        CROSS JOIN LATERAL
                (
                SELECT  wte.values + wpe.values AS values
                FROM    wte
                CROSS JOIN
                        wpe
                WHERE   wte.token = tokens.token
                        AND wpe.place = o.place
                ) embedding
        ),
        transform AS
        (
        SELECT  0 AS block, place, values
        FROM    embeddings
        UNION ALL
        (
        WITH    previous AS
                (
                SELECT  *
                FROM    transform
                )
        SELECT  block + 1 AS block, transformed_layer.*
        FROM    hparams
        CROSS JOIN LATERAL
                (
                SELECT  block
                FROM    previous
                WHERE   block < 12
                LIMIT   1
                ) q
        CROSS JOIN LATERAL
                (
                WITH    ln_2_b AS
                        (
                        SELECT  *
                        FROM    ln_2_b
                        WHERE   block = q.block
                        ),
                        ln_2_g AS
                        (
                        SELECT  *
                        FROM    ln_2_g
                        WHERE   block = q.block
                        ),
                        c_proj_w AS
                        (
                        SELECT  *
                        FROM    c_proj_w
                        WHERE   block = q.block
                        ),
                        c_proj_b AS
                        (
                        SELECT  *
                        FROM    c_proj_b
                        WHERE   block = q.block
                        ),
                        mlp_c_fc_w AS
                        (
                        SELECT  *
                        FROM    mlp_c_fc_w
                        WHERE   block = q.block
                        ),
                        mlp_c_fc_b AS
                        (
                        SELECT  *
                        FROM    mlp_c_fc_b
                        WHERE   block = q.block
                        ),
                        mlp_c_proj_w AS
                        (
                        SELECT  *
                        FROM    mlp_c_proj_w
                        WHERE   block = q.block
                        ),
                        mlp_c_proj_b AS
                        (
                        SELECT  *
                        FROM    mlp_c_proj_b
                        WHERE   block = q.block
                        ),
                        c_attn_w AS
                        (
                        SELECT  *
                        FROM    c_attn_w
                        WHERE   block = q.block
                        ),
                        c_attn_b AS
                        (
                        SELECT  *
                        FROM    c_attn_b
                        WHERE   block = q.block
                        ),
                        ln_1_g AS
                        (
                        SELECT  *
                        FROM    ln_1_g
                        WHERE   block = q.block
                        ),
                        ln_1_b AS
                        (
                        SELECT  *
                        FROM    ln_1_b
                        WHERE   block = q.block
                        ),
                        mha_norm AS
                        (
                        SELECT  place, mm.values + c_attn_b.values AS values
                        FROM    (
                                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                                FROM    (
                                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                                        FROM    (
                                                SELECT  place, norm.values
                                                FROM    previous
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  AVG(value) AS mean,
                                                                VAR_POP(value) AS variance
                                                        FROM    UNNEST(values::REAL[]) value
                                                        ) agg
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
                                                        ) norm
                                                ) agg
                                        CROSS JOIN
                                                ln_1_b
                                        CROSS JOIN
                                                ln_1_g
                                        ) layer_norm
                                CROSS JOIN
                                        c_attn_w
                                GROUP BY
                                        place
                                ) mm
                        CROSS JOIN
                                c_attn_b
                        ),
                        heads AS
                        (
                        SELECT  place, head,
                                (values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
                                (values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS k,
                                (values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
                        FROM    mha_norm
                        CROSS JOIN
                                GENERATE_SERIES(0, 11) head
                        ),
                        sm_input AS
                        (
                        SELECT  head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
                        FROM    heads h1
                        JOIN    heads h2
                        USING   (head)
                        ),
                        sm_diff AS
                        (
                        SELECT  head, x, y, value - MAX(value) OVER (PARTITION BY head, x) AS diff
                        FROM    sm_input
                        ),
                        sm_exp AS
                        (
                        SELECT  head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
                        FROM    sm_diff
                        ),
                        softmax AS
                        (
                        SELECT  head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS value
                        FROM    sm_exp
                        ),
                        attention AS
                        (
                        SELECT  place, ARRAY_AGG(value ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
                        FROM    (
                                SELECT  head, x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * heads.v) AS values
                                FROM    softmax
                                JOIN    heads
                                USING   (head, place)
                                GROUP BY
                                        head, x
                                ) q
                        CROSS JOIN LATERAL
                                UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
                        GROUP BY
                                place
                        ),
                        mha AS
                        (
                        SELECT  place, w.values + c_proj_b.values + previous.values AS values
                        FROM    (
                                SELECT  attention.place, ARRAY_AGG(INNER_PRODUCT(attention.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
                                FROM    attention
                                CROSS JOIN
                                        c_proj_w
                                GROUP BY
                                        attention.place
                                ) w
                        CROSS JOIN
                                c_proj_b
                        JOIN    previous
                        USING   (place)
                        ),
                        ffn_norm AS
                        (
                        SELECT  place, agg.values * ln_2_g.values + ln_2_b.values AS values
                        FROM    (
                                SELECT  place, norm.values
                                FROM    mha
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  AVG(value) AS mean,
                                                VAR_POP(value) AS variance
                                        FROM    UNNEST(values::REAL[]) value
                                        ) agg
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
                                        ) norm
                                ) agg
                        CROSS JOIN
                                ln_2_b
                        CROSS JOIN
                                ln_2_g
                        ),
                        ffn_a AS
                        (
                        SELECT  gelu.place, gelu.values
                        FROM    (
                                SELECT  place, w.values + mlp_c_fc_b.values AS values
                                FROM    (
                                        SELECT  ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
                                        FROM    ffn_norm
                                        CROSS JOIN
                                                mlp_c_fc_w
                                        GROUP BY
                                                ffn_norm.place
                                        ) w
                                CROSS JOIN
                                        mlp_c_fc_b
                                ) v
                        CROSS JOIN LATERAL
                                (
                                SELECT  place, ARRAY_AGG(0.5 * value * (1 + TANH(0.797884560802 * (value + 0.044715 * value*value*value))) ORDER BY ordinality)::VECTOR(3072) AS values
                                FROM    UNNEST(values::REAL[]) WITH ORDINALITY n (value, ordinality)
                                GROUP BY
                                        place
                                ) gelu
                        ),
                        ffn AS
                        (
                        SELECT  place, w.values + mlp_c_proj_b.values + mha.values AS values
                        FROM    (
                                SELECT  ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
                                FROM    ffn_a
                                CROSS JOIN
                                        mlp_c_proj_w
                                GROUP BY
                                        ffn_a.place
                                ) w
                        CROSS JOIN
                                mlp_c_proj_b
                        JOIN    mha
                        USING   (place)
                        )
                SELECT  *
                FROM    ffn
                ) transformed_layer
        )
        ),
        block_output AS
        (
        SELECT  *
        FROM    hparams
        JOIN    transform
        ON      transform.block = n_block
        ),
        ln_f AS
        (
        SELECT  place, norm.values * ln_f_g.values + ln_f_b.values AS values
        FROM    block_output
        CROSS JOIN LATERAL
                (
                SELECT  AVG(value) AS mean,
                        VAR_POP(value) AS variance
                FROM    UNNEST(values::REAL[]) AS n(value)
                ) agg
        CROSS JOIN LATERAL
                (
                SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n (value, ordinality)
                ) norm
        CROSS JOIN
                ln_f_b
        CROSS JOIN
                ln_f_g
        ),
        logits AS
        (
        SELECT  logits.*
        FROM    hparams
        CROSS JOIN LATERAL
                (
                SELECT  token, INNER_PRODUCT(ln_f.values, wte.values) AS value
                FROM    ln_f
                CROSS JOIN
                        wte
                WHERE   ln_f.place = n_seq - 1
                ORDER BY
                        value DESC
                LIMIT   (top_n)
                ) logits
        ),
        temperatures (temperature) AS
        (
        VALUES
        (0.5),
        (1),
        (2)
        ),
        tokens AS
        (
        SELECT  token, value, softmax, temperature
        FROM    temperatures
        CROSS JOIN LATERAL
                (
                SELECT  *, (e / SUM(e) OVER ()) AS softmax
                FROM    (
                        SELECT  *,
                                (value - MAX(value) OVER ()) / temperature AS diff
                        FROM    logits
                        ) exp_x
                CROSS JOIN LATERAL
                        (
                        SELECT  CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
                        ) exp
                ) q
        )
SELECT  token,
        cluster,
        TO_CHAR(t1.value, 'S00.000') AS score,
        TO_CHAR(t1.softmax, '0.00') AS "temperature = 0.5",
        TO_CHAR(t2.softmax, '0.00') AS "temperature = 1",
        TO_CHAR(t3.softmax, '0.00') AS "temperature = 2"
FROM    (
        SELECT  *
        FROM    tokens
        WHERE   temperature = 0.5
        ) t1
JOIN    (
        SELECT  *
        FROM    tokens
        WHERE   temperature = 1
        ) t2
USING   (token)
JOIN    (
        SELECT  *
        FROM    tokens
        WHERE   temperature = 2
        ) t3
USING   (token)
JOIN    tokenizer
USING   (token)

Inference

Finally, we are ready to do some real inference: run the model, select a token according to its probability, add it to the prompt and repeat until enough tokens are generated.

The LLM itself, as we saw before, is deterministic: it's just a series of matrix multiplications and other math operations on predefined constants. As long as the prompt and the hyperparameters like temperature and top_n are the same, the output will also be the same.

The only non-deterministic process is token selection. There is randomness involved in it (to a variable degree). That's why GPT-based chatbots can give different answers to the same prompt.

We will use the phrase "Happy New Year! I wish" as the prompt and make the model generate 10 new tokens for this prompt. The temperature will be set to 2, and top_n will be set to 5.

The query runs for 2:44 minutes on my machine. Here's its output:

SELECT SETSEED(0.20231231);

WITH    RECURSIVE
        input AS
        (
        SELECT  'Happy New Year! I wish you' AS prompt,
                10 AS threshold,
                2 AS temperature,
                1 AS top_n
        ),
        clusters AS
        (
        SELECT  part_position, bpe.*
        FROM    input
        CROSS JOIN LATERAL
                REGEXP_MATCHES(prompt, '''s|''t|''re|''ve|''m|''ll|''d| ?\w+| ?\d+| ?[^\s\w\d]+|\s+(?!\S)|\s+', 'g') WITH ORDINALITY AS rm (part, part_position)
        CROSS JOIN LATERAL
                (
                WITH    RECURSIVE
                        bpe AS
                        (
                        SELECT  (n + 1)::BIGINT AS position, character, TRUE AS continue
                        FROM    CONVERT_TO(part[1], 'UTF-8') AS bytes
                        CROSS JOIN LATERAL
                                GENERATE_SERIES(0, LENGTH(bytes) - 1) AS n
                        JOIN    encoder
                        ON      byte = GET_BYTE(bytes, n)
                        UNION ALL
                        (
                        WITH    RECURSIVE
                                base AS
                                (
                                SELECT  *
                                FROM    bpe
                                WHERE   continue
                                ),
                                bn AS
                                (
                                SELECT  ROW_NUMBER() OVER (ORDER BY position) AS position,
                                        continue,
                                        character,
                                        character || LEAD(character) OVER (ORDER BY position) AS cluster
                                FROM    base
                                ),
                                top_rank AS
                                (
                                SELECT  tokenizer.*
                                FROM    bn
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  *
                                        FROM    tokenizer
                                        WHERE   tokenizer.cluster = bn.cluster
                                        LIMIT   1
                                        ) tokenizer
                                ORDER BY
                                        token
                                LIMIT   1
                                ),
                                breaks AS
                                (
                                SELECT  0::BIGINT AS position, 1 AS length
                                UNION ALL
                                SELECT  bn.position,
                                        CASE WHEN token IS NULL THEN 1 ELSE 2 END
                                FROM    breaks
                                JOIN    bn
                                ON      bn.position = breaks.position + length
                                LEFT JOIN
                                        top_rank
                                USING   (cluster)
                                )
                        SELECT  position, character, token IS NOT NULL
                        FROM    breaks
                        LEFT JOIN
                                top_rank
                        ON      1 = 1
                        CROSS JOIN LATERAL
                                (
                                SELECT  STRING_AGG(character, '' ORDER BY position) AS character
                                FROM    bn
                                WHERE   bn.position >= breaks.position
                                        AND bn.position < breaks.position + length
                                ) bn
                        WHERE   position > 0
                        )
                        )
                SELECT  position, character AS cluster
                FROM    bpe
                WHERE   NOT continue
                ) bpe
        ),
        tokens AS
        (
        SELECT  ARRAY_AGG(token ORDER BY part_position, position) AS input
        FROM    clusters
        JOIN    tokenizer
        USING   (cluster)
        ),
        gpt AS
        (
        SELECT  input, ARRAY_LENGTH(input, 1) AS original_length
        FROM    tokens
        UNION ALL
        SELECT  input || next_token.token, original_length
        FROM    gpt
        CROSS JOIN
                input
        CROSS JOIN LATERAL
                (
                WITH    RECURSIVE
                        hparams AS
                        (
                        SELECT  ARRAY_LENGTH(input, 1) AS n_seq,
                                12 AS n_block
                        ),
                        embeddings AS
                        (
                        SELECT  place, values
                        FROM    hparams
                        CROSS JOIN LATERAL
                                UNNEST(input) WITH ORDINALITY AS tokens (token, ordinality)
                        CROSS JOIN LATERAL
                                (
                                SELECT  ordinality - 1 AS place
                                ) o
                        CROSS JOIN LATERAL
                                (
                                SELECT  wte.values + wpe.values AS values
                                FROM    wte
                                CROSS JOIN
                                        wpe
                                WHERE   wte.token = tokens.token
                                        AND wpe.place = o.place
                                ) embedding
                        ),
                        transform AS
                        (
                        SELECT  0 AS block, place, values
                        FROM    embeddings
                        UNION ALL
                        (
                        WITH    previous AS
                                (
                                SELECT  *
                                FROM    transform
                                )
                        SELECT  block + 1 AS block, transformed_layer.*
                        FROM    hparams
                        CROSS JOIN LATERAL
                                (
                                SELECT  block
                                FROM    previous
                                WHERE   block < 12
                                LIMIT   1
                                ) q
                        CROSS JOIN LATERAL
                                (
                                WITH    ln_2_b AS
                                        (
                                        SELECT  *
                                        FROM    ln_2_b
                                        WHERE   block = q.block
                                        ),
                                        ln_2_g AS
                                        (
                                        SELECT  *
                                        FROM    ln_2_g
                                        WHERE   block = q.block
                                        ),
                                        c_proj_w AS
                                        (
                                        SELECT  *
                                        FROM    c_proj_w
                                        WHERE   block = q.block
                                        ),
                                        c_proj_b AS
                                        (
                                        SELECT  *
                                        FROM    c_proj_b
                                        WHERE   block = q.block
                                        ),
                                        mlp_c_fc_w AS
                                        (
                                        SELECT  *
                                        FROM    mlp_c_fc_w
                                        WHERE   block = q.block
                                        ),
                                        mlp_c_fc_b AS
                                        (
                                        SELECT  *
                                        FROM    mlp_c_fc_b
                                        WHERE   block = q.block
                                        ),
                                        mlp_c_proj_w AS
                                        (
                                        SELECT  *
                                        FROM    mlp_c_proj_w
                                        WHERE   block = q.block
                                        ),
                                        mlp_c_proj_b AS
                                        (
                                        SELECT  *
                                        FROM    mlp_c_proj_b
                                        WHERE   block = q.block
                                        ),
                                        c_attn_w AS
                                        (
                                        SELECT  *
                                        FROM    c_attn_w
                                        WHERE   block = q.block
                                        ),
                                        c_attn_b AS
                                        (
                                        SELECT  *
                                        FROM    c_attn_b
                                        WHERE   block = q.block
                                        ),
                                        ln_1_g AS
                                        (
                                        SELECT  *
                                        FROM    ln_1_g
                                        WHERE   block = q.block
                                        ),
                                        ln_1_b AS
                                        (
                                        SELECT  *
                                        FROM    ln_1_b
                                        WHERE   block = q.block
                                        ),
                                        mha_norm AS
                                        (
                                        SELECT  place, mm.values + c_attn_b.values AS values
                                        FROM    (
                                                SELECT  place, ARRAY_AGG(INNER_PRODUCT(c_attn_w.values, layer_norm.values) ORDER BY y)::VECTOR(2304) AS values
                                                FROM    (
                                                        SELECT  place, agg.values * ln_1_g.values + ln_1_b.values AS values
                                                        FROM    (
                                                                SELECT  place, norm.values
                                                                FROM    previous
                                                                CROSS JOIN LATERAL
                                                                        (
                                                                        SELECT  AVG(value) AS mean,
                                                                                VAR_POP(value) AS variance
                                                                        FROM    UNNEST(values::REAL[]) value
                                                                        ) agg
                                                                CROSS JOIN LATERAL
                                                                        (
                                                                        SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
                                                                        ) norm
                                                                ) agg
                                                        CROSS JOIN
                                                                ln_1_b
                                                        CROSS JOIN
                                                                ln_1_g
                                                        ) layer_norm
                                                CROSS JOIN
                                                        c_attn_w
                                                GROUP BY
                                                        place
                                                ) mm
                                        CROSS JOIN
                                                c_attn_b
                                        ),
                                        heads AS
                                        (
                                        SELECT  place, head,
                                                (values::REAL[])[(head * 64 + 1):(head * 64 + 64)]::VECTOR(64) AS q,
                                                (values::REAL[])[(head * 64 + 1 + 768):(head * 64 + 64 + 768)]::VECTOR(64) AS k,
                                                (values::REAL[])[(head * 64 + 1 + 1536):(head * 64 + 64 + 1536)]::VECTOR(64) AS v
                                        FROM    mha_norm
                                        CROSS JOIN
                                                GENERATE_SERIES(0, 11) head
                                        ),
                                        sm_input AS
                                        (
                                        SELECT  head, h1.place AS x, h2.place AS y, INNER_PRODUCT(h1.q, h2.k) / 8 + CASE WHEN h2.place > h1.place THEN -1E10 ELSE 0 END AS value
                                        FROM    heads h1
                                        JOIN    heads h2
                                        USING   (head)
                                        ),
                                        sm_diff AS
                                        (
                                        SELECT  head, x, y, value - MAX(value) OVER (PARTITION BY head, x) AS diff
                                        FROM    sm_input
                                        ),
                                        sm_exp AS
                                        (
                                        SELECT  head, x, y, CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
                                        FROM    sm_diff
                                        ),
                                        softmax AS
                                        (
                                        SELECT  head, x, y AS place, e / SUM(e) OVER (PARTITION BY head, x) AS value
                                        FROM    sm_exp
                                        ),
                                        attention AS
                                        (
                                        SELECT  place, ARRAY_AGG(value ORDER BY head * 64 + ordinality)::VECTOR(768) AS values
                                        FROM    (
                                                SELECT  head, x AS place, SUM(ARRAY_FILL(softmax.value, ARRAY[64])::VECTOR(64) * heads.v) AS values
                                                FROM    softmax
                                                JOIN    heads
                                                USING   (head, place)
                                                GROUP BY
                                                        head, x
                                                ) q
                                        CROSS JOIN LATERAL
                                                UNNEST(values::REAL[]) WITH ORDINALITY v (value, ordinality)
                                        GROUP BY
                                                place
                                        ),
                                        mha AS
                                        (
                                        SELECT  place, w.values + c_proj_b.values + previous.values AS values
                                        FROM    (
                                                SELECT  attention.place, ARRAY_AGG(INNER_PRODUCT(attention.values, c_proj_w.values) ORDER BY c_proj_w.place)::VECTOR(768) AS values
                                                FROM    attention
                                                CROSS JOIN
                                                        c_proj_w
                                                GROUP BY
                                                        attention.place
                                                ) w
                                        CROSS JOIN
                                                c_proj_b
                                        JOIN    previous
                                        USING   (place)
                                        ),
                                        ffn_norm AS
                                        (
                                        SELECT  place, agg.values * ln_2_g.values + ln_2_b.values AS values
                                        FROM    (
                                                SELECT  place, norm.values
                                                FROM    mha
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  AVG(value) AS mean,
                                                                VAR_POP(value) AS variance
                                                        FROM    UNNEST(values::REAL[]) value
                                                        ) agg
                                                CROSS JOIN LATERAL
                                                        (
                                                        SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                                        FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n(value, ordinality)
                                                        ) norm
                                                ) agg
                                        CROSS JOIN
                                                ln_2_b
                                        CROSS JOIN
                                                ln_2_g
                                        ),
                                        ffn_a AS
                                        (
                                        SELECT  gelu.place, gelu.values
                                        FROM    (
                                                SELECT  place, w.values + mlp_c_fc_b.values AS values
                                                FROM    (
                                                        SELECT  ffn_norm.place, ARRAY_AGG(INNER_PRODUCT(ffn_norm.values, mlp_c_fc_w.values) ORDER BY mlp_c_fc_w.place)::VECTOR(3072) AS values
                                                        FROM    ffn_norm
                                                        CROSS JOIN
                                                                mlp_c_fc_w
                                                        GROUP BY
                                                                ffn_norm.place
                                                        ) w
                                                CROSS JOIN
                                                        mlp_c_fc_b
                                                ) v
                                        CROSS JOIN LATERAL
                                                (
                                                SELECT  place, ARRAY_AGG(0.5 * value * (1 + TANH(0.797884560802 * (value + 0.044715 * value*value*value))) ORDER BY ordinality)::VECTOR(3072) AS values
                                                FROM    UNNEST(values::REAL[]) WITH ORDINALITY n (value, ordinality)
                                                GROUP BY
                                                        place
                                                ) gelu
                                        ),
                                        ffn AS
                                        (
                                        SELECT  place, w.values + mlp_c_proj_b.values + mha.values AS values
                                        FROM    (
                                                SELECT  ffn_a.place, ARRAY_AGG(INNER_PRODUCT(ffn_a.values, mlp_c_proj_w.values) ORDER BY mlp_c_proj_w.place)::VECTOR(768) AS values
                                                FROM    ffn_a
                                                CROSS JOIN
                                                        mlp_c_proj_w
                                                GROUP BY
                                                        ffn_a.place
                                                ) w
                                        CROSS JOIN
                                                mlp_c_proj_b
                                        JOIN    mha
                                        USING   (place)
                                        )
                                SELECT  *
                                FROM    ffn
                                ) transformed_layer
                        )
                        ),
                        block_output AS
                        (
                        SELECT  *
                        FROM    hparams
                        JOIN    transform
                        ON      transform.block = n_block
                        ),
                        ln_f AS
                        (
                        SELECT  place, norm.values * ln_f_g.values + ln_f_b.values AS values
                        FROM    block_output
                        CROSS JOIN LATERAL
                                (
                                SELECT  AVG(value) AS mean,
                                        VAR_POP(value) AS variance
                                FROM    UNNEST(values::REAL[]) AS n(value)
                                ) agg
                        CROSS JOIN LATERAL
                                (
                                SELECT  ARRAY_AGG((value - mean) / SQRT(variance + 1E-5) ORDER BY ordinality)::VECTOR(768) AS values
                                FROM    UNNEST(values::REAL[]) WITH ORDINALITY AS n (value, ordinality)
                                ) norm
                        CROSS JOIN
                                ln_f_b
                        CROSS JOIN
                                ln_f_g
                        ),
                        logits AS
                        (
                        SELECT  token, INNER_PRODUCT(ln_f.values, wte.values) AS value
                        FROM    hparams
                        JOIN    ln_f
                        ON      ln_f.place = n_seq - 1
                        CROSS JOIN
                                wte
                        ORDER BY
                                value DESC
                        LIMIT   (top_n)
                        ),
                        tokens AS
                        (
                        SELECT  token,
                                high - softmax AS low,
                                high
                        FROM    (
                                SELECT  *,
                                        SUM(softmax) OVER (ORDER BY softmax) AS high
                                FROM    (
                                        SELECT  *, (e / SUM(e) OVER ()) AS softmax
                                        FROM    (
                                                SELECT  *,
                                                        (value - MAX(value) OVER ()) / temperature AS diff
                                                FROM    logits
                                                ) exp_x
                                        CROSS JOIN LATERAL
                                                (
                                                SELECT  CASE WHEN diff < -745.13 THEN 0 ELSE EXP(diff) END AS e
                                                ) exp
                                        ) q
                                ) q
                        ),
                        next_token AS
                        (
                        SELECT  *
                        FROM    (
                                SELECT  RANDOM() AS rnd
                                ) r
                        CROSS JOIN LATERAL
                                (
                                SELECT  *
                                FROM    tokens
                                WHERE   rnd >= low
                                        AND rnd < high
                                ) nt
                        )
                SELECT  *
                FROM    next_token
                ) next_token
        WHERE   ARRAY_LENGTH(input, 1) < original_length + threshold
                AND next_token.token <> 50256
        ),
        output AS
        (
        SELECT  CONVERT_FROM(STRING_AGG(SET_BYTE('\x00', 0, byte), '' ORDER BY position), 'UTF8') AS response
        FROM    (
                SELECT  STRING_AGG(cluster, '' ORDER BY ordinality) AS response
                FROM    input
                JOIN    gpt
                ON      ARRAY_LENGTH(input, 1) = original_length + threshold
                CROSS JOIN LATERAL
                        UNNEST(input) WITH ORDINALITY n (token, ordinality)
                JOIN    tokenizer
                USING   (token)
                ) q
        CROSS JOIN LATERAL
                STRING_TO_TABLE(response, NULL) WITH ORDINALITY n (character, position)
        JOIN    encoder
        USING   (character)
        )
SELECT  *
FROM    output

This part the AI got right. I do wish you all the best in your new year!

You can find the queries and the installation code in the GitHub repository: quassnoi/explain-extended-2024

Previous New Year posts:

• 2010: SQL graphics in Oracle, MySQL, SQL Server and PostgreSQL
• 2011: Drawing a clock in SQL
• 2012: Drawing snowflakes in SQL
• 2013: View of Earth from space in SQL
• 2014: Drawing fractals in SQL
• 2015: Composing music in SQL
• 2016: Conway’s Game of Life in SQL
• 2017: The Sultan’s Riddle in SQL
• 2018: Settlers of Catan in SQL
• 2019: GIF decoder in SQL
• 2020: A stereogram in SQL
• 2021: 3D picture of the coronavirus in SQL
• 2022: Quantum computer emulator in SQL
• 2023: Solving the Rubik’s Cube in SQL

The post Happy New Year: GPT in 500 lines of SQL appeared first on EXPLAIN EXTENDED.

This year, we will be solving the Rubik's Cube in SQL.

The Rubik's Cube probably needs little introduction, but just in case. This is a small plastic cube made of 3×3×3 = 27 smaller cubes (called cubies in the community's parlance). It's equipped with a clever mechanism which lets you turn its faces about all six axes. Initially, the cubies are positioned so that every face has its own uniform color, but as you turn the faces in different directions, all the colors mix and spread around. Your goal is to restore the original colors by putting the cubies back in place.

The cubies in the centers of the faces never move relatively to each other. The red center always opposes the orange center, the blue the green, and the white the yellow. Also, if you look at the red, white and blue centers, they will always run clockwise in that order. No matter how you turn the faces on the cube, the center cubies will always stay this way.

Each cubie is unique. Apart from the center cubie in the middle (which we never see) and the six face center cubies, there are 8 corner cubies (with three stickers on them) and 12 edge cubies (with two stickers on them). In a solved cubie, all faces have the same color. This means that every corner cubie has its own unique set of colors, and every edge cubie too. The blue-white edge is the one sitting on the blue and white faces in a solved cube, and so it. Naturally, it means that there's no such thing as a red-orange edge or a red-orange-green corner (because the red face opposes the orange). Since the stickers stay on the cubies as we move them about, we can just name the cubies by their colors. There is only one red-white-blue corner which would move about but always stay red-white-blue.

When the cube is scrambled, it might seem to the untrained eye that the cubies can turn any old way. Yet, that's not the case. Because of the way the mechanism works, there are certain principles that limit the cubies' positions. For instance, no matter how you turn the cube, you cannot make a single corner or a single edge cubie turn in its place, or make just two cubies trade their places, with all the rest staying in place. These limitations are known as cube laws. In fact, if you take the cube apart and put the cubies back in random order, there's only one chance in 12 that you will get it right (i.e. you make a cube that can be solved by making legal turns). It will come handy later.

To describe a cube scramble, we could just list the stickers' colors ("the colors on the face with the red center, counting from the red-white-blue corner clockwise are red, white, yellow, green…") or cubie positions ("the red-yellow-green corner cubie is between the orange, white and green faces…"). But turns out, there is another way.

Similar to other famous combination puzzles, the Rubik's Cube runs on group theory. Even the Wikipedia page on group theory is illustrated by nothing else than an image of a Rubik's Cube. For those wishing to delve into the finer detail, there's a nice article on Wikipedia on the Rubik's Cube group. For the rest of us laypeople, this means that every valid state of the cube can be described by the sequence of moves (face turns) that have lead to this state. Come to think of it, that's only natural: if the cube came out of the box solved and we only scrambled it using the legal moves, then if we replay these moves on another cube, we will get the same scramble with the same colors. So, instead of naming the colors or cubies' positions, we can just say "take a solved cube, turn the white face clockwise, then the red face counterclockwise, then…"). In the end, it will be the same thing.

Note that in the previous paragraph I said that it would be possible, not that it would be easy. In fact, if we take a scrambled cube described by the series of moves from the solved cube, and replay these moves in reverse, we will get back to the solved cube. And that's the very essence of the problem: how do we, looking at a scrambled cube, find the sequence of moves that has lead to this particular state? Once we find the answer to this question, we'll just need to reverse this sequence and the cube will be solved. This answer, however, is not trivial to find.

The original slogan for the Rubik's Cube was "over three billion combinations, and only one right". Albeit technically correct, this was a sheer understatement. If you just take the cube apart and assemble it, there are ways you can put the edge cubies in and ways you can put the corner cubies in. In addition, there are ways you can turn the corner cubies and ways you can turn the edge cubies. As I told before, only one in 12 of those combinations would be correct, still, it leaves us with possible cube states — way way over three billion!

Even the "only one right" part is not quite correct, depending on how you look at it. If we represent the cube by defining the permutations and turns of its corners and edges, there is exactly one representation of the solved state indeed. But let's look at the cube which is one red clockwise turn away from solved. This state is achievable from the solved state by turning the red face counterclockwise once. But so it is by turning the red face counterclockwise five times, or nine; or clockwise three times, or seven. This means that the representation of the cube's state by sequences of moves is not unique. Given that there is an infinite number of move sequences, but only a finite number of valid cube states, it's hardly a surprise.

A naïve way to solve the cube on a computer would be to represent it as a graph. The nodes of the graph would be cube states. The edges would be face turns turning the states one into another. Finding the cube solution would then be equivalent to the task of finding the shortest path in a graph. In general, this is not a hard task, and efficient algorithms do exist for it. However, the number of the cube states has a lot of zeros. It makes straightforward approaches to solving this problem computationally infeasible. However efficient the algorithms are, is too many numbers to crunch.

If you see a puzzle which works in discrete turns, and the turns can be reversed, then most probably you are looking at a group in disguise. One way to deal with a nasty group is to split it into subgroups and see if they would be easier to reason about. If you are not familiar with group theory, this is probably best illustrated with an example.

Let's assume you locked your bike to a rail with a five-digit combination lock and forgot the combination (this, kids, is exactly what we assume when we are talking about cracking combination locks). This is a puzzle not unlike the Rubik's Cube: by turning the dials, you try to set the device into the target state, and the dials are cyclic and reversible. You want to get there as fast as possible, in a minimum number of turns (because it's late, you're tired and want to get home, and not for any other reason you might be thinking of). The lock states are a group as well, with single turns of every dial as generators, and composition of these turns as the group operation.

There are exactly 100 000 states of a five-digit lock. If you work fast and it takes you a second to switch the lock combination and try the lock, on average, you will still need to try 50 000 combinations before you open the lock. This would take you all day and a good portion of the night.

However, if the lock is crappy (most of them are), you can pull the shackle and turn the last dial (or the first, depending on the lock mechanism), while the shackle is under tension. You will feel or even hear a faint click as the dial locks into its place, and after that it will be somewhat hard to turn it further. You then leave this dial be, and, still pulling the shackle, try the next dial, and so on. This way, you will need to make at most 50 turns to get the combination right, less than a minute.

What you just did was an exercise in group theory. You, incrementally, identified subgroups containing the target state and pruned the graph so that it would only connect vertices of those subgroups. Should the police ask you what you were doing, you could just tell them you were doing math!

The Rubik's Cube puzzle is a little bit different from the padlock puzzle. With the Rubik's Cube we know the target state but don't know how to get there. This is exactly the opposite of the combination lock problem. However, the subgroup strategy remains the same: first, we figure out how to get into the subgroup, then how to get to the target state within the subgroup.

If you ever start googling for computer algorithms solving the Rubik's Cube, you will inevitably land on Jaap Scherphuis's Puzzle Page, describing the Thistlethwaite's algorithm for solving the cube. This algorithm was developed by Morwen Thistlethwaite in the 80's. It's quite old, and better algorithms have been developed since, but this one is still of high theoretical and practical importance. It's less resource intensive than the others, relatively easy to implement, and for a quite a long time the humanity knew nothing better.

Thistlethwaite's algorithm sequentially breaks the Rubik's Cube group into subgroups by limiting the inventory of turns. The indexes of each subgroup (the factor by which we shrink the space of target states on each step) are of palatable size. When solving the padlock, each time we locked a dial, we had been limiting our moves within the graph to a subgroup of index 10 (thus decreasing the space of possible solutions by the factor of 10 on each step). Thistlethwaite's algorithm uses four subgroups with indexes of 2 048, 1 082 656, 29 400, and 663 552, accordindly. The product of these numbers is the enormous cardinality of the Rubik's Cube group. But by breaking the solution into subgroups, we can, roughly speaking, reduce the algorithm complexity from product of these numbers to their sum, which is in the range of millions. Millions is still a big number, but nothing a today's computer can't handle, even in SQL.

Jaap's page hosts the original scans of Dr. Thistlethwaite's letter with the description of the algorithm. The letter begins with an optimistic passage:

I should perhaps point out that this strategy isn't easy to perform (even John Conway finds it quite hard!), for two reasons: first, only one representative of each symmetry class is given in the tables; second, I have given only the barest documentation.

This is true in every possible way. Grasping that algorithm was a very hard task for me, even though knowing that John Conway had the same problem was in some weird way a little bit reassuring. I spent a great deal of time, first trying to figure it out on my own, and, then, looking for an existing implementation. I found lots of questions and answers on Stack Exchange and other resources, including those from Mr. Scherphuis himself, but they didn't quite click with me until I found this implementation in Go by Drew Finnis and Anya Schukin. This one was really easy to follow, and I adopted it for this post with some changes.

Terminology

Rubik's Cube's states, moves and solutions are traditionally written down and evaluated using so called Singmaster's notation. The cube is fixed in space so that, say, the red face is always up, the blue one right, and the white one front, but you are free to select any initial reference position you want, as long as you stick to it. Then, the moves are written down using the acronyms (for up, right, front, down, left and back, respectively). A letter on its own means a clockwise quarter-turn, a letter followed by a prime symbol means a counterclockwise quarter-turn, and a letter followed by a square means a half-turn (traditionally, a half-turn counts as one turn when evaluating the solution length, even though it's composed of two quarter-turns). So, in Singmaster's notation, would mean "turn the down face clockwise by 90 degrees, then the up face counterclockwise by 90 degrees, then the front face by 180 degrees".

We will use a slightly modified Singmaster's notation, because the prime symbol and the superscript are not that easy to type on my keyboard. I first thought about replacing the prime symbol with a single quote, but this is a special symbol in SQL and I would need to escape it in strings. So we will be using uppercase letters, like , for 90° turns (quarter-turn clockwise); uppercase letters appended with 2, like , for 180° turns (half-turns); and lowercase letters, like , for 270° turns (quarter-turn counterclockwise).

As the algorithm works its way through the moves, it needs to look at the cube's representation in terms of cubies' positions. As we saw above, to describe the cube, it's enough to list the positions and turns of its corner and edge cubies. We can do that by looking at the solved state and assigning a number from 0 to 7 to every corner cubie, a number from 0 to 11 to every edge cubie, and also encode the corner cubies' orientations and edge cubies' orientations with a number from 0 to 2 and 0 to 1, respectively. So, to completely encode the cube's state, we would need two arrays of size 12 (positions and orientations of the edge cubies) and two arrays of size 8 (positions and orientation of the corner cubies). The first array in these pairs would hold the permutations (unique numbers from 0 to the size of the array less one, in different order); the second would orientations (any combination of numbers from 0 to 2 and from 0 to 1, respectively). This notation is slightly redundant because it can encode invalid states, but it's easy to work with.

The way we enumerate the cubies doesn't really matter, as long as it's consistent, but a smart choice of the enumeration system will make our SQL queries a little bit easier to grasp. I used the same encoding as the Go program did, which in its turn had been adopted from Dr. Thistletwaite's paper. At first glance it looks a little bit confusing, but it will make sense later.

The permutations are easy to read: if we see something like for the corner cubies, it means that the cubies 0 and 1 are in their original corners (ULB and DLF, respectively), the cubie 5 (the DLB cubie) sits in the corner 2 (the DRB corner), and so on. means that all the corner cubies sit in their right places (although some of them might be turned the wrong way). It works the same way for the edge cubies.

Orientations are a little bit trickier. Let's consider the corners first. If a cubie sits in its own corner, then 0 means that the cubie is turned the right way, 1 that it's turned one third clockwise and 2 that it's turned one third counterclockwise. On that much we can easily agree. But what exactly is the "right" way if it doesn't sit in its own corner?

We can notice that every corner cubie has one U or D facet, one L or R facet and one F or B facet. Let's, for a moment, imagine that the opposing colors are the same. We'll call these colors "dup", "reft" and "fack" "bront". This way, all corner cubies will have the same colors, but four of them will become mirror reflections of the other four. We'll randomly assign one of the colors (say, "reft") to be the reference color. This way it becomes easy: every cubie, regardless of its position, will have a natural orientation — that of its "reft" facet. Let's notice that turning the faces about the L/R axis (making the L or R turns, or combinations thereof) does not change the orientations of the cubies. It will become important later.

We can reason about the edge orientation in a similar way. The edge cubies have just two facets with stickers on them, but let's pretend they have all six and we just don't see the rest. We can assign a reference axis (let it be the U/B axis) and just say that a quarter turn of an edge cubie in place about the U/D axis changes its orientation, while the turns about all other axes keep it. So if an F/L cubie sits in the U/R slot, we just need to count how many quarter-turns about the U/D axis we need to make to drive it into the correct orientation in space. We have two special axes for the edge cubies (L/R and F/B) which don't change their orientation being turned about.

Equipped with this knowledge, we can start attacking the problem.

Subgroup 1

A scrambled cube can have, modulo cube laws, any old combination of corner and edge permutations and orientations. However, we can note that the only way we can change the edge orientations is by quarter-turning up or down faces (i.e. making moves from this list: ). Since the other two axes are special, turns about them don't change edge orientations.

If we can orient all the edges correctly, and then refrain from making the edge-turning moves (quarter-turns about the U/D axis; half-turns are OK), then the edges will stay oriented correctly. This is exactly what we did with the padlock: as soon as we had found a part of the solution, we locked it in place and then refrained from the moves which would change it.

From the group-theoretic point of view, we would need to find an element in the cube's group so that its product with the scrambled state as a left factor would be in subgroup . In plain English it means that we need to look at the cube, find a sequence of moves which would fix the edges' orientations, and then stop making any moves which would break it.

We will further call the subgroup Subgroup 1. How do we find out how to get into Subgroup 1?

There are 12 edge cubies, which, if you take the cube apart and assemble it, can be oriented different ways. Due to the cube's laws, only half of these orientations are reachable through legal moves. This means that there are only 2 048 different orientation combinations.

Of course, not only edge orientations change as we turn the cube. The cubies also travel and turn all around the surface. But at this stage, these other things don't interest us. Right now, we should be laser-focused on this one particular aspect. When we were probing the padlock for the correct position of the first dial, we didn't pay attention to the other ones. From the shackle's perspective, the other dials don't exist until you solve the first one. So would could as well cover them with the painter's tape to focus on the first one.

We can do a similar thing with the cube. Let's take a piece of painter's tape and cover all the cubies' stickers with it. Then let's put new images on the edge cubies. The F and B facets of all the edge cubies get a purple sticker, and the R and L facets get a teal one. The U and D facets of the edge cubies get the opposite of what it is on their other sticker. The solved cube would look like this:

To get us into Subgroup 1, we would just need to solve this simpler cube. And this is something we can just do with extensive search in a smaller graph, only containing 2 048 vertices.

There are several algorithms that can find the shortest path in a graph. Of them, the breadth-first search is the most simple one, although it's usually the most memory-demanding. We're working with SQL, though, which is not shy of storing data.

For this task, we will encode the states of the edge orientations with numbers 0 and 1. We will start from the solved cube (all 0's). For the first step we will see what happens one turn away from the solved state, and there are 18 ways we can do that. These turns will takes us to the new cube states. For these new states, we will know that there is a one-turn sequence which can bring us back to solved.

Some of these moves (F, B, L, R, their inverses and squares, and also U2 and D2) will get us back to the solved state. We don't really need any moves to get us back to where we were, so these sequences we will discard.

Other moves will get us to a new state, materially different from the solved one. We just learned something new about these states: to get from these states back to solved, we need exactly one move, the inverse of what we just did. Since it's really hard to compete with one-turn, we can be sure that it's the optimal path. There will be some duplicates: and are really the same move, as long as the edge orientations are concerned. We don't need the duplicates either, so we will toss them as well. We will be left with a list of new edge orientation states and sequences of one-turn moves that have lead to them.

Starting from these new sequences, we continue our search. To every sequence, we will append all of the moves in our inventory and see where they will lead us. Sometimes, they will lead us back to the solved state, or back to some state we saw earlier. Since everything that we saw earlier required less turns to get there (otherwise we wouldn't see it), this new sequence is not optimal. We will discard it and repeat our search.

On each stage of this algorithm, we will have an ever-growing table of states. With each state comes a sequence of or less turns. It may seem that this list of states would be exponentially increasing on every step of the algorithm, but it's not. As we travel about the graph, we remember where we already have been, how many turns it got us to get there, and make sure not to take the same route twice. At some point the duplicates will start to outnumber new solutions. On a certain step of the algorithm we won't be able to add anything new into the lookup table: we will already have seen it all. This is our cue to stop searching and return the results.

Applying the same transformation to a recordset in a loop is something that recursive CTE's lend themselves naturally to. So let's try to implement it.

First, we will need to code states and turns of the Rubik's cube. We will start with basic moves, and encode the permutations and orientation changes these moves make. PostgreSQL arrays are a good way to do this. We will also create a simple helper function to encode the group multiplication operation. This function will take two group elements, encoded as four arrays each, and return them as a single record with four arrays:

CREATE OR REPLACE FUNCTION turn(cornerPosition INT[], edgePosition INT[], cornerOrientation INT[], edgeOrientation INT[], cornerMove INT[], edgeMove INT[], cornerTurn INT[], edgeTurn INT[])
RETURNS TABLE
        (
        cornerPosition INT[],
        edgePosition INT[],
        cornerOrientation INT[],
        edgeOrientation INT[]
        )
AS
$$

SELECT  ARRAY
        (
        SELECT  cornerPosition
        FROM    UNNEST(cornerMove) AS move (source)
        ),
        ARRAY
        (
        SELECT  edgePosition
        FROM    UNNEST(edgeMove) AS move (source)
        ),
        ARRAY
        (
        SELECT  (cornerOrientation + cornerTurn[destination] + 3) % 3
        FROM    UNNEST(cornerMove) WITH ORDINALITY AS move (source, destination)
        ),
        ARRAY
        (
        SELECT  (edgeOrientation + edgeTurn[destination] + 2) % 2
        FROM    UNNEST(edgeMove) WITH ORDINALITY AS move (source, destination)
        );
$$
LANGUAGE 'sql'
STRICT
IMMUTABLE
PARALLEL SAFE
ROWS 1

Of course, using procedural code in SQL would be cheating. But this function is not procedural, it's just a little syntax sugar to help us with bulky expressions. We could easily make do without it, at expense of doing more typing.

With this function, we can define clockwise quarter-turns in query code and generate all other single turns from them:

WITH    RECURSIVE
        basicMoves (initial, cornerMove, edgeMove, cornerTurn, edgeTurn) AS
        (
        VALUES
                (
                'U',
                ARRAY[ 4,  1,  2,  7,  3,  5,  6,  0]::INT[],  ARRAY[ 8,  1,  2, 11,  4,  5,  6,  7,  3,  9, 10,  0]::INT[],
                ARRAY[ 2,  0,  0,  2,  1,  0,  0,  1]::INT[],  ARRAY[ 1,  0,  0,  1,  0,  0,  0,  0,  1,  0,  0,  1]::INT[]
                ),
                (
                'D',
                ARRAY[ 0,  5,  6,  3,  4,  2,  1,  7]::INT[],  ARRAY[ 0, 10,  9,  3,  4,  5,  6,  7,  8,  1,  2, 11]::INT[],
                ARRAY[ 0,  2,  2,  0,  0,  1,  1,  0]::INT[],  ARRAY[ 0,  1,  1,  0,  0,  0,  0,  0,  0,  1,  1,  0]::INT[]
                ),
                (
                'F',
                ARRAY[ 0,  6,  2,  4,  1,  5,  3,  7]::INT[],  ARRAY[ 0,  1,  2,  3,  4,  9,  8,  7,  5,  6, 10, 11]::INT[],
                ARRAY[ 0,  1,  0,  1,  2,  0,  2,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'B',
                ARRAY[ 7,  1,  5,  3,  4,  0,  6,  2]::INT[],  ARRAY[ 0,  1,  2,  3, 11,  5,  6, 10,  8,  9,  4,  7]::INT[],
                ARRAY[ 1,  0,  1,  0,  0,  2,  0,  2]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'L',
                ARRAY[ 5,  4,  2,  3,  0,  1,  6,  7]::INT[],  ARRAY[ 4,  5,  2,  3,  1,  0,  6,  7,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'R',
                ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3]::INT[],  ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                )
        ),
        moves (turns, initial, move, reverse, cornerMove, edgeMove, cornerTurn, edgeTurn) AS MATERIALIZED
        (
        SELECT  1 AS turns, initial, initial, LOWER(initial),
                cornerMove, edgeMove, cornerTurn, edgeTurn
        FROM    basicMoves bm
        UNION ALL
        SELECT  turns + 1, initial,
                CASE turns WHEN 1 THEN initial || '2' WHEN 2 THEN LOWER(initial) END,
                CASE turns WHEN 1 THEN initial || '2' WHEN 2 THEN initial END,
                t.*
        FROM    moves am
        JOIN    basicMoves bm
        USING   (initial)
        CROSS JOIN LATERAL
                turn(am.cornerMove, am.edgeMove, am.cornerTurn, am.edgeTurn, bm.cornerMove, bm.edgeMove, bm.cornerTurn, bm.edgeTurn) t
        WHERE   turns <= 2
        )
SELECT  move, reverse, cornerMove, edgeMove, cornerTurn, edgeTurn
FROM    moves

This CTE returns the moves, their reverses and the group elements corresponding to these moves. Note the AS MATERIALIZED hint at the end of the CTE definition. We will be using this CTE a lot in joins, and it makes more sense to store it in an internal temp table rather than recalculate every time.

Next, we need to implement breadth-first search algorithm to find the sequences of moves leading to correct edge orientations. As I said above, we will be doing it in a recursive CTE.

We encode the solved state as string '000000000000'. For the anchor part of the CTE, we will select this solved state along with three more service fields:

WITH    g1s AS MATERIALIZED
        (
        SELECT  0 AS depth, 0 AS distance,
                '000000000000' AS edgeOrientation,
                ARRAY[]::TEXT[] AS moves
        )
SELECT  *
FROM    g1s

Now, let's do the breadth-first search part. We need to take all possible moves, apply them to the state and see what happens. To do this, we will cross join our anchor part with moves, do some array computations and append the results to the anchor part. For the sake of clarity, we are not doing the any recursive CTE's just yet and code everything explicitly.

WITH    RECURSIVE
        basicMoves (initial, cornerMove, edgeMove, cornerTurn, edgeTurn) AS
        (
        VALUES
                (
                'U',
                ARRAY[ 4,  1,  2,  7,  3,  5,  6,  0]::INT[],  ARRAY[ 8,  1,  2, 11,  4,  5,  6,  7,  3,  9, 10,  0]::INT[],
                ARRAY[ 2,  0,  0,  2,  1,  0,  0,  1]::INT[],  ARRAY[ 1,  0,  0,  1,  0,  0,  0,  0,  1,  0,  0,  1]::INT[]
                ),
                (
                'D',
                ARRAY[ 0,  5,  6,  3,  4,  2,  1,  7]::INT[],  ARRAY[ 0, 10,  9,  3,  4,  5,  6,  7,  8,  1,  2, 11]::INT[],
                ARRAY[ 0,  2,  2,  0,  0,  1,  1,  0]::INT[],  ARRAY[ 0,  1,  1,  0,  0,  0,  0,  0,  0,  1,  1,  0]::INT[]
                ),
                (
                'F',
                ARRAY[ 0,  6,  2,  4,  1,  5,  3,  7]::INT[],  ARRAY[ 0,  1,  2,  3,  4,  9,  8,  7,  5,  6, 10, 11]::INT[],
                ARRAY[ 0,  1,  0,  1,  2,  0,  2,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'B',
                ARRAY[ 7,  1,  5,  3,  4,  0,  6,  2]::INT[],  ARRAY[ 0,  1,  2,  3, 11,  5,  6, 10,  8,  9,  4,  7]::INT[],
                ARRAY[ 1,  0,  1,  0,  0,  2,  0,  2]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'L',
                ARRAY[ 5,  4,  2,  3,  0,  1,  6,  7]::INT[],  ARRAY[ 4,  5,  2,  3,  1,  0,  6,  7,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'R',
                ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3]::INT[],  ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                )
        ),
        moves (turns, initial, move, cornerMove, edgeMove, cornerTurn, edgeTurn) AS MATERIALIZED
        (
        SELECT  1 AS turns, initial, bm.*
        FROM    basicMoves bm
        UNION ALL
        SELECT  turns + 1, move,
                CASE turns + 1 WHEN 2 THEN initial || '2' WHEN 3 THEN LOWER(initial) END,
                t.*
        FROM    moves am
        JOIN    basicMoves bm
        USING   (initial)
        CROSS JOIN LATERAL
                turn(am.cornerMove, am.edgeMove, am.cornerTurn, am.edgeTurn, bm.cornerMove, bm.edgeMove, bm.cornerTurn, bm.edgeTurn) t
        WHERE   turns <= 2
        ),
        g1s AS MATERIALIZED
        (
        SELECT  0 AS depth, 0 AS distance,
                '000000000000' AS state,
                ARRAY[]::TEXT[] AS moves
        )
SELECT  distance, state, moves
FROM    g1s
UNION ALL
SELECT  distance + 1, newState, moves || move
FROM    g1s
CROSS JOIN
        moves
CROSS JOIN LATERAL
        (
        SELECT  STRING_AGG(((SUBSTRING(state FROM source + 1 FOR 1)::INT + edgeTurn[destination] + 2) % 2)::TEXT, '')
        FROM    UNNEST(edgeMove) WITH ORDINALITY AS move (source, destination)
        ) AS n1 (newState)

We see a lot of duplicates in the results. Most of them are from the previous (solved) stage with distance 0. Others are from the current stage. We will need to get rid of all of them. The former we will need to eliminate with a simple NOT IN construct. For the latter, we will use PostgreSQL's native DISTINCT ON construct. In other SQL dialects, we would need to use ROW_NUMBER and wrap it into an additional layer of nested views to be able to filter on it.

Here's how looks (again, without the recursion just yet):

WITH    RECURSIVE
        basicMoves (initial, cornerMove, edgeMove, cornerTurn, edgeTurn) AS
        (
        VALUES
                (
                'U',
                ARRAY[ 4,  1,  2,  7,  3,  5,  6,  0]::INT[],  ARRAY[ 8,  1,  2, 11,  4,  5,  6,  7,  3,  9, 10,  0]::INT[],
                ARRAY[ 2,  0,  0,  2,  1,  0,  0,  1]::INT[],  ARRAY[ 1,  0,  0,  1,  0,  0,  0,  0,  1,  0,  0,  1]::INT[]
                ),
                (
                'D',
                ARRAY[ 0,  5,  6,  3,  4,  2,  1,  7]::INT[],  ARRAY[ 0, 10,  9,  3,  4,  5,  6,  7,  8,  1,  2, 11]::INT[],
                ARRAY[ 0,  2,  2,  0,  0,  1,  1,  0]::INT[],  ARRAY[ 0,  1,  1,  0,  0,  0,  0,  0,  0,  1,  1,  0]::INT[]
                ),
                (
                'F',
                ARRAY[ 0,  6,  2,  4,  1,  5,  3,  7]::INT[],  ARRAY[ 0,  1,  2,  3,  4,  9,  8,  7,  5,  6, 10, 11]::INT[],
                ARRAY[ 0,  1,  0,  1,  2,  0,  2,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'B',
                ARRAY[ 7,  1,  5,  3,  4,  0,  6,  2]::INT[],  ARRAY[ 0,  1,  2,  3, 11,  5,  6, 10,  8,  9,  4,  7]::INT[],
                ARRAY[ 1,  0,  1,  0,  0,  2,  0,  2]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'L',
                ARRAY[ 5,  4,  2,  3,  0,  1,  6,  7]::INT[],  ARRAY[ 4,  5,  2,  3,  1,  0,  6,  7,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'R',
                ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3]::INT[],  ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                )
        ),
        moves (turns, initial, move, cornerMove, edgeMove, cornerTurn, edgeTurn) AS MATERIALIZED
        (
        SELECT  1 AS turns, initial, bm.*
        FROM    basicMoves bm
        UNION ALL
        SELECT  turns + 1, move,
                CASE turns + 1 WHEN 2 THEN initial || '2' WHEN 3 THEN LOWER(initial) END,
                t.*
        FROM    moves am
        JOIN    basicMoves bm
        USING   (initial)
        CROSS JOIN LATERAL
                turn(am.cornerMove, am.edgeMove, am.cornerTurn, am.edgeTurn, bm.cornerMove, bm.edgeMove, bm.cornerTurn, bm.edgeTurn) t
        WHERE   turns <= 2
        ),
        g1s AS MATERIALIZED
        (
        SELECT  0 AS depth, 0 AS distance,
                '000000000000' AS state,
                ARRAY[]::TEXT[] AS moves
        )
SELECT  distance, state, moves
FROM    g1s
UNION ALL
SELECT  DISTINCT ON (newState)
        distance + 1, newState, moves || move
FROM    g1s
CROSS JOIN
        moves
CROSS JOIN LATERAL
        (
        SELECT  STRING_AGG(((SUBSTRING(state FROM source + 1 FOR 1)::INT + edgeTurn[destination] + 2) % 2)::TEXT, '')
        FROM    UNNEST(edgeMove) WITH ORDINALITY AS move (source, destination)
        ) AS n1 (newState)
WHERE   newState NOT IN
        (
        SELECT  state
        FROM    g1s
        )

This means that the only meaningful moves that get the edge orientations out of the solved state are and . Of course, so do and , but they will lead to the same exact result, as far as Subgroup 1 is concerned, so we don't really need to clutter our solution table with them.

For the second pass of the algorithm (again, without recursion):

WITH    RECURSIVE
        basicMoves (initial, cornerMove, edgeMove, cornerTurn, edgeTurn) AS
        (
        VALUES
                (
                'U',
                ARRAY[ 4,  1,  2,  7,  3,  5,  6,  0]::INT[],  ARRAY[ 8,  1,  2, 11,  4,  5,  6,  7,  3,  9, 10,  0]::INT[],
                ARRAY[ 2,  0,  0,  2,  1,  0,  0,  1]::INT[],  ARRAY[ 1,  0,  0,  1,  0,  0,  0,  0,  1,  0,  0,  1]::INT[]
                ),
                (
                'D',
                ARRAY[ 0,  5,  6,  3,  4,  2,  1,  7]::INT[],  ARRAY[ 0, 10,  9,  3,  4,  5,  6,  7,  8,  1,  2, 11]::INT[],
                ARRAY[ 0,  2,  2,  0,  0,  1,  1,  0]::INT[],  ARRAY[ 0,  1,  1,  0,  0,  0,  0,  0,  0,  1,  1,  0]::INT[]
                ),
                (
                'F',
                ARRAY[ 0,  6,  2,  4,  1,  5,  3,  7]::INT[],  ARRAY[ 0,  1,  2,  3,  4,  9,  8,  7,  5,  6, 10, 11]::INT[],
                ARRAY[ 0,  1,  0,  1,  2,  0,  2,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'B',
                ARRAY[ 7,  1,  5,  3,  4,  0,  6,  2]::INT[],  ARRAY[ 0,  1,  2,  3, 11,  5,  6, 10,  8,  9,  4,  7]::INT[],
                ARRAY[ 1,  0,  1,  0,  0,  2,  0,  2]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'L',
                ARRAY[ 5,  4,  2,  3,  0,  1,  6,  7]::INT[],  ARRAY[ 4,  5,  2,  3,  1,  0,  6,  7,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'R',
                ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3]::INT[],  ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                )
        ),
        moves (turns, initial, move, cornerMove, edgeMove, cornerTurn, edgeTurn) AS MATERIALIZED
        (
        SELECT  1 AS turns, initial, bm.*
        FROM    basicMoves bm
        UNION ALL
        SELECT  turns + 1, move,
                CASE turns + 1 WHEN 2 THEN initial || '2' WHEN 3 THEN LOWER(initial) END,
                t.*
        FROM    moves am
        JOIN    basicMoves bm
        USING   (initial)
        CROSS JOIN LATERAL
                turn(am.cornerMove, am.edgeMove, am.cornerTurn, am.edgeTurn, bm.cornerMove, bm.edgeMove, bm.cornerTurn, bm.edgeTurn) t
        WHERE   turns <= 2
        ),
        g1s AS MATERIALIZED
        (
        SELECT  0 AS depth, 0 AS distance,
                '000000000000' AS state,
                ARRAY[]::TEXT[] AS moves
        ),
        pass1 AS
        (
        SELECT  distance, state, moves
        FROM    g1s
        UNION ALL
        SELECT  DISTINCT ON (newState)
                distance + 1, newState, moves || move
        FROM    g1s
        CROSS JOIN
                moves
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(((SUBSTRING(state FROM source + 1 FOR 1)::INT + edgeTurn[destination] + 2) % 2)::TEXT, '')
                FROM    UNNEST(edgeMove) WITH ORDINALITY AS move (source, destination)
                ) AS n1 (newState)
        WHERE   newState NOT IN
                (
                SELECT  state
                FROM    g1s
                )
        )
SELECT  distance, state, moves
FROM    pass1
UNION ALL
SELECT  DISTINCT ON (newState)
        distance + 1, newState, moves || move
FROM    pass1
CROSS JOIN
        moves
CROSS JOIN LATERAL
        (
        SELECT  STRING_AGG(((SUBSTRING(state FROM source + 1 FOR 1)::INT + edgeTurn[destination] + 2) % 2)::TEXT, '')
        FROM    UNNEST(edgeMove) WITH ORDINALITY AS move (source, destination)
        ) AS n1 (newState)
WHERE   newState NOT IN
        (
        SELECT  state
        FROM    pass1
        )

Now we're on to something! We are getting some really meaningful sequences that get us out of the solved state out into the wild, and no duplicates to boot. Note that the SELECT part of the query is almost a literal copy-paste of the CTE pass1.

Now we are ready to pack it into a real recursive CTE. There are three issues to consider.

First, to get rid of duplicates, we should drag the results of all passes through the recursion, not just the last one.

Second, we need to refer to the recursive part of the CTE in something other than a join, namely, in a NOT IN query. PostgreSQL won't let us do this straight away, but we can trick it by wrapping our recursive part in another layer of CTE (more on that later).

Third, we need to know when to stop. We could have written a stop condition by comparing the max distance with the current depth, but this, albeit possible, would require additional CTE trickery and make the query slow. We'll just take Dr. Thistlethwaite's word for this step requiring no more than 7 turns, and hardcode it into the query as a stop condition.

WITH    RECURSIVE
        basicMoves (initial, cornerMove, edgeMove, cornerTurn, edgeTurn) AS
        (
        VALUES
                (
                'U',
                ARRAY[ 4,  1,  2,  7,  3,  5,  6,  0]::INT[],  ARRAY[ 8,  1,  2, 11,  4,  5,  6,  7,  3,  9, 10,  0]::INT[],
                ARRAY[ 2,  0,  0,  2,  1,  0,  0,  1]::INT[],  ARRAY[ 1,  0,  0,  1,  0,  0,  0,  0,  1,  0,  0,  1]::INT[]
                ),
                (
                'D',
                ARRAY[ 0,  5,  6,  3,  4,  2,  1,  7]::INT[],  ARRAY[ 0, 10,  9,  3,  4,  5,  6,  7,  8,  1,  2, 11]::INT[],
                ARRAY[ 0,  2,  2,  0,  0,  1,  1,  0]::INT[],  ARRAY[ 0,  1,  1,  0,  0,  0,  0,  0,  0,  1,  1,  0]::INT[]
                ),
                (
                'F',
                ARRAY[ 0,  6,  2,  4,  1,  5,  3,  7]::INT[],  ARRAY[ 0,  1,  2,  3,  4,  9,  8,  7,  5,  6, 10, 11]::INT[],
                ARRAY[ 0,  1,  0,  1,  2,  0,  2,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'B',
                ARRAY[ 7,  1,  5,  3,  4,  0,  6,  2]::INT[],  ARRAY[ 0,  1,  2,  3, 11,  5,  6, 10,  8,  9,  4,  7]::INT[],
                ARRAY[ 1,  0,  1,  0,  0,  2,  0,  2]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'L',
                ARRAY[ 5,  4,  2,  3,  0,  1,  6,  7]::INT[],  ARRAY[ 4,  5,  2,  3,  1,  0,  6,  7,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'R',
                ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3]::INT[],  ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                )
        ),
        moves (turns, initial, move, cornerMove, edgeMove, cornerTurn, edgeTurn) AS MATERIALIZED
        (
        SELECT  1 AS turns, initial, bm.*
        FROM    basicMoves bm
        UNION ALL
        SELECT  turns + 1, move,
                CASE turns + 1 WHEN 2 THEN initial || '2' WHEN 3 THEN LOWER(initial) END,
                t.*
        FROM    moves am
        JOIN    basicMoves bm
        USING   (initial)
        CROSS JOIN LATERAL
                turn(am.cornerMove, am.edgeMove, am.cornerTurn, am.edgeTurn, bm.cornerMove, bm.edgeMove, bm.cornerTurn, bm.edgeTurn) t
        WHERE   turns <= 2
        ),
        g1s AS MATERIALIZED
        (
        SELECT  0 AS depth, 0 AS distance,
                '000000000000' AS state,
                ARRAY[]::TEXT[] AS moves
        UNION ALL
        (
        WITH    q AS
                (
                SELECT  depth + 1 AS depth, distance, state, moves
                FROM    g1s
                )
        SELECT  *
        FROM    q
        WHERE   depth <= 7
        UNION ALL
        SELECT  DISTINCT ON (newState)
                depth, distance + 1, newState, moves || move
        FROM    q
        CROSS JOIN
                moves
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(((SUBSTRING(state FROM source + 1 FOR 1)::INT + edgeTurn[destination] + 2) % 2)::TEXT, '')
                FROM    UNNEST(edgeMove) WITH ORDINALITY AS move (source, destination)
                ) AS state (newState)
        WHERE   distance = depth - 1
                AND newState NOT IN
                (
                SELECT  state
                FROM    q q2
                )
                AND depth <= 7
        )
        )
SELECT  state, moves
FROM    g1s
WHERE   depth = 7 

Digression: limitations or recursive CTE's

This is not exactly relevant to the main topic of the article, so if you're only here to see the Rubik's Cube solved, feel free to skip to the next chapter.

Recursive CTE's are a way to bring, well, recursion into a declarative language that SQL is. In fact, their existence is the crucial part for SQL being Turing complete.

In SQL, you can create functions and put the values of their variables into WHERE, LIMIT and some other clauses of SQL statements. You can treat the query SELECT * FROM mytable WHERE field1 = 2 as a function, which takes an integer (2) as an argument, and returns a resultset as a result. But if you squint hard enough, you can see how this query can also be treated as a function, which takes a resultset (mytable) as an argument and produces a resultset with the same set of fields as a result. Resultsets are not first-class citizens in SQL, so there is no real way to do this short of making dynamic SQL queries. Still, let us make this assumption and pretend that we can.

A recursive CTE takes form of WITH q AS (SELECT * FROM anchor UNION ALL SELECT * FROM f(q)). Here, f(q) is a function over a resultset. If you could actually pass a resultset into a function variable and use the variable in the FROM clause: SELECT * FROM _variable JOIN othertable, that would be it.

Now let's remember properties of UNION ALL and JOIN. If we union all two resultsets with and rows, respectively, in each, we get records back. If we cross join these two resultsets, we get records back. UNION ALL distributes over JOIN: this means that these two queries:

SELECT * FROM a CROSS JOIN c UNION ALL SELECT * FROM b CROSS JOIN c

and

SELECT * FROM (SELECT * FROM a UNION ALL SELECT * FROM b) q CROSS JOIN c

are the same. Both queries return records, which is the same as . In other words, we can treat UNION ALL as an addition of recordsets, and a JOIN as their multiplication.

Put in these terms, this CTE:

WITH q AS (SELECT * FROM anchor UNION ALL SELECT * FROM f(q))

returns

In this form, you can see how it can be useful to work with hierarchical tree structures: if stores records in a child-parent relationship and means "children of records in ", then the recursive CTE would say: "give me , plus children of , plus children of children of , plus …". In other words, recursive CTE's help build transitive closures of binary relations.

That's one, undoubtedly the most common, way of using a recursive CTE, but not the only one possible. This can be any old function, and most of my New Year posts make a good use of this fact.

But database developers are not fond of allowing any old functions in the recursive part, for a good reason. If you let people put anything they want into the recursive part, the database engine never knows when to terminate the query. At first glance, a good time to stop would be when f(q) returns an empty resultset. But people can program the function so that it returns a nonempty resultset when passed an empty one (say, like this: SELECT 1 EXCEPT SELECT id FROM q). Of course, the engine could check that on the next pass and stop when both and return empty resultsets. In other words, the empty resultset should be a fixpoint for the function . Yet, SQL is not free of side effects, so a really nasty developer could make the code depend on time of day or something like that. Strictly speaking, the engine should apply the recursive part to the empty resultset over and over, in the vain hope that it will return something one day.

Different engines solve this problem differently. SQL Server, for instance, simply doesn't allow anything except inner joins in the recursive part of the expression. With this limitation, the arbitrary function composition series turns into a polynomial series, which is way more simple for the database engine to work with. In particular, the polynomial series can use depth-first search instead of breadth-first search, which will eventually yield the same result, and SQL Server puts this this fact to good use (see this old post of mine for details).

PostgreSQL is not that strict, but still makes some compromises.

First, it forcibly puts an end to this nonsense about empty record not being the fixpoint:

1. Evaluate the non-recursive term. For UNION (but not UNION ALL), discard duplicate rows. Include all remaining rows in the result of the recursive query, and also place them in a temporary working table.
2. So long as the working table is not empty, repeat these steps:
   1. Evaluate the recursive term, substituting the current contents of the working table for the recursive self-reference. For UNION (but not UNION ALL), discard duplicate rows and rows that duplicate any previous result row. Include all remaining rows in the result of the recursive query, and also place them in a temporary intermediate table.
   2. Replace the contents of the working table with the contents of the intermediate table, then empty the intermediate table.

If, at some point, the recursive part returns no results, the query is considered done, period, by definition.

Second, people don't usually write non-terminating Turing machines in SQL, and the main use case of recursive CTE's is building good old transitive closures of adjacency lists. PostgreSQL kind of tries to put some safeguard against misusing the recursion. If you simply try to reuse the CTE name in the recursive part, or put it in a subquery or something, you will get an error. This limitation, however, is easily circumvented by wrapping the recursive part in another layer of CTE, so PostgreSQL lets you shoot yourself in the foot, if you insist hard enough.

This is exactly why we had to do that in the previous query.

Subgroup 2

This subgroup, , is generated by half-turns on all faces except and . To get into this subgroup from an arbitrary point in Subgroup 1 (meaning to get to the state which leads to the solved state using only Subgroup 2 turns), we might still need to make and , although not or .

The principle is the same as on the previous step, although the index of Subgroup 2 in respect to its immediate supergroup, Subgroup 1, is the largest in this algorithm, and equals to 1 082 656. It means that if we take all states of the cube with correct orientations of the edges, and see which states are reachable from each other using only half-turns on U/D and F/R, you will find that there are more than a million disjoint sets of states.

In terms of padlock analogy, if you limit the turns to the last three dials out of five, you can never reach 12345 from 00000, but you can from 12000. You can see that there are 100 different sets of padlock states (corresponding to the 100 different combinations of the first two dials), which are not reachable from each other no matter what you do with the last three dials. Each of these sets is called a coset of the last three dials subgroup, and is identified by the first two digits. To get into the right coset, you will first need to identify the first two dials and lock them in place, otherwise you can turn the last three dials till your fingers get numb and still not get anywhere. The cube works in a similar way.

Let's remember the definition of the corner cubies' orientations. A corner cubie is oriented correctly if its L/R facet is aligned with the L/R faces of the cube (it can be on the same side or the opposite, it doesn't matter). We can see that L or R turns can't change corner cubies' orientations, because the facets which are currently on the L or R faces, stay there after any number of L or R turns. Half-turns of the other faces can't change corner orientations either, because all they can do is bring the L facet to the R side or vice versa. It means that if we start from the solved state and refrain from making quarter-turns on U/D or F/B, we cannot break corner orientations. So any cube state in the same coset as the solved state will have all the corners oriented correctly.

Now look at the edge cubies in the middle layer between the L and R facets (that is the cubies numbered from 8 through 11). Quarter-turns (or any other turns) on L and R don't affect them at all. Half-turns on other faces may move them, but they will stay in that middle layer. Using the same logic as before, Subgroup 2 moves starting from the solved state won't be able to kick the middle-layer edges out of the middle layer. It means that if we have a middle-layer edge outside the middle layer, we're in the wrong coset.

So here's the rule to tell if we're in the right coset of Subgroup 2: all corners are oriented correctly, and all middle-layer cubies (8, 9, 10, 11) are in the middle layer (on positions 8, 9, 10, 11). Since by this time we have all the edges oriented correctly and not making any quarter-turns on U and D, the edge orientation will remain correct, so we don't have to pay attention to it.

What happens with the cube state outside the aspect we're working with, does not interest us just yet. As on the previous pass, we can cover all the facets with painter's tape. Then, we would paint the centers of L and R facets purple, and so we would the L/R facets of all the corner cubies; all the middle-layer edge cubies we would paint pink. If we can get all the purple stickers on the faces opposite to each other, and all the pink stickers on the middle layer between the purple faces, we're done for this step.

The Subgroup 2 aspect of the cube would look like this:

We encode the part of cube's state that interests us in a 20-character string. The first 8 characters are the corner cubies' orientations (which can be 0, 1 or 3), the last 12 are the edge cubies. The solved state looks like this: '000000000000001111'. The 8 cubies outside the middle layer are indistinguishable from each other to us, so we mark all them with 0's, and so are those in the middle layer, which we mark with 1's. As soon as all the 1's are at the end of the string, we're good.

Here's the CTE to build the list of moves to get into the solution coset:

WITH    RECURSIVE
        basicMoves (initial, cornerMove, edgeMove, cornerTurn, edgeTurn) AS
        (
        VALUES
                (
                'U',
                ARRAY[ 4,  1,  2,  7,  3,  5,  6,  0]::INT[],  ARRAY[ 8,  1,  2, 11,  4,  5,  6,  7,  3,  9, 10,  0]::INT[],
                ARRAY[ 2,  0,  0,  2,  1,  0,  0,  1]::INT[],  ARRAY[ 1,  0,  0,  1,  0,  0,  0,  0,  1,  0,  0,  1]::INT[]
                ),
                (
                'D',
                ARRAY[ 0,  5,  6,  3,  4,  2,  1,  7]::INT[],  ARRAY[ 0, 10,  9,  3,  4,  5,  6,  7,  8,  1,  2, 11]::INT[],
                ARRAY[ 0,  2,  2,  0,  0,  1,  1,  0]::INT[],  ARRAY[ 0,  1,  1,  0,  0,  0,  0,  0,  0,  1,  1,  0]::INT[]
                ),
                (
                'F',
                ARRAY[ 0,  6,  2,  4,  1,  5,  3,  7]::INT[],  ARRAY[ 0,  1,  2,  3,  4,  9,  8,  7,  5,  6, 10, 11]::INT[],
                ARRAY[ 0,  1,  0,  1,  2,  0,  2,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'B',
                ARRAY[ 7,  1,  5,  3,  4,  0,  6,  2]::INT[],  ARRAY[ 0,  1,  2,  3, 11,  5,  6, 10,  8,  9,  4,  7]::INT[],
                ARRAY[ 1,  0,  1,  0,  0,  2,  0,  2]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'L',
                ARRAY[ 5,  4,  2,  3,  0,  1,  6,  7]::INT[],  ARRAY[ 4,  5,  2,  3,  1,  0,  6,  7,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'R',
                ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3]::INT[],  ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                )
        ),
        moves (turns, initial, move, cornerMove, edgeMove, cornerTurn, edgeTurn) AS MATERIALIZED
        (
        SELECT  1 AS turns, initial, bm.*
        FROM    basicMoves bm
        UNION ALL
        SELECT  turns + 1, move,
                CASE turns + 1 WHEN 2 THEN initial || '2' WHEN 3 THEN LOWER(initial) END,
                t.*
        FROM    moves am
        JOIN    basicMoves bm
        USING   (initial)
        CROSS JOIN LATERAL
                turn(am.cornerMove, am.edgeMove, am.cornerTurn, am.edgeTurn, bm.cornerMove, bm.edgeMove, bm.cornerTurn, bm.edgeTurn) t
        WHERE   turns <= 2
        ),
        g2s AS MATERIALIZED
        (
        SELECT  0 AS depth, 0 AS distance,
                '00000000000000001111' AS state,
                ARRAY[]::TEXT[] AS moves
        UNION ALL
        (
        WITH    q AS
                (
                SELECT  depth + 1 AS depth, distance, state, moves
                FROM    g2s
                )
        SELECT  *
        FROM    q
        WHERE   depth <= 10
        UNION ALL
        SELECT  DISTINCT ON (newState)
                depth, distance + 1, newState, moves || move
        FROM    q
        JOIN    moves
        ON      move NOT IN ('U', 'u', 'D', 'd')
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(((SUBSTRING(state FROM source + 1 FOR 1)::INT + cornerTurn[destination] + 3) % 3)::TEXT, '')
                FROM    UNNEST(cornerMove) WITH ORDINALITY AS move (source, destination)
                ) AS q1 (newCornerOrientation)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(SUBSTRING(state FROM source + 9 FOR 1), '')
                FROM    UNNEST(edgeMove) AS move (source)
                ) AS q2 (newMiddleEdgePosition)
        CROSS JOIN LATERAL
                (
                SELECT  newCornerOrientation || newMiddleEdgePosition
                ) AS q3 (newState)
        WHERE   distance = depth - 1
                AND depth <= 10
                AND newState NOT IN
                (
                SELECT  state
                FROM    q
                )
        )
        )
SELECT  state, moves
FROM    g2s
WHERE   depth = 10

The results of this query are trimmed for readability, but it returns 1 082 656 records (one per coset of Subgroup 2) and takes about 4 minutes to complete on my machine.

Subgroup 3

Let me quote another excerpt from Dr. Thistlethwaite's letter:

Getting from G2 into G3. This is the trickiest stage theoretically, and may be broken down for purposes of clarification into two sub-stages: first get corners into their natural orbits, and second permute the corners within their orbits so as to obtain one of the 96 corner permutations in the squares group (G3), while at the same time sorting out the edge pieces into their correct slices.

Boy is this stage tricky! Unlike the previous stages, which are easy-ish to understand, and straightforward to encode, this one is neither of these things.

Subgroup 3 is half-turns only. What exactly happens when we ban quarter-turns altogether?

First. On the previous pass, we fixed the L/R middle-layer edges in their own layer. We had left some quarter-turns allowed, so other the edge pieces could still move between then U/D layer and the F/R layer. Not anymore! By banning the quarter-turns, we fix all the edges in their respective layers.

Second. Let's look at the corner cubies. We can see that some of them are connected by face diagonals (like 1 and 2), the others by vertices (like 1 and 5), and still others by main diagonals (like 0 and 6). 0, 1, 2 and 3 are connected by diagonals, and so are 4, 5, 6 and 7. Every pair of corner cubies between these two sets are connected either by vertices, or by main diagonals. 0, 1, 2 and 3 form a tetrahedron, as well as their twins in the other set. If we look closer, we can see that half-turns can only move corner cubies through face diagonals, but not through vertices or main diagonals. It means that once a corner cubie gets into either of the two tetrahedra formed by the corners, it will stay there if only half-turns are allowed. These two tetrahedra are exactly what Dr. Thistlethwaite calls "natural orbits". We need to get corner cubies 1, 2, 3 and 4 into slots 1, 2, 3 and 4 (regardless of the order).

The third stage is the trickiest part of all the algorithm. Once cubies 1, 2, 3 and 4 get into their natural orbit, 4, 5, 6 and 7 will follow suit. There are 24 possible permutations of the cubies in the first orbit, and 24 in the second, for the total of 576 combinations of corner cubies in their natural orbits. However, only 96 of them, or one out of six, are in Subgroup 3.

If the cubies in the first orbit are on their own places: , then the second orbit can be any of:

, but not any other permutation. All other permutations are outside Subgroup 3 and cannot be solved by half-turns only. Every other permutation of orbit 1 also defines the four permutations on the orbit 2 that can go with it. Since only 4 out of 4! = 24 possible permutations are good, it means that there is an additional factor of 6 we need to account for.

It looks like this time a piece of painter's tape and a Sharpie ain't gonna cut it. In order to be using the same approach as before, we would need to identify this factor and make sure that it is calculatable from the state, persists during the turns and ends up in the lookup table.

This question has bothered minds brighter than mine. Here's a list (far from being complete) of resources I consulted while writing this post:

• What is the meaning of a "tetrad twist" in Thistlethwaite's algorithm?
• How can I generate a pruning table for tetrads?
• Generating lookup tables for Thistlethwaite's algorithm
• What is "tetrad twist" in relation to Rubik's cubes?
• Is it possible to calculate group 3's factor of 3 in Thistlethwaite algorithm?
• Thistlethwaite Algorithm

I spent a great deal of time figuring out how to implement a good mapping scheme that would, first, map all possible Subgroup 2 states into 29 600 cosets of Subgroup 3, and, second, would allow to make transformations of the image itself using just corner and edge permutation rules, to no avail. It looks like this is not a trivial task. One suggestion provided by Jaap in the answers would be "temporarily solve the corner cubies", which would mean making another graph search once per cube state. This, however, would most probably come at great computational price, which is, to be honest, not something SQL is good at.

One common way to deal with in is to extend the mapping, allowing multiple states from the same coset to map to different entries.

In Drew Finnis's solution, the test for Subgroup 3 requires all corners to be in their correct spots, which is a stronger condition than needed, so his pruning table for Subgroup 3 is almost 3 million records in size (because he tries to reach one particular corner permutation in Subgroup 3, while reaching any of the valid 96 would suffice).

Stefan Pochmann's entry for Tomas Rokicki's cube solvers contest also cuts some corners:

There's only one difference to the original algorithm, namely in phase 3 my subgoal is more restrictive. I could not find out a nice way to determine the "tetrad twists", instead I split up the tetrads in half to get the four groups UFR+UBL, DFL+DBR, URB+ULF and DRF+DLB. Getting the edges in their correct slices, the corners in their correct pair groups and ensuring overall even parity results in a cube solvable by phase 4.

Now that we're in good company, let's generate a larger than needed lookup table for Subgroup 3 as well.

In each orbit, there are six "twists", or classes of corner permutations. There are four possible permutations in each twist. For instance, one of the twists has the following permutations:

and another one has these:

We can identify the twist by the last three elements of the permutation with 0 in the first position. The first twist is "123", and the second one is "123".

To be in Subgroup 3, both orbits should have the same twist (modulo 4).

As an example, let's look at this combination of corner cubies:

The first orbit's permutation is , which is in the same twist as . So we say that the first orbit has twist "312".

The second orbit's permutation is , which is in the same twist as (or modulo 4). The two orbits have different twists, so this combination is not in Subgroup 3 and cannot be solved using only half-moves.

Let's look at another combination:

The first orbit's twist is "132", which the second one's is "576" ("132" modulo 4). The twists are the same, and the combination is in Subgroup 3.

We can notice that if we have a combination of moves which bring corners to the Subgroup 3, this combination will work on all permutations with the same twist. If some series of turns fixes corners on certain positions, we can put the corners or , or on the same positions, and it will fix them as well. It also works on the second orbit. In other words, if we find a series of turns to fix a certain position of corners, the same series will work on all 16 other positions with the same orbit twists.

We can come up with a canonical representation for all these 16 positions and use it in the lookup table. Here's how it works.

Let's say we have the corners arranged this way:

1. Split the corners by the orbit: . The first orbit is in boxes, the second in circles
2. The first orbit's corners, in the arrangement order, are . Permute them within the twist so that 0 is leading:
3. The second orbit's corners, in the arrangement order, are . Permute them within the twist so that 4 is leading:
4. Put them back into their places in the new order. First orbit's corners go into boxes, second orbit's into circles:
5. All sixteen permutations within two orbits' twists will be mapped into the same value. The value preserves both orbits' positions and twists, so we can calculate the moves right on the canonical value, and re-canonicalize it after the moves are applied

Let's create a helper function to canonicalize the corner positions. Again, this function is not procedural and only saves us some typing:

CREATE OR REPLACE FUNCTION twist(state TEXT)
RETURNS TEXT
AS
$$

WITH    perms (perm, i) AS
        (
        VALUES
        (ARRAY[0, 1, 2, 3], 1),
        (ARRAY[1, 0, 3, 2], 2),
        (ARRAY[2, 3, 0, 1], 3),
        (ARRAY[3, 2, 1, 0], 4)
        )
SELECT  targetState
FROM    (
        VALUES
                (
                ARRAY
                (
                SELECT  value::INT
                FROM    STRING_TO_TABLE(state, NULL) value
                WHERE   value::INT <= 3
                ),
                ARRAY
                (
                SELECT  value::INT
                FROM    STRING_TO_TABLE(state, NULL) value
                WHERE   value::INT > 3
                )
                )
        ) orbits (orbit1, orbit2)
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(orbit1[index + 1])
        FROM    perms
        CROSS JOIN LATERAL
                UNNEST(perm) index
        WHERE   i = ARRAY_POSITION(orbit1, 0)
        ) c1 (permuted1)
CROSS JOIN LATERAL
        (
        SELECT  ARRAY_AGG(orbit2[index + 1])
        FROM    perms
        CROSS JOIN LATERAL
                UNNEST(perm) index
        WHERE   i = ARRAY_POSITION(orbit2, 4)
        ) c2 (permuted2)
CROSS JOIN LATERAL
        (
        SELECT  STRING_AGG(corner::TEXT, NULL ORDER BY slot)
        FROM    (
                SELECT  corner, position
                FROM    UNNEST(permuted1) WITH ORDINALITY u (corner, position)
                UNION ALL
                SELECT  corner, position + 4
                FROM    UNNEST(permuted2) WITH ORDINALITY u (corner, position)
                ) corners (corner, cornerPosition)
        JOIN    (
                SELECT  position - 1 AS slot,
                        ROW_NUMBER() OVER (ORDER BY corner::INT / 4, position) AS cornerPosition
                FROM    STRING_TO_TABLE(state, NULL) WITH ORDINALITY st (corner, position)
                ) slots (slot, cornerPosition)
        USING   (cornerPosition)
        ) ts (targetState)

$$
LANGUAGE 'sql'
IMMUTABLE
STRICT

We can now use this function in a CTE to build the lookup table to get from Subgroup 2 to Subgroup 3. We don't account for the twist factor of 6, so we have 6 times as many states as we theoretically need. Unlike the other CTE, we initialize our table with 6 different initial positions. This won't help us keep it size down, but it will reduce the average number of moves.

After two or so minutes of heavy number crunching, we get the lookup table for Subgroup 3:

WITH    RECURSIVE
        basicMoves (initial, cornerMove, edgeMove, cornerTurn, edgeTurn) AS
        (
        VALUES
                (
                'U',
                ARRAY[ 4,  1,  2,  7,  3,  5,  6,  0]::INT[],  ARRAY[ 8,  1,  2, 11,  4,  5,  6,  7,  3,  9, 10,  0]::INT[],
                ARRAY[ 2,  0,  0,  2,  1,  0,  0,  1]::INT[],  ARRAY[ 1,  0,  0,  1,  0,  0,  0,  0,  1,  0,  0,  1]::INT[]
                ),
                (
                'D',
                ARRAY[ 0,  5,  6,  3,  4,  2,  1,  7]::INT[],  ARRAY[ 0, 10,  9,  3,  4,  5,  6,  7,  8,  1,  2, 11]::INT[],
                ARRAY[ 0,  2,  2,  0,  0,  1,  1,  0]::INT[],  ARRAY[ 0,  1,  1,  0,  0,  0,  0,  0,  0,  1,  1,  0]::INT[]
                ),
                (
                'F',
                ARRAY[ 0,  6,  2,  4,  1,  5,  3,  7]::INT[],  ARRAY[ 0,  1,  2,  3,  4,  9,  8,  7,  5,  6, 10, 11]::INT[],
                ARRAY[ 0,  1,  0,  1,  2,  0,  2,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'B',
                ARRAY[ 7,  1,  5,  3,  4,  0,  6,  2]::INT[],  ARRAY[ 0,  1,  2,  3, 11,  5,  6, 10,  8,  9,  4,  7]::INT[],
                ARRAY[ 1,  0,  1,  0,  0,  2,  0,  2]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'L',
                ARRAY[ 5,  4,  2,  3,  0,  1,  6,  7]::INT[],  ARRAY[ 4,  5,  2,  3,  1,  0,  6,  7,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'R',
                ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3]::INT[],  ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                )
        ),
        moves (turns, initial, move, cornerMove, edgeMove, cornerTurn, edgeTurn) AS MATERIALIZED
        (
        SELECT  1 AS turns, initial, bm.*
        FROM    basicMoves bm
        UNION ALL
        SELECT  turns + 1, move,
                CASE turns + 1 WHEN 2 THEN initial || '2' WHEN 3 THEN LOWER(initial) END,
                t.*
        FROM    moves am
        JOIN    basicMoves bm
        USING   (initial)
        CROSS JOIN LATERAL
                turn(am.cornerMove, am.edgeMove, am.cornerTurn, am.edgeTurn, bm.cornerMove, bm.edgeMove, bm.cornerTurn, bm.edgeTurn) t
        WHERE   turns <= 2
        ),
        g3s AS MATERIALIZED
        (
        SELECT  0 AS depth, 0 AS distance,
                corners || '000011112222' AS state,
                ARRAY[]::TEXT[] AS moves
        FROM    (
                VALUES
                ('01234567'),
                ('01324576'),
                ('02134657'),
                ('02314675'),
                ('03124756'),
                ('03214765')
                ) o2 (corners)
        UNION ALL
        (
        WITH    q AS
                (
                SELECT  depth + 1 AS depth, distance, state, moves
                FROM    g3s
                )
        SELECT  *
        FROM    q
        WHERE   depth <= 13
        UNION ALL
        SELECT  DISTINCT ON (newState)
                depth, distance + 1, newState, moves || move
        FROM    q
        JOIN    moves
        ON      move NOT IN ('U', 'u', 'D', 'd', 'F', 'f', 'B', 'b')
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(SUBSTRING(state FROM source + 1 FOR 1), '')
                FROM    UNNEST(cornerMove) AS source
                ) AS q1 (newCornerPosition)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(SUBSTRING(state FROM source + 9 FOR 1), '')
                FROM    UNNEST(edgeMove) AS source
                ) AS q2 (newEdgeLayerPosition)
        CROSS JOIN LATERAL
                twist(newCornerPosition) AS twistedCornerPosition
        CROSS JOIN LATERAL
                (
                SELECT  twistedCornerPosition || newEdgeLayerPosition
                ) q3 (newState)
        WHERE   distance = depth - 1
                AND depth <= 13
                AND newState NOT IN
                (
                SELECT  q.state
                FROM    q
                )
        )
        )
SELECT  state, moves
FROM    g3s
WHERE   depth = 13

Subgroup 4 (identity)

The final stage is very straightforward: get from Subgroup 3 to identity (a trivial subgroup of every group consisting only of the identity element). It corresponds to the solved state. The states are very simple to encode: there is no hashing or anything, every single bit or exact positions and orientations of all the cubies matters.

There are 600-something thousands elements in Subgroup 3, and this is what the size of the lookup table for the final stage will be.

Here's the CTE:

WITH    RECURSIVE
        basicMoves (initial, cornerMove, edgeMove, cornerTurn, edgeTurn) AS
        (
        VALUES
                (
                'U',
                ARRAY[ 4,  1,  2,  7,  3,  5,  6,  0]::INT[],  ARRAY[ 8,  1,  2, 11,  4,  5,  6,  7,  3,  9, 10,  0]::INT[],
                ARRAY[ 2,  0,  0,  2,  1,  0,  0,  1]::INT[],  ARRAY[ 1,  0,  0,  1,  0,  0,  0,  0,  1,  0,  0,  1]::INT[]
                ),
                (
                'D',
                ARRAY[ 0,  5,  6,  3,  4,  2,  1,  7]::INT[],  ARRAY[ 0, 10,  9,  3,  4,  5,  6,  7,  8,  1,  2, 11]::INT[],
                ARRAY[ 0,  2,  2,  0,  0,  1,  1,  0]::INT[],  ARRAY[ 0,  1,  1,  0,  0,  0,  0,  0,  0,  1,  1,  0]::INT[]
                ),
                (
                'F',
                ARRAY[ 0,  6,  2,  4,  1,  5,  3,  7]::INT[],  ARRAY[ 0,  1,  2,  3,  4,  9,  8,  7,  5,  6, 10, 11]::INT[],
                ARRAY[ 0,  1,  0,  1,  2,  0,  2,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'B',
                ARRAY[ 7,  1,  5,  3,  4,  0,  6,  2]::INT[],  ARRAY[ 0,  1,  2,  3, 11,  5,  6, 10,  8,  9,  4,  7]::INT[],
                ARRAY[ 1,  0,  1,  0,  0,  2,  0,  2]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'L',
                ARRAY[ 5,  4,  2,  3,  0,  1,  6,  7]::INT[],  ARRAY[ 4,  5,  2,  3,  1,  0,  6,  7,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'R',
                ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3]::INT[],  ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                )
        ),
        moves (turns, initial, move, cornerMove, edgeMove, cornerTurn, edgeTurn) AS MATERIALIZED
        (
        SELECT  1 AS turns, initial, bm.*
        FROM    basicMoves bm
        UNION ALL
        SELECT  turns + 1, move,
                CASE turns + 1 WHEN 2 THEN initial || '2' WHEN 3 THEN LOWER(initial) END,
                t.*
        FROM    moves am
        JOIN    basicMoves bm
        USING   (initial)
        CROSS JOIN LATERAL
                turn(am.cornerMove, am.edgeMove, am.cornerTurn, am.edgeTurn, bm.cornerMove, bm.edgeMove, bm.cornerTurn, bm.edgeTurn) t
        WHERE   turns <= 2
        ),
        g4s (depth, distance, state, moves) AS MATERIALIZED
        (
        SELECT  0, 0, '012345670123456789AB', ARRAY[]::TEXT[]
        UNION ALL
        (
        WITH    q AS
                (
                SELECT  depth + 1 AS depth, distance, state, moves
                FROM    g4s
                )
        SELECT  *
        FROM    q
        WHERE   depth <= 15
        UNION ALL
        SELECT  DISTINCT ON (newState)
                depth, distance + 1, newState, moves || move
        FROM    q
        JOIN    moves
        ON      turns = 2
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(SUBSTRING(state FROM source + 1 FOR 1), '')
                FROM    UNNEST(cornerMove) AS source
                ) AS q1 (newCornerPositions)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(SUBSTRING(state FROM source + 9 FOR 1), '')
                FROM    UNNEST(edgeMove) AS source
                ) AS q2 (newEdgePositions)
        CROSS JOIN LATERAL
                (
                SELECT  newCornerPositions || newEdgePositions
                ) AS q3 (newState)
        WHERE   distance = depth - 1
                AND depth <= 15
                AND newState NOT IN
                (
                SELECT  q.state
                FROM    q
                )
        )
        )
SELECT  state, moves
FROM    g4s
WHERE   depth = 15

Solving the cube

Now, let's put all the pieces of the solution together.

Here is what we are going to do:

• Take a series of moves to shuffle the cube
• By sequentially applying the moves, put the cube into the shuffled state
• Encode the cube's state the way the Subgroup 1 lookup CTE expects it to be and look it up in the materialized output of the Subgroup 1's CTE
• Inverse the moves from the lookup table (both the order of turns and their direction) to get to Subgroup 2 state
• Repeat the steps for each of the next subgroups. Unlike the padlock puzzle, cube's subgroups are not normal, meaning that the order in which we move to each subgroup matters. Had they been they normal, we could have encoded the source state, look it up in all the four CTE's with simple joins and combine the moves in any order. Instead, we will need to do this sequentially and only in correct order (Subgroup 1 first, then Subgroup 2 etc.)
• Concatenate the series of reversed moves from each state, and we're done!

Note that every time we run this query, the lookup tables are being regenerated from scratch (which is only fair, considering that we want to do this in a single query with no saved data whatsoever). It takes some time (about 7 minutes on my machine).

Here's a series of moves to shuffle the cube:

Let's try to solve it in SQL! That's what we get:

WITH    RECURSIVE
        basicMoves (initial, cornerMove, edgeMove, cornerTurn, edgeTurn) AS
        (
        VALUES
                (
                'U',
                ARRAY[ 4,  1,  2,  7,  3,  5,  6,  0]::INT[],  ARRAY[ 8,  1,  2, 11,  4,  5,  6,  7,  3,  9, 10,  0]::INT[],
                ARRAY[ 2,  0,  0,  2,  1,  0,  0,  1]::INT[],  ARRAY[ 1,  0,  0,  1,  0,  0,  0,  0,  1,  0,  0,  1]::INT[]
                ),
                (
                'D',
                ARRAY[ 0,  5,  6,  3,  4,  2,  1,  7]::INT[],  ARRAY[ 0, 10,  9,  3,  4,  5,  6,  7,  8,  1,  2, 11]::INT[],
                ARRAY[ 0,  2,  2,  0,  0,  1,  1,  0]::INT[],  ARRAY[ 0,  1,  1,  0,  0,  0,  0,  0,  0,  1,  1,  0]::INT[]
                ),
                (
                'F',
                ARRAY[ 0,  6,  2,  4,  1,  5,  3,  7]::INT[],  ARRAY[ 0,  1,  2,  3,  4,  9,  8,  7,  5,  6, 10, 11]::INT[],
                ARRAY[ 0,  1,  0,  1,  2,  0,  2,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'B',
                ARRAY[ 7,  1,  5,  3,  4,  0,  6,  2]::INT[],  ARRAY[ 0,  1,  2,  3, 11,  5,  6, 10,  8,  9,  4,  7]::INT[],
                ARRAY[ 1,  0,  1,  0,  0,  2,  0,  2]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'L',
                ARRAY[ 5,  4,  2,  3,  0,  1,  6,  7]::INT[],  ARRAY[ 4,  5,  2,  3,  1,  0,  6,  7,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'R',
                ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3]::INT[],  ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                )
        ),
        moves (turns, initial, move, reverse, cornerMove, edgeMove, cornerTurn, edgeTurn) AS MATERIALIZED
        (
        SELECT  1 AS turns, initial, initial, LOWER(initial),
                cornerMove, edgeMove, cornerTurn, edgeTurn
        FROM    basicMoves bm
        UNION ALL
        SELECT  turns + 1, initial,
                CASE turns WHEN 1 THEN initial || '2' WHEN 2 THEN LOWER(initial) END,
                CASE turns WHEN 1 THEN initial || '2' WHEN 2 THEN initial END,
                t.*
        FROM    moves am
        JOIN    basicMoves bm
        USING   (initial)
        CROSS JOIN LATERAL
                turn(am.cornerMove, am.edgeMove, am.cornerTurn, am.edgeTurn, bm.cornerMove, bm.edgeMove, bm.cornerTurn, bm.edgeTurn) t
        WHERE   turns <= 2
        ),
        solved (cornerPosition, edgePosition, cornerOrientation, edgeOrientation) AS MATERIALIZED
        (
        VALUES
                (
                ARRAY(SELECT v FROM GENERATE_SERIES(0, 7) v), ARRAY(SELECT v FROM GENERATE_SERIES(0, 11) v),
                ARRAY(SELECT 0 FROM GENERATE_SERIES(0, 7)), ARRAY(SELECT 0 FROM GENERATE_SERIES(0, 11))
                )
        ),
        sequence (turns) AS
        (
        VALUES
        ('L2 d U F2 U F2 D B f D f l B2 f R2 F2 r F D2 l U L R2 f D2 F d')
        ),
        turns (move, i) AS MATERIALIZED
        (
        SELECT  turn.*
        FROM    sequence
        CROSS JOIN LATERAL
                STRING_TO_TABLE(turns, ' ') WITH ORDINALITY AS turn (move, i)
        ),
        scramble (cornerPosition, edgePosition, cornerOrientation, edgeOrientation, currentMove, i) AS MATERIALIZED
        (
        SELECT  cornerPosition, edgePosition, cornerOrientation, edgeOrientation,
                NULL,
                1::BIGINT AS i
        FROM    solved
        UNION ALL
        SELECT  newValues.*,
                move,
                turns.i + 1
        FROM    scramble e
        JOIN    turns
        USING   (i)
        JOIN    moves
        USING   (move)
        CROSS JOIN LATERAL
                turn(cornerPosition, edgePosition, cornerOrientation, edgeOrientation, cornerMove, edgeMove, cornerTurn, edgeTurn) AS newValues
        ),
        unsolved AS MATERIALIZED
        (
        SELECT  cornerPosition, edgePosition, cornerOrientation, edgeOrientation
        FROM    scramble
        ORDER BY
                i DESC
        LIMIT   1
        ),
        g1s AS MATERIALIZED
        (
        SELECT  0 AS depth, 0 AS distance,
                '000000000000' AS state,
                ARRAY[]::TEXT[] AS moves
        UNION ALL
        (
        WITH    q AS
                (
                SELECT  depth + 1 AS depth, distance, state, moves
                FROM    g1s
                )
        SELECT  *
        FROM    q
        WHERE   depth <= 7
        UNION ALL
        SELECT  DISTINCT ON (newState)
                depth, distance + 1, newState, moves || move
        FROM    q
        CROSS JOIN
                moves
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(((SUBSTRING(state FROM source + 1 FOR 1)::INT + edgeTurn[destination] + 2) % 2)::TEXT, '')
                FROM    UNNEST(edgeMove) WITH ORDINALITY AS move (source, destination)
                ) AS state (newState)
        WHERE   distance = depth - 1
                AND newState NOT IN
                (
                SELECT  state
                FROM    q q2
                )
                AND depth <= 7
        )
        ),
        g2s AS MATERIALIZED
        (
        SELECT  0 AS depth, 0 AS distance,
                '00000000000000001111' AS state,
                ARRAY[]::TEXT[] AS moves
        UNION ALL
        (
        WITH    q AS
                (
                SELECT  depth + 1 AS depth, distance, state, moves
                FROM    g2s
                )
        SELECT  *
        FROM    q
        WHERE   depth <= 10
        UNION ALL
        SELECT  DISTINCT ON (newState)
                depth, distance + 1, newState, moves || move
        FROM    q
        JOIN    moves
        ON      move NOT IN ('U', 'u', 'D', 'd')
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(((SUBSTRING(state FROM source + 1 FOR 1)::INT + cornerTurn[destination] + 3) % 3)::TEXT, '')
                FROM    UNNEST(cornerMove) WITH ORDINALITY AS move (source, destination)
                ) AS q1 (newCornerOrientation)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(SUBSTRING(state FROM source + 9 FOR 1), '')
                FROM    UNNEST(edgeMove) AS move (source)
                ) AS q2 (newMiddleEdgePosition)
        CROSS JOIN LATERAL
                (
                SELECT  newCornerOrientation || newMiddleEdgePosition
                ) AS q3 (newState)
        WHERE   distance = depth - 1
                AND depth <= 10
                AND newState NOT IN
                (
                SELECT  state
                FROM    q
                )
        )
        ),
        g3s AS MATERIALIZED
        (
        SELECT  0 AS depth, 0 AS distance,
                corners || '000011112222' AS state,
                ARRAY[]::TEXT[] AS moves
        FROM    (
                VALUES
                ('01234567'),
                ('01324576'),
                ('02134657'),
                ('02314675'),
                ('03124756'),
                ('03214765')
                ) o2 (corners)
        UNION ALL
        (
        WITH    q AS
                (
                SELECT  depth + 1 AS depth, distance, state, moves
                FROM    g3s
                )
        SELECT  *
        FROM    q
        WHERE   depth <= 13
        UNION ALL
        SELECT  DISTINCT ON (newState)
                depth, distance + 1, newState, moves || move
        FROM    q
        JOIN    moves
        ON      move NOT IN ('U', 'u', 'D', 'd', 'F', 'f', 'B', 'b')
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(SUBSTRING(state FROM source + 1 FOR 1), '')
                FROM    UNNEST(cornerMove) AS source
                ) AS q1 (newCornerPosition)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(SUBSTRING(state FROM source + 9 FOR 1), '')
                FROM    UNNEST(edgeMove) AS source
                ) AS q2 (newEdgeLayerPosition)
        CROSS JOIN LATERAL
                twist(newCornerPosition) AS twistedCornerPosition
        CROSS JOIN LATERAL
                (
                SELECT  twistedCornerPosition || newEdgeLayerPosition
                ) q3 (newState)
        WHERE   distance = depth - 1
                AND depth <= 13
                AND newState NOT IN
                (
                SELECT  q.state
                FROM    q
                )
        )
        ),
        g4s (depth, distance, state, moves) AS MATERIALIZED
        (
        SELECT  0, 0, '012345670123456789AB', ARRAY[]::TEXT[]
        UNION ALL
        (
        WITH    q AS
                (
                SELECT  depth + 1 AS depth, distance, state, moves
                FROM    g4s
                )
        SELECT  *
        FROM    q
        WHERE   depth <= 15
        UNION ALL
        SELECT  DISTINCT ON (newState)
                depth, distance + 1, newState, moves || move
        FROM    q
        JOIN    moves
        ON      turns = 2
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(SUBSTRING(state FROM source + 1 FOR 1), '')
                FROM    UNNEST(cornerMove) AS source
                ) AS q1 (newCornerPositions)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(SUBSTRING(state FROM source + 9 FOR 1), '')
                FROM    UNNEST(edgeMove) AS source
                ) AS q2 (newEdgePositions)
        CROSS JOIN LATERAL
                (
                SELECT  newCornerPositions || newEdgePositions
                ) AS q3 (newState)
        WHERE   distance = depth - 1
                AND depth <= 15
                AND newState NOT IN
                (
                SELECT  q.state
                FROM    q
                )
        )
        ),
        g1 AS MATERIALIZED
        (
        SELECT  state, moves
        FROM    g1s
        WHERE   depth = 7
        ),
        g2 AS MATERIALIZED
        (
        SELECT  state, moves
        FROM    g2s
        WHERE   depth = 10
        ),
        g3 AS MATERIALIZED
        (
        SELECT  state, moves
        FROM    g3s
        WHERE   depth = 13
        ),
        g4 AS MATERIALIZED
        (
        SELECT  state, moves
        FROM    g4s
        WHERE   depth = 15
        ),
        g1Entry AS
        (
        SELECT  g1.state, moves
        FROM    (
                SELECT  STRING_AGG(edge::TEXT, '') AS state
                FROM    unsolved
                CROSS JOIN LATERAL
                        UNNEST(edgeOrientation) edge
                ) q
        JOIN    g1
        USING   (state)
        ),
        g1Moves AS MATERIALIZED
        (
        SELECT  u.*, moves, NULL::TEXT AS move
        FROM    unsolved u
        CROSS JOIN
                g1Entry
        UNION ALL
        SELECT  newValues.*, moves[:(ARRAY_LENGTH(moves, 1) - 1)], moves.move
        FROM    g1Moves
        JOIN    moves
        ON      reverse = moves[ARRAY_LENGTH(moves, 1)]
        CROSS JOIN LATERAL
                turn(cornerPosition, edgePosition, cornerOrientation, edgeOrientation, cornerMove, edgeMove, cornerTurn, edgeTurn) AS newValues
        ),
        g2Entry AS
        (
        SELECT  g1Moves.cornerPosition, g1Moves.edgePosition, g1Moves.cornerOrientation, g1Moves.edgeOrientation,
                g2.moves
        FROM    (
                SELECT  *
                FROM    g1Moves
                WHERE   moves = ARRAY[]::TEXT[]
                ) g1Moves
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(value::TEXT, '')
                FROM    UNNEST(cornerOrientation) AS value
                ) AS q (corners)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG((value >= 8)::INT::TEXT, '')
                FROM    UNNEST(edgePosition) AS value
                ) AS q2 (middleEdges)
        CROSS JOIN LATERAL
                (
                SELECT  corners || middleEdges
                ) AS q3 (state)
        JOIN    g2
        USING   (state)
        ),
        g2Moves AS MATERIALIZED
        (
        SELECT  *, NULL::TEXT AS move
        FROM    g2Entry
        UNION ALL
        SELECT  newValues.*, moves[:(ARRAY_LENGTH(moves, 1) - 1)], moves.move
        FROM    g2Moves
        JOIN    moves
        ON      reverse = moves[ARRAY_LENGTH(moves, 1)]
        CROSS JOIN LATERAL
                turn(cornerPosition, edgePosition, cornerOrientation, edgeOrientation, cornerMove, edgeMove, cornerTurn, edgeTurn) AS newValues
        ),
        g3Entry AS
        (
        SELECT  g2Last.cornerPosition, g2Last.edgePosition, g2Last.cornerOrientation, g2Last.edgeOrientation,
                g3.moves
        FROM    (
                SELECT  *
                FROM    g2Moves
                WHERE   moves = ARRAY[]::TEXT[]
                ) g2Last
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(value::TEXT, '')
                FROM    UNNEST(cornerPosition) AS value
                ) AS q (corners)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG((value / 4)::INT::TEXT, '')
                FROM    UNNEST(edgePosition) AS value
                ) AS q2 (layerEdges)
        CROSS JOIN LATERAL
                twist(corners) AS twistedCorners
        CROSS JOIN LATERAL
                (
                SELECT  twistedCorners || layerEdges
                ) AS q3 (state)
        JOIN    g3
        USING   (state)
        ),
        g3Moves AS MATERIALIZED
        (
        SELECT  *, NULL::TEXT AS move
        FROM    g3Entry
        UNION ALL
        SELECT  newValues.*, moves[:(ARRAY_LENGTH(moves, 1) - 1)], moves.move
        FROM    g3Moves
        JOIN    moves
        ON      reverse = moves[ARRAY_LENGTH(moves, 1)]
        CROSS JOIN LATERAL
                turn(cornerPosition, edgePosition, cornerOrientation, edgeOrientation, cornerMove, edgeMove, cornerTurn, edgeTurn) AS newValues
        ),
        g4Entry AS
        (
        SELECT  g3Last.cornerPosition, g3Last.edgePosition, g3Last.cornerOrientation, g3Last.edgeOrientation,
                g4.moves
        FROM    (
                SELECT  *
                FROM    g3Moves
                WHERE   moves = ARRAY[]::TEXT[]
                ) g3Last
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(value::TEXT, '')
                FROM    UNNEST(cornerPosition) AS value
                ) AS q (corners)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(UPPER(TO_HEX(value::INT)), '')
                FROM    UNNEST(edgePosition) AS value
                ) AS q2 (edges)
        CROSS JOIN LATERAL
                (
                SELECT  corners || edges
                ) AS q3 (state)
        JOIN    g4
        USING   (state)
        ),
        g4Moves AS MATERIALIZED
        (
        SELECT  *, NULL::TEXT AS move
        FROM    g4Entry
        UNION ALL
        SELECT  newValues.*, moves[:(ARRAY_LENGTH(moves, 1) - 1)], moves.move
        FROM    g4Moves
        JOIN    moves
        ON      reverse = moves[ARRAY_LENGTH(moves, 1)]
        CROSS JOIN LATERAL
                turn(cornerPosition, edgePosition, cornerOrientation, edgeOrientation, cornerMove, edgeMove, cornerTurn, edgeTurn) AS newValues
        )
SELECT  STRING_AGG(move, ' ') AS result
FROM    (
        SELECT  *
        FROM    g1Moves
        UNION ALL
        SELECT  *
        FROM    g2Moves
        UNION ALL
        SELECT  *
        FROM    g3Moves
        UNION ALL
        SELECT  *
        FROM    g4Moves
        ) q
WHERE   move <> ''

The cube is solved!

This query is very long and boring to play with. This post's GitHub repository has a script to create and store lookup tables in the database. With these tables, solving the cubes becomes instant, less than 50 milliseconds on my machine.

You can use the stored tables if you want to play with the query:

WITH    RECURSIVE
        basicMoves (initial, cornerMove, edgeMove, cornerTurn, edgeTurn) AS
        (
        VALUES
                (
                'U',
                ARRAY[ 4,  1,  2,  7,  3,  5,  6,  0]::INT[],  ARRAY[ 8,  1,  2, 11,  4,  5,  6,  7,  3,  9, 10,  0]::INT[],
                ARRAY[ 2,  0,  0,  2,  1,  0,  0,  1]::INT[],  ARRAY[ 1,  0,  0,  1,  0,  0,  0,  0,  1,  0,  0,  1]::INT[]
                ),
                (
                'D',
                ARRAY[ 0,  5,  6,  3,  4,  2,  1,  7]::INT[],  ARRAY[ 0, 10,  9,  3,  4,  5,  6,  7,  8,  1,  2, 11]::INT[],
                ARRAY[ 0,  2,  2,  0,  0,  1,  1,  0]::INT[],  ARRAY[ 0,  1,  1,  0,  0,  0,  0,  0,  0,  1,  1,  0]::INT[]
                ),
                (
                'F',
                ARRAY[ 0,  6,  2,  4,  1,  5,  3,  7]::INT[],  ARRAY[ 0,  1,  2,  3,  4,  9,  8,  7,  5,  6, 10, 11]::INT[],
                ARRAY[ 0,  1,  0,  1,  2,  0,  2,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'B',
                ARRAY[ 7,  1,  5,  3,  4,  0,  6,  2]::INT[],  ARRAY[ 0,  1,  2,  3, 11,  5,  6, 10,  8,  9,  4,  7]::INT[],
                ARRAY[ 1,  0,  1,  0,  0,  2,  0,  2]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'L',
                ARRAY[ 5,  4,  2,  3,  0,  1,  6,  7]::INT[],  ARRAY[ 4,  5,  2,  3,  1,  0,  6,  7,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                ),
                (
                'R',
                ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3]::INT[],  ARRAY[ 0,  1,  7,  6,  4,  5,  2,  3,  8,  9, 10, 11]::INT[],
                ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0]::INT[],  ARRAY[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0]::INT[]
                )
        ),
        moves (turns, initial, move, reverse, cornerMove, edgeMove, cornerTurn, edgeTurn) AS MATERIALIZED
        (
        SELECT  1 AS turns, initial, initial, LOWER(initial),
                cornerMove, edgeMove, cornerTurn, edgeTurn
        FROM    basicMoves bm
        UNION ALL
        SELECT  turns + 1, initial,
                CASE turns WHEN 1 THEN initial || '2' WHEN 2 THEN LOWER(initial) END,
                CASE turns WHEN 1 THEN initial || '2' WHEN 2 THEN initial END,
                t.*
        FROM    moves am
        JOIN    basicMoves bm
        USING   (initial)
        CROSS JOIN LATERAL
                turn(am.cornerMove, am.edgeMove, am.cornerTurn, am.edgeTurn, bm.cornerMove, bm.edgeMove, bm.cornerTurn, bm.edgeTurn) t
        WHERE   turns <= 2
        ),
        solved (cornerPosition, edgePosition, cornerOrientation, edgeOrientation) AS MATERIALIZED
        (
        VALUES
                (
                ARRAY(SELECT v FROM GENERATE_SERIES(0, 7) v), ARRAY(SELECT v FROM GENERATE_SERIES(0, 11) v),
                ARRAY(SELECT 0 FROM GENERATE_SERIES(0, 7)), ARRAY(SELECT 0 FROM GENERATE_SERIES(0, 11))
                )
        ),
        sequence (turns) AS
        (
        VALUES
        ('L2 d U F2 U F2 D B f D f l B2 f R2 F2 r F D2 l U L R2 f D2 F d')
        ),
        turns (move, i) AS MATERIALIZED
        (
        SELECT  turn.*
        FROM    sequence
        CROSS JOIN LATERAL
                STRING_TO_TABLE(turns, ' ') WITH ORDINALITY AS turn (move, i)
        ),
        scramble (cornerPosition, edgePosition, cornerOrientation, edgeOrientation, currentMove, i) AS MATERIALIZED
        (
        SELECT  cornerPosition, edgePosition, cornerOrientation, edgeOrientation,
                NULL,
                1::BIGINT AS i
        FROM    solved
        UNION ALL
        SELECT  newValues.*,
                move,
                turns.i + 1
        FROM    scramble e
        JOIN    turns
        USING   (i)
        JOIN    moves
        USING   (move)
        CROSS JOIN LATERAL
                turn(cornerPosition, edgePosition, cornerOrientation, edgeOrientation, cornerMove, edgeMove, cornerTurn, edgeTurn) AS newValues
        ),
        unsolved AS MATERIALIZED
        (
        SELECT  cornerPosition, edgePosition, cornerOrientation, edgeOrientation
        FROM    scramble
        ORDER BY
                i DESC
        LIMIT   1
        ),
        g1Entry AS
        (
        SELECT  g1.state, moves
        FROM    (
                SELECT  STRING_AGG(edge::TEXT, '') AS state
                FROM    unsolved
                CROSS JOIN LATERAL
                        UNNEST(edgeOrientation) edge
                ) q
        JOIN    g1stored g1
        USING   (state)
        ),
        g1Moves AS MATERIALIZED
        (
        SELECT  u.*, moves, NULL::TEXT AS move
        FROM    unsolved u
        CROSS JOIN
                g1Entry
        UNION ALL
        SELECT  newValues.*, moves[:(ARRAY_LENGTH(moves, 1) - 1)], moves.move
        FROM    g1Moves
        JOIN    moves
        ON      reverse = moves[ARRAY_LENGTH(moves, 1)]
        CROSS JOIN LATERAL
                turn(cornerPosition, edgePosition, cornerOrientation, edgeOrientation, cornerMove, edgeMove, cornerTurn, edgeTurn) AS newValues
        ),
        g2Entry AS
        (
        SELECT  g1Moves.cornerPosition, g1Moves.edgePosition, g1Moves.cornerOrientation, g1Moves.edgeOrientation,
                g2.moves
        FROM    (
                SELECT  *
                FROM    g1Moves
                WHERE   moves = ARRAY[]::TEXT[]
                ) g1Moves
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(value::TEXT, '')
                FROM    UNNEST(cornerOrientation) AS value
                ) AS q (corners)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG((value >= 8)::INT::TEXT, '')
                FROM    UNNEST(edgePosition) AS value
                ) AS q2 (middleEdges)
        CROSS JOIN LATERAL
                (
                SELECT  corners || middleEdges
                ) AS q3 (state)
        JOIN    g2stored g2
        USING   (state)
        ),
        g2Moves AS MATERIALIZED
        (
        SELECT  *, NULL::TEXT AS move
        FROM    g2Entry
        UNION ALL
        SELECT  newValues.*, moves[:(ARRAY_LENGTH(moves, 1) - 1)], moves.move
        FROM    g2Moves
        JOIN    moves
        ON      reverse = moves[ARRAY_LENGTH(moves, 1)]
        CROSS JOIN LATERAL
                turn(cornerPosition, edgePosition, cornerOrientation, edgeOrientation, cornerMove, edgeMove, cornerTurn, edgeTurn) AS newValues
        ),
        g3Entry AS
        (
        SELECT  g2Last.cornerPosition, g2Last.edgePosition, g2Last.cornerOrientation, g2Last.edgeOrientation,
                g3.moves
        FROM    (
                SELECT  *
                FROM    g2Moves
                WHERE   moves = ARRAY[]::TEXT[]
                ) g2Last
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(value::TEXT, '')
                FROM    UNNEST(cornerPosition) AS value
                ) AS q (corners)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG((value / 4)::INT::TEXT, '')
                FROM    UNNEST(edgePosition) AS value
                ) AS q2 (layerEdges)
        CROSS JOIN LATERAL
                twist(corners) AS twistedCorners
        CROSS JOIN LATERAL
                (
                SELECT  twistedCorners || layerEdges
                ) AS q3 (state)
        JOIN    g3stored g3
        USING   (state)
        ),
        g3Moves AS MATERIALIZED
        (
        SELECT  *, NULL::TEXT AS move
        FROM    g3Entry
        UNION ALL
        SELECT  newValues.*, moves[:(ARRAY_LENGTH(moves, 1) - 1)], moves.move
        FROM    g3Moves
        JOIN    moves
        ON      reverse = moves[ARRAY_LENGTH(moves, 1)]
        CROSS JOIN LATERAL
                turn(cornerPosition, edgePosition, cornerOrientation, edgeOrientation, cornerMove, edgeMove, cornerTurn, edgeTurn) AS newValues
        ),
        g4Entry AS
        (
        SELECT  g3Last.cornerPosition, g3Last.edgePosition, g3Last.cornerOrientation, g3Last.edgeOrientation,
                g4.moves
        FROM    (
                SELECT  *
                FROM    g3Moves
                WHERE   moves = ARRAY[]::TEXT[]
                ) g3Last
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(value::TEXT, '')
                FROM    UNNEST(cornerPosition) AS value
                ) AS q (corners)
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(UPPER(TO_HEX(value::INT)), '')
                FROM    UNNEST(edgePosition) AS value
                ) AS q2 (edges)
        CROSS JOIN LATERAL
                (
                SELECT  corners || edges
                ) AS q3 (state)
        JOIN    g4stored g4
        USING   (state)
        ),
        g4Moves AS MATERIALIZED
        (
        SELECT  *, NULL::TEXT AS move
        FROM    g4Entry
        UNION ALL
        SELECT  newValues.*, moves[:(ARRAY_LENGTH(moves, 1) - 1)], moves.move
        FROM    g4Moves
        JOIN    moves
        ON      reverse = moves[ARRAY_LENGTH(moves, 1)]
        CROSS JOIN LATERAL
                turn(cornerPosition, edgePosition, cornerOrientation, edgeOrientation, cornerMove, edgeMove, cornerTurn, edgeTurn) AS newValues
        )
SELECT  STRING_AGG(move, ' ') AS result
FROM    (
        SELECT  *
        FROM    g1Moves
        UNION ALL
        SELECT  *
        FROM    g2Moves
        UNION ALL
        SELECT  *
        FROM    g3Moves
        UNION ALL
        SELECT  *
        FROM    g4Moves
        ) q
WHERE   move <> ''

You can view the queries here: https://github.com/quassnoi/explain-extended-2023

In this New Year, I wish you a large subgroup of good people. May your moves be smooth, colors bright and turns expected.

Previous New Year posts:

• 2010: SQL graphics in Oracle, MySQL, SQL Server and PostgreSQL
• 2011: Drawing a clock in SQL
• 2012: Drawing snowflakes in SQL
• 2013: View of Earth from space in SQL
• 2014: Drawing fractals in SQL
• 2015: Composing music in SQL
• 2016: Conway’s Game of Life in SQL
• 2017: The Sultan’s Riddle in SQL
• 2018: Settlers of Catan in SQL
• 2019: GIF decoder in SQL
• 2020: A stereogram in SQL
• 2021: 3D picture of the coronavirus in SQL
• 2022: Quantum computer emulator in SQL

The post Happy New Year: solving the Rubik’s Cube in SQL appeared first on EXPLAIN EXTENDED.

For those who have been living under a rock for the past few weeks, Wordle is a relatively new online word game which has become viral. It is a variation of Bulls and Cows.

You have six tries to guess a five-letter word. Each try reveals letters that the target word has. If your guess has the same letter on the same spot as the target word, the letter is colored green. If the letter is there, but on the wrong spot, it's colored yellow. Finally, if the letter is not in the target word at all, it's colored black. There is a little twist when it comes to the words with repeating letters. If your guess has a repeating letter, it will be colored green or yellow, but only as many times as the target word has it (more about it later).

You can only use dictionary words to make your guesses, the game won't allow you to post garbage and reveal positions of the letters.

Every new try reveals some information about the target word. But when the playfield is empty, we don't have any information at all. We need a good word to start the game of Wordle. And, since the words are (supposedly) chosen randomly, and at the beginning of the game we are absolutely blind, it makes sense to use the same word to start every new game.

Let's use SQL to find this word!

The word list

I don't know what word list Wordle is using, but it's probably the same one that is being used by different variations of Scrabble, or close enough to it. I used SOWPODS which I found on Github: https://github.com/jesstess/Scrabble/blob/master/scrabble/sowpods.txt. This list contains exactly 12478 five-letter words, from "aahed" (which means "made an aah sound") to "zymic" (which means "pertaining to fermentation"). I uploaded them all into a PostgreSQL table.

The best worst case

Every time we make a move, the game reveals some information about the target word.

How much information does it reveal? It depends on the answer. If we entered the word and guessed four letters correctly, we would be happy: just one letter to go! If we guessed none of the letters right, we would probably not be so happy, although we did learn something new about the target word (namely, that it has none of the letters we used). Somehow, it feels that guessing letters right gives us more information about the words than not guessing them.

This feeling is (mostly) right, and it comes with a measure. There were 12478 potential candidates for the first in the list before our first guess. After our first guess, there will be fewer candidates.

If we entered the word RUBUS and got GBGGG (green, black, green, green, green) back, we know that the answer is REBUS: there is just one single word compatible with this coloring in the word list.

If we entered XYLAN, and got BBBBB back (no letters), then we would know that TUXES, or YAWED, or many other words could not be the answers (TUXES has "X" and YAWED has "Y"). But BIJOU, or FORTH, or PROBE, or yet many other words could, because they don't have any letters in common with XYLAN indeed. In fact, for the word XYLAN, there is 1 word compatible with the answer BYGGG, and 3303 words compatible with the answer BBBBB. This means that BYGGG gives us a lot more information about the answer than BBBBB does.

It would be nice to always get a lot of green letters in the answer, but the words are chosen randomly. This means that for the word XYLAN, the answer BBBBB narrows the pool of potential candidates down to 26% of the original pool. If we do the math for the other color combinations, we'll see that BBBYB narrows it down to 2049 potential candidates, BBBBY to 972 and so on. This also means that BBBBB is what we are likely to get the most.

It's not always the case, by the way, not for every word. For instance, if we enter the word YORES, we are more likely to get BBBBG (guess the last letter on its spot) than BBBBB (guess no letters at all). This is because so many letters in the list end in "s". For YORES, 1084 words in the list would give back BBBBG, vs. 969 for BBBBB. It means that at least for the first step, BBBBB would actually be a better answer for us, because it does a better job of narrowing down the pool of answers.

If you ever played the comparison game, where you should guess the number from "yes" or "no" answers to the questions "is it more than N", you would know that the optimal strategy for winning this game, in the long run, is always splitting the interval of possible values in two equal parts. If you pick uneven intervals, every now and then you will get lucky and guess the number faster, but most of the times, the target number will be in the larger interval (because of the trivial fact that the larger interval is, well, larger, and the random number is more likely to be there), and you will just have wasted a try. Every new answer should, on average, bring the same amount of information.

This means that a good word for Wordle, on every step, would be the word which would break the pool of candidate answers into more or less equal bins. Of course, unlike the number guessing game, it's not always possible to make these bins exactly equal: four green letters cannot possibly leave you more than 25 candidates, even in theory. But what we can do is pick the word which will give us the smallest pool of answers in the worst case. I call this strategy "the best worst case". If we get lucky, all the better. But even if we don't (and the larger pool of candidates the answer gives us, the more likely are we to get this answer), we still don't end up with too large a pool.

Formulation of the problem

We need to find the word which, in the worst case, would limit the pool of answers to the number that is less than any other word would. If the word XYLAN gives us 3033 potential candidates in the worst case, and the word YORES gives 1084, then YORES is a better candidate for the first guess than XYLAN. We need to find the word with the best worst case, so that we are guaranteed to have more information on the first attempt.

If we look into the potential answers that we can get, we can see that the first letter can be black, yellow or green; the second letter can be black, yellow or green and so on. Every letter can have three colors, and there are five of them, which gives us 3⁵ = 273 possible combinations. Some combinations, though, are impossible, like four green letters and one yellow. This means that every guess divides the list of potential candidates into 273 (possibly empty) bins, some of them larger, some smaller. We want the bins to be as even as possible. We want to find the word for which the largest bin is smaller than the largest bin for every other word.

Implementation in SQL

I have created a table called wordle and loaded the list of 12478 five-letter words there. All the words are uppercase.

To encode the answer, we will be using ternary encoding. For the first letter, black is 0, yellow is 1, green is 2. For the second letter, black is 0×3 = 0, yellow is 1×3 = 3, green is 2×3 = 6 and so on. An answer like BGYGB would be encoded as 0×1 + 2×3 + 1×9 + 2×27 + 0×81 = 69.

Let's create a function to find the answer a target would give for a certain guess. It's not that straightforward because we need to handle repeating letters correctly (kudos to very astute @rcv from the comments section for pointing this out to me!).

First of all, let's notice that in case of an exact match on a spot, this spot will always be green. We will handle the green first, by replacing the matching letters with a period in both words.

With the greens out of the way, we need to calculate yellows and blacks. We don't have exact matches anymore in either word, because we got rid of them on the previous step. Now, if we have two of the same letter in the guess, but only one in the target word, we want only the first one of them to be yellow. We will evaluate the guess from left to right and notice how many occurrences of the current letter we've had so far. If it's less or equal to the number of occurrences of the same letter in the target word, then we color it yellow, otherwise black.

Let's see it on an example. Say, we are guessing ADDED for DUVET.

First of all, we handle the exact matches: ADD.D and DUV.T

Then we need to go over all spots in the guess, from 1 to 5, and count how many occurrences of the current letter we've had before (and including) this spot:

1. 'A': current letter 'A', 1 occurrence in the guess so far, 0 in the target. 1 is more than 0, it's a black
2. 'AD': current letter 'D', 1 occurrence in the guess so far, 1 in the target. It's a yellow
3. 'ADD': current letter 'D', 2 occurrences in the guess so far, 1 in the target. 2 is more than 1, it's a black.
4. 'ADD.': current letter '.'. It's an exact match which had been handled before, so it's a green.
5. 'ADD.D': current letter 'D', 3 occurrences in the guess so far, 1 in the target. 3 is more than 1, it's a black.

The answer, therefore, would be 'BYBGB'.

We will create some helper functions to implement this logic in SQL.

CREATE OR REPLACE FUNCTION fn_fix_green(one TEXT, other TEXT, ix INTEGER)
RETURNS TEXT
IMMUTABLE PARALLEL SAFE
AS
$$

SELECT  COALESCE(NULLIF(SUBSTRING(one FROM ix FOR 1), SUBSTRING(other FROM ix FOR 1)), '.')

$$
LANGUAGE 'sql';


CREATE OR REPLACE FUNCTION fn_fix_green(guess TEXT, target TEXT)
RETURNS TEXT
IMMUTABLE PARALLEL SAFE
AS
$$

SELECT  fn_fix_green(guess, target, 1) ||
        fn_fix_green(guess, target, 2) ||
        fn_fix_green(guess, target, 3) ||
        fn_fix_green(guess, target, 4) ||
        fn_fix_green(guess, target, 5)

$$
LANGUAGE 'sql';


CREATE OR REPLACE FUNCTION fn_count_character(word TEXT, letter TEXT)
RETURNS INTEGER
IMMUTABLE PARALLEL SAFE
AS
$$

SELECT  LENGTH(word) - LENGTH(REPLACE(word, letter, ''))

$$
LANGUAGE 'sql';


CREATE OR REPLACE FUNCTION fn_fix_other(guess TEXT, target TEXT, ix INTEGER)
RETURNS INT
IMMUTABLE PARALLEL SAFE
AS
$$

SELECT  CASE
        WHEN SUBSTRING(guess FROM ix FOR 1) = '.' THEN
                2
        WHEN fn_count_character(SUBSTRING(guess FROM 1 FOR ix), SUBSTRING(guess FROM ix FOR 1)) <=
                fn_count_character(target, SUBSTRING(guess FROM ix FOR 1)) THEN
                1
        ELSE
                0
        END

$$
LANGUAGE 'sql';

CREATE OR REPLACE FUNCTION fn_fix_other(guess TEXT, target TEXT)
RETURNS INT
IMMUTABLE PARALLEL SAFE
AS
$$
SELECT  fn_fix_other(guess, target, 1) +
        fn_fix_other(guess, target, 2) * 3 +
        fn_fix_other(guess, target, 3) * 9 +
        fn_fix_other(guess, target, 4) * 27 +
        fn_fix_other(guess, target, 5) * 81
$$
LANGUAGE 'sql';


CREATE OR REPLACE FUNCTION fn_match(guess text, target text)
RETURNS INT
IMMUTABLE PARALLEL SAFE
AS
$$

SELECT  fn_fix_other(fn_fix_green(guess, target), fn_fix_green(target, guess))

$$
LANGUAGE 'sql';


CREATE OR REPLACE FUNCTION fn_match_letter(color INT)
RETURNS TEXT
IMMUTABLE PARALLEL SAFE
AS
$$

SELECT  CASE color WHEN 0 THEN 'B' WHEN 1 THEN 'Y' ELSE 'G' END

$$
LANGUAGE 'sql';


CREATE OR REPLACE FUNCTION fn_match_text(colors INT)
RETURNS TEXT
IMMUTABLE PARALLEL SAFE
AS
$$

SELECT  fn_match_letter(colors % 3) ||
        fn_match_letter((colors / 3) % 3) ||
        fn_match_letter((colors / 9) % 3) ||
        fn_match_letter((colors / 27) % 3) ||
        fn_match_letter((colors / 81) % 3)

$$
LANGUAGE 'sql';

I could have just combined these expressions in the SELECT list of the query, but it would be hard to read. I am not using any kind of aggregates or FROM clause here, because this function is on the hot path and it works much faster this way.

Let's unit test these functions. I'm using examples from the articles I found online which explain how Wordle handles duplicate letters. We will compare the actual output of the function to the expected value. If there will be no mismatches, the query will come back empty:

SELECT  target, guess, fn_match_text(fn_match(guess, target))
FROM    (
        VALUES
        ('AAABB', 'BBAAA'),
        ('GHOTI', 'GHOTI'),
        ('REBUS', 'RUBUS'),
        ('BANAL', 'ANNAL'),
        ('BANAL', 'UNION'),
        ('BANAL', 'ALLOY'),
        ('BANAL', 'BANAL'),
        ('ABBEY', 'ABBEY'),
        ('ABBEY', 'ABYSS'),
        ('ABBEY', 'KEBAB'),
        ('ABBEY', 'BABES'),
        ('ABBEY', 'OPENS'),
        ('DUVET', 'ADDED')
        ) AS q (target, guess)
EXCEPT
VALUES
        ('AAABB', 'BBAAA', 'YYGYY'),
        ('GHOTI', 'GHOTI', 'GGGGG'),
        ('REBUS', 'RUBUS', 'GBGGG'),
        ('BANAL', 'ANNAL', 'YBGGG'),
        ('BANAL', 'UNION', 'BYBBB'),
        ('BANAL', 'ALLOY', 'YYBBB'),
        ('BANAL', 'BANAL', 'GGGGG'),
        ('ABBEY', 'ABBEY', 'GGGGG'),
        ('ABBEY', 'ABYSS', 'GGYBB'),
        ('ABBEY', 'KEBAB', 'BYGYY'),
        ('ABBEY', 'BABES', 'YYGGB'),
        ('ABBEY', 'OPENS', 'BBYBB'),
        ('DUVET', 'ADDED', 'BYBGB')

The output of the query is empty, which means that all our unit tests are passing.

Now, we need to split the guesses into the bins and find the one with the best worst case. This guess will have the largest bin which is smaller, than the largest bins of all other guesses.

Since we need to evaluate every possible guess and compare it with every possible target word, we will have to cross join the word list with itself and find the answer for every combination of every pair of words from the list. This means that the database will have to make whopping 155,700,484 comparisons and process them. Fortunately, modern computers are very fast.

SELECT  guess.word AS guess, target.word AS target, fn_match(guess.word, target.word) AS colors
FROM    wordle guess
CROSS JOIN
        wordle target

Now that we have the answer for every possible combination of words, we will need to group them by the guess and the answer, to know how large is the bin that every possible answer for every possible word gives us. For some of the guesses, some combination of colors will never pop up, because there would be no words in the list which could give these combinations as an answer. Yet, most of them, or at least many enough of them, would come back not empty. For every possible guess, we will have no more than 273 rows with different combinations of the colors, and the number of answers which would give this combination.

SELECT  guess, colors, COUNT(*) matches
FROM    (
        SELECT  guess.word AS guess, target.word AS target, fn_match(guess.word, target.word) AS colors
        FROM    wordle guess
        CROSS JOIN
                wordle target
        ) q
GROUP BY
        guess, colors

Then, we would need to take the worst case (the largest bin) for every answer. Since we want to return not just the number of the candidates in the largest bin, but also the answer which would give us this bin, we would use a window function to order these bins and give every one of them its rank from the worst to the best:

SELECT  guess, colors, matches,
        ROW_NUMBER() OVER (PARTITION BY guess ORDER BY matches DESC) AS rn
FROM    (
        SELECT  guess, colors, COUNT(*) matches
        FROM    (
                SELECT  guess.word AS guess, target.word AS target, fn_match(guess.word, target.word) AS colors
                FROM    wordle guess
                CROSS JOIN
                        wordle target
                ) q
        GROUP BY
                guess, colors
        ) q
) q

Finally, we want to select just the worst bins for each guess (they would have rn = 1), and, among those, select the best, ordering them by the number of matches:

SELECT  *
FROM    (
        SELECT  guess, colors, matches,
                ROW_NUMBER() OVER (PARTITION BY guess ORDER BY matches DESC) AS rn
        FROM    (
                SELECT  guess, colors, COUNT(*) matches
                FROM    (
                        SELECT  guess.word AS guess, target.word AS target, fn_match(guess.word, target.word) AS colors
                        FROM    wordle guess
                        CROSS JOIN
                                wordle target
                        ) q
                GROUP BY
                        guess, colors
                ) q
        ) q
WHERE   rn = 1
ORDER BY
        matches

This query ran for 76 minutes for my laptop.

So, it looks like a good first word for Wordle would be serai (a Turkish palace). Even in the worst case, it would narrow the list of candidate answers down to 659 entries.

It might not be the optimal word in the strict sense. There is a good chance that two-level or deeper analysis would bring a better first guess, that would not be optimal on the first step, but would be on the second, even in the worst case. But implementing it in SQL would require a triple cartesian self-join of a 12k records table, which would take weeks to complete and would probably run out of space on my laptop.

A whole game or Wordle in SQL

Let's solve yesterday's Wordle (#221) completely in SQL!

We will be playing by the hardest rules: every green letter should be on its spot, every yellow letter should be used elsewhere, no black letters should be used. Ironically, this is easiest to implement in SQL: we will just need to make sure that every next guess is in the pool of all the previous guesses. This will keep pool of potential guesses and potential answers shorter and make the query run faster.

We will create a materialized CTE called valid to calculate the pool. It will be only populated by the words which are in line with the answer Wordly gave us.

Here's the game. We will start with SERAI, our top pick. Wordle gives us back BBBYB (27).

Let's find the next best candidate for the guess:

WITH    guesses (guess, colors) AS
        (
        VALUES
        ('SERAI', 27)
        ),
        valid AS MATERIALIZED
        (
        SELECT  *
        FROM    wordle
        WHERE   (
                SELECT  BOOL_AND(fn_match(guess, word) = colors)
                FROM    guesses
                )
        )
SELECT  guess, matches
FROM    (
        SELECT  guess, colors, matches,
                ROW_NUMBER() OVER (PARTITION BY guess ORDER BY matches DESC) AS rn
        FROM    (
                SELECT  guess, colors, COUNT(*) matches
                FROM    (
                        SELECT  guess.word AS guess, target.word AS target, fn_match(guess.word, target.word) AS colors
                        FROM    valid guess
                        CROSS JOIN
                                valid target
                        ) q
                GROUP BY
                        guess, colors
                ) q
        ) q
WHERE   rn = 1
ORDER BY
        matches
LIMIT 1

Alright, the next best candidate is NYALA. We put it into Wordle, and it gives us back BBGBB (18).

What's the next word?

WITH    guesses (guess, colors) AS
        (
        VALUES
        ('SERAI', 27),
        ('NYALA', 18)
        ),
        valid AS MATERIALIZED
        (
        SELECT  *
        FROM    wordle
        WHERE   (
                SELECT  BOOL_AND(fn_match(guess, word) = colors)
                FROM    guesses
                )
        )
SELECT  guess, matches
FROM    (
        SELECT  guess, colors, matches,
                ROW_NUMBER() OVER (PARTITION BY guess ORDER BY matches DESC) AS rn
        FROM    (
                SELECT  guess, colors, COUNT(*) matches
                FROM    (
                        SELECT  guess.word AS guess, target.word AS target, fn_match(guess.word, target.word) AS colors
                        FROM    valid guess
                        CROSS JOIN
                                valid target
                        ) q
                GROUP BY
                        guess, colors
                ) q
        ) q
WHERE   rn = 1
ORDER BY
        matches
LIMIT 1

The next word is COAPT (which would narrow us down to just 6 candidates, worst case). What does Wordle think about it?

It's YBGBB (19). On to the next one.

WITH    guesses (guess, colors) AS
        (
        VALUES
        ('SERAI', 27),
        ('NYALA', 18),
        ('COAPT', 19)
        ),
        valid AS MATERIALIZED
        (
        SELECT  *
        FROM    wordle
        WHERE   (
                SELECT  BOOL_AND(fn_match(guess, word) = colors)
                FROM    guesses
                )
        )
SELECT  guess, matches
FROM    (
        SELECT  guess, colors, matches,
                ROW_NUMBER() OVER (PARTITION BY guess ORDER BY matches DESC) AS rn
        FROM    (
                SELECT  guess, colors, COUNT(*) matches
                FROM    (
                        SELECT  guess.word AS guess, target.word AS target, fn_match(guess.word, target.word) AS colors
                        FROM    valid guess
                        CROSS JOIN
                                valid target
                        ) q
                GROUP BY
                        guess, colors
                ) q
        ) q
WHERE   rn = 1
ORDER BY
        matches
LIMIT   1

The next one is WHACK, with just two candidates. Let's put it into Wordle and… we win!

Have fun!

The post A good first word for Wordle appeared first on EXPLAIN EXTENDED.

Challenge accepted!

This year we will be building an SQL query which will emulate a quantum computer. This query will process quantum assembly, build the circuit, run the emulation and make the measurements.

First things first, a little bit of theory. I won't go deep into quantum mechanics now (primarily because I don't understand it well enough to talk about it in public). What we really need to know about emulating a quantum computer, is that it's all about matrix multiplication. Quantum computers run on physical effects which are hard to wrap one's head around, but relatively easy to express using quite simple math. This math is something you can work with, even if you don't understand the physics behind it on an intuitive level.

Theory

For this article, I will assume that you are familiar with the mathematics of matrix multiplication. If you're not, you'll need to read up a little bit on linear algebra. This is not a particularly hard topic, and it's being used heavily in many areas of programming: image processing, sound processing, quantitative finance analysis and many others. It is very rewarding to be familiar with it.

Qubits

So, quantum computers have registers (tiny blocks of memory), in pretty much the same way as the CPU in your laptop or phone does. The data stored in your CPU registers tells it what to do next, and these registers are being constantly updated as your CPU runs code.

Classic registers have bits, which store zeros and ones. These are exclusive: if the bit is on, it's not off, and if it's off, it's not on.

Quantum registers have qubits, which also store zeros and ones. But these zeros and ones are not exclusive. A qubit may be on, may be off, and may be somewhere in between. It's not like an on-off switch, but more like a computer trackball with a permanent marker dot on it. You can turn any way you like, and the dot position reflects the state of the qubit. The closer the dot is to the top (or to the bottom), the more "zero" or "one" the qubit is. The marked dot on the trackball can also turn about the vertical axis, which is also something that the qubit can store.

This is confusing, and you might need to read up on quantum mechanics to grasp it. But don't worry about it now. What we have to know about the qubit is that we need two complex numbers to describe it. The first number describes how much of "zero" it is, and the second one describes how much of "one" it is. These complex numbers' norms should add up to 1.

With classical registers, when we add another bit in the mix, we double the number of states the register can be in. So, a two-bit register has four possible states: 00, 01, 10, and 11. These states are mutually exclusive. If the register is 00, it cannot be 01 at the same time. This is why we are able to use two bits to describe the four states.

This does not work the same way with qubits. One qubit can be somewhat |0⟩ and somewhat |1⟩. Two qubits can be somewhat |00⟩, somewhat |01⟩, somewhat |10⟩, and somewhat |11⟩. To describe two qubits, we need four complex numbers. Their norms still have to add up to 1, but otherwise they can be any old numbers. This means that when two qubits are used together, they are not just two separate qubits anymore. It's possible that a two-qubits register is somewhat |00⟩, and somewhat |11⟩, but none of |01⟩ or |10⟩, all at the same time. It is, in a sense, larger than the sum of its parts. This is called entanglement. Three qubits can be in any mix of states from |000⟩ through |111⟩, which means 8 different complex numbers to describe every state's share in the mix. And so on. It is not enough to describe the state of every qubit in the register. We need two complex numbers to describe every combination of the qubits.

You might have wondered what these weird notations with the vertical var and the angle bracket mean. They are a part of Dirac's bra-ket notation. When I use it, it means I'm talking about these strange half this, half that qubit states, as opposed to straight and honest classical bits. But again, if you are not already familiar with all that, don't worry about it now. Many bright scientific minds have struggled with understanding this problem in depth.

For now, we just need to remember that if we have a register of N qubits, we need 2^N complex numbers to describe its state.

As you can probably see now, this is where the problem of emulating quantum computers starts to show. A laptop or phone CPU these days probably has high tens or maybe low hundreds of 64-bit and 128-bit registers, so let's say something in the order of magnitude of 10000 bits of state. If we were to emulate a classical CPU using a computer program, we would need to have 10000 bits of memory to keep this state. These days, 10000 bits of memory is nothing.

But to emulate 10000 qubits, we would need to use about 2¹⁰⁰⁰⁰ bits of memory. That's about 10³⁰⁰⁰ bytes. It's hell of a lot of memory. If we take all the memory that has been and will ever be produced on the planet Earth, and plug it into our emulator, we will still be 10³⁰⁰⁰ bytes short of our target goal.

In practice, a laptop can emulate 20-qubit or so quantum registers. Supercomputers can do high forties.

State evolution

When a CPU is running code, it looks into its registers and decides what to do next, using quite complicated logic. The CPU also usually has access to external devices (like RAM), and its control flow can be changed by external events like interrupts and exceptions.

The quantum flow is much dumber. A purely quantum algorithm does not have branches, external data, or any kind of control flow. It consists of a series of gates, one after another, each of them changing the state of the quantum register. There is relatively simple math behind those changes.

If we have a quantum register with N qubits, we can describe its state using 2^N complex numbers. Each of these numbers can be completely independent of the others (again, as long as their norms add up to one), so we kind of need to store all of them. We store them in an array. Mathematically speaking, this array represents a normalized vector in 2^N-dimensional complex space.

When we run the register through a quantum gate (or, as they put it, "evolve" the register), its state changes according to certain math rules. Each quantum gate is, in essence, a square 2^N by 2^N complex matrix. The old state of the register gets multiplied by the gate's matrix and becomes the new state.

Unitary matrices

The gates' matrices have to be unitary. This means that if we take its conjugate transpose (turn the matrix about the diagonal and change the signs of all the imaginary parts in every element) and multiply it by the original matrix, we'll get the identity matrix back. Identity means a matrix with all 1's in the diagonal cells and all 0's in all the other cells. Multiplying any vector by the identity matrix always gives the same vector back. This is why it's called identity.

Unitary matrices have two important properties.

First of all, if you multiply a vector by it, the result will have the same norm. This means that if the register state was normalized before the gate, it will remain normalized after.

Second, they are reversible. If you take another gate, whose matrix is the conjugate transpose of the original gate, and evolve the register through both of these gates, you will get the same exact state back. Can you see why? Remember that matrix multiplication is associative.

Unitary matrices let you get from any normalized vector to any other normalized vector and back.

So, in essence, all that quantum computing allows you to do is take a normalized vector with 2^N complex elements and transform it to another normalized vector with 2^N complex elements, according to some rules. That's all there is to it.

Turns out, that being able to do these transformations fast is a very good thing. An N-qubit register can in fact store 2^N complex numbers and transforming this register through the unitary matrices is done by the same machinery that runs our universe, which is very fast. These transformations can sometimes be designed to do the same things as really important functions which would take very long to compute on a classical computer. To be able to make these transformations fast would be the same as to be able to compute these functions 2^N times at once.

Every ideal quantum circuit will evolve the state in a deterministic way. If we run the same input state through the circuit, we will always get the same output state once all the evolutions are done. In this sense, a quantum circuit is not that much a program, as it is a function, which rotates (and possibly reflects) vectors about a constant axis in 2^N-dimensional complex space. That's what all the fuss is about.

Come to think of it, a quantum computer is just a glorified circular slide rule — in 2^N dimensions. Yet, it is immensely useful to have these kinds of transformations implemented in, so to say, hardware, because they do solve real-world problems and they are very hard to compute on classical computer.

Measurement

Once the quantum register has gone through all the evolutions, it gets into a new state. But here's the thing: we cannot read this state. The universe machinery that is doing the matrix multiplication so fast for us, unfortunately, does not allow to read its results. When we try to read a quantum register, the universe randomly selects a position in the array, and magically puts the number one into that position, and zeros on all the others. By trying to read a quantum register, we change its state, and there is no way around it. This phenomenon is called wave function collapse, which is another mystic thing you might have heard about.

There is still a way to retrieve at least some information about the final state. When we measure qubits, the probability to get a certain combination of 0's and 1's is equal to the norm of the complex number in the state which describes this combination. If we have a two-qubit register in this state: , then the probability of measuring every possible combinations of 0's and 1's is equal to 0.25 (which is the norm of the complex number 0.5). If the state looks like this: , then the probability of measuring 00 or 11 is equal to 0.5, and the probabilities of measuring 01 or 10 are zero. But the measurement will return either 00 or 11, and there is no way of telling in advance which one it will be. And since the measurement of 00 would replace the state with , all subsequent measurements of the register would return the same value of 00 (because it is the only one left with a non-zero probability amplitude).

The universe does not come equipped with a debugger. We cannot tap into the inner workings of quantum mechanics and see what is going on there. If we tried to do that, this would count as a measurement, and this would break the state (or "collapse" it).

We cannot look inside a quantum circuit. But something we can do is re-run the experiment multiple times. As we remember, a quantum circuit is a deterministic function. If we feed it the same input state, it will evolve it into the same output state every time. When we try to measure this state, it will collapse differently for each experiment. By looking at how often different combinations of 0's and 1's turn up in measurements, we can estimate what the coefficients in the original state vector were.

And this is exactly how we use the quantum computer:

1. We use our math knowledge to build a set of gates that would turn the initial state vector into another state vector, which would somehow encode the answer that we need. The initial state vector usually has all the qubits initialized to |0⟩. This means that it has 1 for the state |000..0⟩, and 0's for all the other states
2. The target state vector will be very large, and we will not be able to calculate all its values directly. But we will know that these values will be larger for certain states and smaller for the others, because this is how we arranged the gates
3. On a hardware quantum device, we arrange the quantum gates, run the initial state through them many times, and make the measurements of each result
4. From looking at the distribution of the measurement results, we estimate the norms of the components of the target vector. The distribution of frequencies of the register states turning up in measurements will encode the answer to our problem

Phases

The coefficients in the state vector (they are called probability amplitudes) are complex numbers. If a certain combination of 0's and 1's (each one of these combinations is called an eigenstate) has the probability amplitude of 0.5, this particular combination will show up in the measurements one time of four, on average. But the same would be true if the probability amplitude of this eigenstate were -0.5, or 0.5i, or -0.5i, or, in fact, any complex number whose norm is 0.25 (there are infinitely many of them). We cannot tell these probability amplitudes apart by looking at the measurements. The state vector does have them as complex numbers, and it does matter when the state evolves. It is called a "relative phase", or phase difference between probability amplitudes of different eigenstates. Relative phases have physical significance and do matter in register evolutions, but cannot be measured, statistically or otherwise. The final state cannot have the answer encoded in the relative phases of the probability amplitudes, because they are impossible to measure. The answer has to be encoded in the amplitudes' norms.

On the other hand, if you multiply the state vector by a complex number with the norm 1 (which means multiplying all its components by this number), this factor will remain in the state all the way through its evolution. This is easy to verify: the quantum circuit, ultimately, multiples the state vector by a matrix, and the scalar multiplication of vectors and matrices is associative. This is called a "global phase", because it acts on all the eigenstates in the same way. Unlike relative phases, the global phase does not have any physical significance. It does not affect evolutions in any meaningful way, and of course it cannot be measured experimentally. Register states which only differ in global phase, do not really differ at all, not in any way that we can detect.

A little digression. An astute reader might have noticed that earlier I was talking about encoding a single qubit as a position of a trackball with a dot on it. The position of a trackball can be encoded with two real numbers (elevation and azimuth angles), but the qubit state is described with two complex numbers. Turns out that if we disregard the global phase and take into account the fact that for a single qubit the amplitudes have to be normalized (their norms need to add up to 1), we can indeed encode these two complex numbers with two real ones. But this is only true for a single qubit. To describe two or more qubits, we will still need the complex vector space.

Gates

Now, what kind of gates (or matrices) can we use to build our quantum circuits?

Different types of quantum computers use different kinds of gates. The IBM Q family uses the U₁(λ) gate, the R_X(π/2) gate, and the CNOT gate. Here are their matrices:

The first two gates act on one qubit, the CNOT (otherwise called CX) acts on two.

If our quantum register has more qubits than the gate transforms (and any useful register does), we extend the matrix by Kronecker multiplying it with an identity matrix. Here's what the transformation matrix looks like for a three-qubit register with a CNOT gate on qubits 0 and 1:

Turns out that by combining (multiplying) these three basic transformations, you can build any unitary matrix.

Hardware implementation

All above works in theory. With the real hardware, there are some implementation problems:

• The qubits are very hard to build and maintain. IBM recently announced that they had developed a 127-qubit processor, which is the bleeding edge technology at the time of this writing
• Quantum machines are very sensitive to environment noise. They have to live in a deep freezer at mK temperatures and be shielded from any radiation, including radio waves. Any interaction with the outer world halfway through the experiment collapses the wave function and messes up the register state. This means that the measurement statistics will be off, possibly to the point where the peaks in probabilities for certain states drown in noise
• Theoretically, we can use gates to connect any pair of qubits in the register. In practice, qubits have connectivity diagrams. This means that you can apply the CNOT gates only between certain pairs of qubits, and only in certain directions. There are ways to overcome these limitations by stacking the gates in clever ways, but this makes circuits longer and more prone to noise.

Despite all these limitations, quantum computers do exist and work. IBM offers free access to them, and a really nice SDK to work with them called Qiskit. You can use it to design and build quantum circuits and run them on real quantum hardware in the cloud.

I hope I didn't make any blatant mistakes in my overview. With that out of the way, let's jump to the implementation.

Implementation in SQL

To test our simulator, we will implement a 2x2 binary Sudoku solver using Grover's algorithm. I adapted this algorithm straight from the Qiskit textbook.

Complex number helpers

To work with complex numbers, it would be really helpful to have complex types. PostgreSQL does not have a complex number type built-in, so I will create a type for complex numbers, as well as some functions to work with this type. Usually, I use pure SQL for my SQL magic posts, because using procedural code is cheating in kills the magic. But these types are just thin wrappers around some lengthy expressions, and the functions don't have any procedural code in them.

Here are the types I created:

DROP TYPE IF EXISTS COMPLEX CASCADE;

CREATE TYPE COMPLEX AS (r DOUBLE PRECISION, i DOUBLE PRECISION);

CREATE OR REPLACE FUNCTION complex(a DOUBLE PRECISION)
RETURNS complex
AS
$$
        SELECT  (a, 0)::COMPLEX;
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP CAST IF EXISTS (DOUBLE PRECISION AS COMPLEX);

CREATE CAST (DOUBLE PRECISION AS COMPLEX)
        WITH FUNCTION complex(DOUBLE PRECISION);

CREATE OR REPLACE FUNCTION complex(a INT)
RETURNS complex
AS
$$
        SELECT  a::DOUBLE PRECISION::COMPLEX;
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP CAST IF EXISTS (INT AS COMPLEX);

CREATE CAST (INT AS COMPLEX)
        WITH FUNCTION complex(INT);


CREATE OR REPLACE FUNCTION pos(a COMPLEX)
RETURNS COMPLEX
AS
$$
        SELECT  a
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS + (NONE, complex);

CREATE OPERATOR +
        (
        RIGHTARG = complex,
        PROCEDURE = pos
        );

CREATE OR REPLACE FUNCTION neg(a COMPLEX)
RETURNS COMPLEX
AS
$$
        SELECT  (-a.r, -a.i)::COMPLEX
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS - (NONE, complex);

CREATE OPERATOR -
        (
        RIGHTARG = complex,
        PROCEDURE = neg
        );

CREATE OR REPLACE FUNCTION add (a COMPLEX, b COMPLEX)
RETURNS COMPLEX
AS
$$
        SELECT  (a.r + b.r, a.i + b.i)::COMPLEX
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS + (complex, complex);

CREATE OPERATOR +
        (
        LEFTARG = complex,
        RIGHTARG = complex,
        PROCEDURE = add,
        COMMUTATOR = +
        );

CREATE OR REPLACE FUNCTION mul (a COMPLEX, b COMPLEX)
RETURNS COMPLEX
AS
$$
        SELECT  (a.r * b.r - a.i * b.i, a.r * b.i + a.i * b.r)::COMPLEX
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS * (complex, complex);

CREATE OPERATOR *
        (
        LEFTARG = complex,
        RIGHTARG = complex,
        PROCEDURE = mul,
        COMMUTATOR = *
        );

CREATE OR REPLACE FUNCTION mul (a COMPLEX, b DOUBLE PRECISION)
RETURNS COMPLEX
AS
$$
        SELECT  (a.r * b, a.i * b)::COMPLEX
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS * (complex, DOUBLE PRECISION);

CREATE OPERATOR *
        (
        LEFTARG = complex,
        RIGHTARG = DOUBLE PRECISION,
        PROCEDURE = mul,
        COMMUTATOR = *
        );

CREATE OR REPLACE FUNCTION mul (a DOUBLE PRECISION, b COMPLEX)
RETURNS COMPLEX
AS
$$
        SELECT  b * a
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS * (DOUBLE PRECISION, COMPLEX);

CREATE OPERATOR *
        (
        LEFTARG = DOUBLE PRECISION,
        RIGHTARG = COMPLEX,
        PROCEDURE = mul,
        COMMUTATOR = *
        );

CREATE OR REPLACE FUNCTION norm(a COMPLEX)
RETURNS DOUBLE PRECISION
AS
$$
        SELECT  a.r * a.r + a.i * a.i
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

CREATE OR REPLACE FUNCTION magnitude(a COMPLEX)
RETURNS DOUBLE PRECISION
AS
$$
        SELECT  SQRT(norm(a))
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS ~ (NONE, complex);

CREATE OPERATOR ~
        (
        RIGHTARG = complex,
        PROCEDURE = magnitude
        );

CREATE OR REPLACE FUNCTION conjugate(a COMPLEX)
RETURNS COMPLEX
AS
$$
        SELECT  (a.r, -a.i)::COMPLEX
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS # (NONE, complex);

CREATE OPERATOR #
        (
        RIGHTARG = complex,
        PROCEDURE = conjugate
        );

CREATE OR REPLACE FUNCTION div (a COMPLEX, b DOUBLE PRECISION)
RETURNS COMPLEX
AS
$$
        SELECT  (a.r / b, a.i / b);
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS / (complex, DOUBLE PRECISION);

CREATE OPERATOR /
        (
        LEFTARG = complex,
        RIGHTARG = DOUBLE PRECISION,
        PROCEDURE = div
        );

CREATE OR REPLACE FUNCTION rec (a COMPLEX)
RETURNS COMPLEX
AS
$$
        SELECT  #a / norm(a)
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS // (NONE, COMPLEX);

CREATE OPERATOR //
        (
        RIGHTARG = COMPLEX,
        PROCEDURE = rec
        );

CREATE OR REPLACE FUNCTION div (a COMPLEX, b COMPLEX)
RETURNS COMPLEX
AS
$$
        SELECT  a * //b
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

DROP OPERATOR IF EXISTS / (COMPLEX, COMPLEX);

CREATE OPERATOR /
        (
        LEFTARG = COMPLEX,
        RIGHTARG = COMPLEX,
        PROCEDURE = div
        );

CREATE OR REPLACE FUNCTION exp(a COMPLEX)
RETURNS COMPLEX
AS
$$
        SELECT  EXP(a.r) * (COS(a.i), SIN(a.i))::COMPLEX
$$
LANGUAGE 'sql'
IMMUTABLE
STRICT;

CREATE AGGREGATE SUM(COMPLEX)
        (
        STYPE = COMPLEX,
        SFUNC = add
        );

Gate definitions

For our quantum circuit, we need to define some gates. As I told before, there are just three gates that the IBM hardware is using under the hood, and you can build all other gates by combining these three. In the SDK, you can use more complex gates. The hardware which runs the circuit will figure out how to construct them from the building blocks.

We will be using the Pauli gates X, Y, and Z; the Hadamard gate H; and the multi-qubit controlled-NOT (generalized Toffoli) gates C_nX, connecting from 2 to 5 qubits.

For the single-qubit gates, we will define the matrices explicitly; for the Toffoli, we will do it programmatically.

WITH    gates (opcode, arity, matrix) AS MATERIALIZED
        (
        VALUES
                ('X', 1, ARRAY[
                        [0, 1],
                        [1, 0]
                        ]::COMPLEX[][]),
                ('Y', 1, ARRAY[
                        [0, -(0, 1)],
                        [-(0, 1), 0]
                        ]::COMPLEX[][]),
                ('Z', 1, ARRAY[
                        [1, 0],
                        [0, -1]
                        ]::COMPLEX[][]),
                ('H', 1, ARRAY[
                        [1 / SQRT(2), 1 / SQRT(2)],
                        [1 / SQRT(2), -1 / SQRT(2)]
                        ]::COMPLEX[][])
        UNION ALL
        SELECT  REPEAT('C', arity - 1) || 'X', arity, matrix
        FROM    GENERATE_SERIES(2, 5) arity
        CROSS JOIN LATERAL
                (
                WITH    constants AS
                        (
                        SELECT  (1 << (arity - 1)) - 1 AS mask,
                                (1 << arity) - 1 AS rank
                        )
                SELECT  ARRAY_AGG(cols ORDER BY row) AS matrix
                FROM    constants
                CROSS JOIN LATERAL
                        (
                        SELECT  row, ARRAY_AGG((CASE WHEN row & mask = mask AND col & mask = mask THEN row <> col ELSE row = col END)::INT::COMPLEX ORDER BY col) cols
                        FROM    GENERATE_SERIES(0, rank) row
                        CROSS JOIN
                                GENERATE_SERIES(0, rank) col
                        GROUP BY
                                row
                        ) gate
                ) toffoli

        )
SELECT  opcode, arity,
        (
        SELECT  STRING_AGG(cols::TEXT, E'\n' ORDER BY row)
        FROM    (
                SELECT  row, ARRAY_AGG(coefficient ORDER BY col) cols
                FROM    (
                        SELECT  (r, i)::COMPLEX coefficient,
                                (ordinality - 1) / (1 << arity) AS row, (ordinality -1 ) % (1 << arity) AS col
                        FROM    UNNEST(matrix) WITH ORDINALITY matrix (r, i, ordinality)
                        ) q
                GROUP BY
                        row
                ) q
        ) matrix
FROM    gates;

As you can see, the more qubits are in the gate, the larger the matrix is. Fortunately, most of the coefficients in the matrices that we are going to use are zeros. This will help us with the performance when we get there.

Assembly language

Now that we have the gates, we can build the circuit. We will encode it in a text string, using assembly language of my own invention. The grammar is dirt simple: semicolon-separated gates, colon-separated opcodes, and comma-separated inputs. Whitespaces and line breaks to taste.

WITH    RECURSIVE
        settings AS
        (
        SELECT  9 AS qubits,
                ARRAY[0, 1, 2, 3] AS measurements,
                '
H: 8;
Z: 8;
H: 0;
H: 1;
H: 2;
H: 3;

CX: 0,4;
CX: 1,4;
CX: 0,5;
CX: 2,5;
CX: 1,6;
CX: 3,6;
CX: 2,7;
CX: 3,7;
CCCCX: 4,5,6,7,8;
CX: 0,4;
CX: 1,4;
CX: 0,5;
CX: 2,5;
CX: 1,6;
CX: 3,6;
CX: 2,7;
CX: 3,7;

H: 0;
H: 1;
H: 2;
H: 3;
X: 0;
X: 1;
X: 2;
X: 3;
H: 3;
CCCX: 0,1,2,3;
H: 3;
X: 0;
X: 1;
X: 2;
X: 3;
H: 0;
H: 1;
H: 2;
H: 3;

CX: 0,4;
CX: 1,4;
CX: 0,5;
CX: 2,5;
CX: 1,6;
CX: 3,6;
CX: 2,7;
CX: 3,7;
CCCCX: 4,5,6,7,8;
CX: 0,4;
CX: 1,4;
CX: 0,5;
CX: 2,5;
CX: 1,6;
CX: 3,6;
CX: 2,7;
CX: 3,7;

H: 0;
H: 1;
H: 2;
H: 3;
X: 0;
X: 1;
X: 2;
X: 3;
H: 3;
CCCX: 0,1,2,3;
H: 3;
X: 0;
X: 1;
X: 2;
X: 3;
H: 0;
H: 1;
H: 2;
H: 3;
' AS program
        ),
        circuit AS MATERIALIZED
        (
        SELECT  step, parts[1] AS opcode, inputs::INT[]
        FROM    settings
        CROSS JOIN
                REGEXP_SPLIT_TO_TABLE(program, E'\\s*;\\s*') WITH ORDINALITY instructions(instruction, step)
        CROSS JOIN LATERAL
                REGEXP_MATCHES(instruction, E'(\\w+)\\s*:\\s(.*)') parts
        CROSS JOIN LATERAL
                REGEXP_SPLIT_TO_ARRAY(parts[2], E'\\s*,\\s*') inputs
        )
SELECT  *
FROM    circuit
ORDER BY
        step
LIMIT 10

Global unitary matrices from state matrices

To evolve the state, we need to build the complete 2^N×2^N unitary matrix for each step. To do this, we will need to Kronecker multiply the identity matrix (for the non-input qubits) by the gate matrix (for the input qubits). We will return the resulting matrix as a rowset with row numbers, column numbers, and only non-zero coefficients in it.

For the Kronecker multiplication, we will use a little optimization trick. Because the identity matrix has zeros in non-diagonal elements, we can be sure that in the resulting matrix, the non-zero elements will only be in the rows and columns where the eigenstates of non-input qubits are the same.

Ok, this might be a little bit hard to parse. Here's an example.

Our circuit has 9 qubits. Let's assume we have a CX gate between inputs 0 and 1. The total unitary matrix for this gate will be a Kronecker product of the identity matrix for the qubits 2 through 9 (2⁷×2⁷ = 128×128), and the 4×4 CX matrix for the qubits 0 and 1.

In a transformation matrix, every row and every column corresponds to some eigenstate of the state vector. There is a row for the state |000000000⟩, a row for the state |000000001⟩, and so on. Similarly, there is a column for the state |000000000⟩, a column for the state |000000001⟩, and so on.

We can make a quick optimization when filling up the cells. We can look at the seven leftmost bits in the row's eigenstate and compare them to the seven leftmost bits in the column's eigenstate. If they are not equal, they will have a zero in the identity matrix. It means we should not be worrying about them at all, we can just skip them when building the query.

Only if the leftmost seven bits for the row and the column are equal, there is a change of having a non-zero value in the matrix.

This trick reduces the number of calculations from 4^N to 2^N for every matrix — in our case, by the factor of 512.

To calculate the resulting matrix, we will need to map it to the actual inputs used in the circuit, and then cross join it with the eigenstates of non-input qubits.

Let's see what the matrix looks like for the gate CCCCX between the inputs 0, 1, 2, 3 and 4.

WITH    settings AS
        (
        SELECT  9 AS qubits
        ),
        gates (opcode, arity, matrix) AS MATERIALIZED
        (
        VALUES
                ('X', 1, ARRAY[
                        [0, 1],
                        [1, 0]
                        ]::COMPLEX[][]),
                ('Y', 1, ARRAY[
                        [0, -(0, 1)],
                        [-(0, 1), 0]
                        ]::COMPLEX[][]),
                ('Z', 1, ARRAY[
                        [1, 0],
                        [0, -1]
                        ]::COMPLEX[][]),
                ('H', 1, ARRAY[
                        [1 / SQRT(2), 1 / SQRT(2)],
                        [1 / SQRT(2), -1 / SQRT(2)]
                        ]::COMPLEX[][])
        UNION ALL
        SELECT  REPEAT('C', arity - 1) || 'X', arity, matrix
        FROM    GENERATE_SERIES(2, 5) arity
        CROSS JOIN LATERAL
                (
                WITH    constants AS
                        (
                        SELECT  (1 << (arity - 1)) - 1 AS mask,
                                (1 << arity) - 1 AS rank
                        )
                SELECT  ARRAY_AGG(cols ORDER BY row) AS matrix
                FROM    constants
                CROSS JOIN LATERAL
                        (
                        SELECT  row, ARRAY_AGG((CASE WHEN row & mask = mask AND col & mask = mask THEN row <> col ELSE row = col END)::INT::COMPLEX ORDER BY col) cols
                        FROM    GENERATE_SERIES(0, rank) row
                        CROSS JOIN
                                GENERATE_SERIES(0, rank) col
                        GROUP BY
                                row
                        ) gate
                ) toffoli

        ),
        gate AS MATERIALIZED
        (
        SELECT  arity, matrix, ARRAY[0, 1, 2, 3, 4] AS inputs
        FROM    gates
        WHERE   gates.opcode = 'CCCCX'
        ),
        identity_qubits AS MATERIALIZED
        (
        SELECT  ARRAY_AGG(input ORDER BY input) identity_qubits
        FROM    settings
        CROSS JOIN
                gate
        CROSS JOIN LATERAL
                (
                SELECT  input
                FROM    GENERATE_SERIES(0, qubits - 1) input
                EXCEPT
                SELECT  input
                FROM    UNNEST(inputs) input
                ) q
        ),
        unitary AS
        (
        SELECT  row, col, coefficient
        FROM    settings
        CROSS JOIN
                gate
        CROSS JOIN LATERAL
                (
                SELECT  circuit_identity_basis | circuit_gate_row_basis AS row,
                        circuit_identity_basis | circuit_gate_col_basis AS col,
                        coefficient
                FROM    gate
                CROSS JOIN
                        identity_qubits
                CROSS JOIN LATERAL
                        (
                        WITH    circuit_gate_basis AS
                                (
                                SELECT  gate_basis, circuit_gate_basis
                                FROM    GENERATE_SERIES(0, (1 << arity) - 1) gate_basis
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  COALESCE(BIT_OR(1 << inputs[input + 1]), 0) AS circuit_gate_basis
                                        FROM    GENERATE_SERIES(0, arity - 1) input
                                        WHERE   gate_basis & (1 << input) > 0
                                        ) circuit_gate_basis
                                )
                        SELECT  row.circuit_gate_basis AS circuit_gate_row_basis,
                                col.circuit_gate_basis AS circuit_gate_col_basis,
                                matrix[row.gate_basis + 1][col.gate_basis + 1]::COMPLEX AS coefficient
                        FROM    circuit_gate_basis row
                        CROSS JOIN
                                circuit_gate_basis col
                        ) circuit_gate_basis
                CROSS JOIN LATERAL
                        (
                        SELECT  circuit_identity_basis
                        FROM    GENERATE_SERIES(0, (1 << (qubits - arity)) - 1) identity_basis
                        CROSS JOIN LATERAL
                                (
                                SELECT  COALESCE(BIT_OR(1 << identity_qubit), 0) AS circuit_identity_basis
                                FROM    UNNEST(identity_qubits) WITH ORDINALITY AS identity_qubits (identity_qubit, input)
                                WHERE   identity_basis & (1 << (input - 1)::INT) > 0
                                ) circuit_identity_basis
                        ) circuit_identity_basis
                ) unitary
        WHERE   coefficient <> 0::COMPLEX
        )
SELECT  *
FROM    unitary
ORDER BY
        row, col, coefficient

State evolution

To evolve the state of the register, we need to unnest its array, multiply it by the gate matrix and aggregate it back into the array. Matrix multiplication works very smoothly with SQL and translates quite naturally into joins, sums and group-by clauses.

There is a little problem with unnesting arrays of record types in PostgreSQL. The function UNNEST splits the record types into individual fieds. In the output of UNNEST there will be separate double precision fields for the real and the imaginary parts of the number. We will need to combine them back into the COMPLEX type.

Let us see how a single Hadamard gate acts on the register in the initial state (1 for the eigenstate |000000000⟩, 0 for all others):

WITH    RECURSIVE
        settings AS
        (
        SELECT  9 AS qubits
        ),
        basis AS
        (
        SELECT  eigenstate
        FROM    settings
        CROSS JOIN
                generate_series(0, (1 << qubits) - 1) AS eigenstate
        ),
        gates (opcode, arity, matrix) AS MATERIALIZED
        (
        VALUES
                ('X', 1, ARRAY[
                        [0, 1],
                        [1, 0]
                        ]::COMPLEX[][]),
                ('Y', 1, ARRAY[
                        [0, -(0, 1)],
                        [-(0, 1), 0]
                        ]::COMPLEX[][]),
                ('Z', 1, ARRAY[
                        [1, 0],
                        [0, -1]
                        ]::COMPLEX[][]),
                ('H', 1, ARRAY[
                        [1 / SQRT(2), 1 / SQRT(2)],
                        [1 / SQRT(2), -1 / SQRT(2)]
                        ]::COMPLEX[][])
        UNION ALL
        SELECT  REPEAT('C', arity - 1) || 'X', arity, matrix
        FROM    GENERATE_SERIES(2, 5) arity
        CROSS JOIN LATERAL
                (
                WITH    constants AS
                        (
                        SELECT  (1 << (arity - 1)) - 1 AS mask,
                                (1 << arity) - 1 AS rank
                        )
                SELECT  ARRAY_AGG(cols ORDER BY row) AS matrix
                FROM    constants
                CROSS JOIN LATERAL
                        (
                        SELECT  row, ARRAY_AGG((CASE WHEN row & mask = mask AND col & mask = mask THEN row <> col ELSE row = col END)::INT::COMPLEX ORDER BY col) cols
                        FROM    GENERATE_SERIES(0, rank) row
                        CROSS JOIN
                                GENERATE_SERIES(0, rank) col
                        GROUP BY
                                row
                        ) gate
                ) toffoli

        ),
        initial_state AS
        (
        SELECT  ARRAY_AGG((CASE eigenstate WHEN 0 THEN 1 ELSE 0 END)::COMPLEX ORDER BY eigenstate) AS state
        FROM    basis
        ),
        new_state AS
        (
        SELECT  new_state.state
        FROM    settings
        CROSS JOIN
                initial_state
        CROSS JOIN LATERAL
                (
                WITH    gate AS MATERIALIZED
                        (
                        SELECT  arity, matrix, ARRAY[0] AS inputs
                        FROM    gates
                        WHERE   gates.opcode = 'H'
                        ),
                        identity_qubits AS MATERIALIZED
                        (
                        SELECT  ARRAY_AGG(input ORDER BY input) identity_qubits
                        FROM    gate
                        CROSS JOIN LATERAL
                                (
                                SELECT  input
                                FROM    GENERATE_SERIES(0, qubits - 1) input
                                EXCEPT
                                SELECT  input
                                FROM    UNNEST(inputs) input
                                ) q
                        ),
                        unitary AS
                        (
                        SELECT  circuit_identity_basis | circuit_gate_row_basis AS row,
                                circuit_identity_basis | circuit_gate_col_basis AS col,
                                coefficient
                        FROM    gate
                        CROSS JOIN
                                identity_qubits
                        CROSS JOIN LATERAL
                                (
                                WITH    circuit_gate_basis AS
                                        (
                                        SELECT  gate_basis, circuit_gate_basis
                                        FROM    GENERATE_SERIES(0, (1 << arity) - 1) gate_basis
                                        CROSS JOIN LATERAL
                                                (
                                                SELECT  COALESCE(BIT_OR(1 << inputs[input + 1]), 0) AS circuit_gate_basis
                                                FROM    GENERATE_SERIES(0, arity - 1) input
                                                WHERE   gate_basis & (1 << input) > 0
                                                ) circuit_gate_basis
                                        )
                                SELECT  row.circuit_gate_basis AS circuit_gate_row_basis,
                                        col.circuit_gate_basis AS circuit_gate_col_basis,
                                        matrix[row.gate_basis + 1][col.gate_basis + 1]::COMPLEX AS coefficient
                                FROM    circuit_gate_basis row
                                CROSS JOIN
                                        circuit_gate_basis col
                                ) circuit_gate_basis
                        CROSS JOIN LATERAL
                                (
                                SELECT  circuit_identity_basis
                                FROM    GENERATE_SERIES(0, (1 << (qubits - arity)) - 1) identity_basis
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  COALESCE(BIT_OR(1 << identity_qubit), 0) AS circuit_identity_basis
                                        FROM    UNNEST(identity_qubits) WITH ORDINALITY AS identity_qubits (identity_qubit, input)
                                        WHERE   identity_basis & (1 << (input - 1)::INT) > 0
                                        ) circuit_identity_basis
                                ) circuit_identity_basis
                        WHERE   coefficient <> 0::COMPLEX
                        ),
                        state AS
                        (
                        WITH    state AS
                                (
                                SELECT  (r, i)::COMPLEX amplitude, ordinality - 1 AS eigenstate
                                FROM    UNNEST(state) WITH ORDINALITY state (r, i, ordinality)
                                )
                        SELECT  row, SUM(amplitude * coefficient) AS amplitude
                        FROM    state
                        JOIN    unitary
                        ON      col = eigenstate
                        GROUP BY
                                row
                        )
                SELECT  ARRAY_AGG(amplitude ORDER BY row) state
                FROM    state
                ) new_state
        )
SELECT  RIGHT((ordinality - 1)::BIT(9)::TEXT, qubits) eigenstate,
        (r, i)::COMPLEX AS amplitude
FROM    settings
CROSS JOIN
        new_state
CROSS JOIN LATERAL
        UNNEST(state) WITH ORDINALITY state(r, i, ordinality)

The Hadamard gate puts the qubit 0 into a superposition of the states |0⟩ and |1⟩ with equal amplitudes. When we do the measurement, we will get 0 or 1 in the first qubit with equal probability. All the other qubits will always measure as 0.

Let us do a simulation of measurements of this register. For illustration purposes, we won't be bothering with emulating random experiments. Instead, we will calculate the theoretical probabilities. We will be measuring all qubits. For each state, we will just take the norm of its probability amplitude. This will be the probability to measure a register in this state.

WITH    RECURSIVE
        settings AS
        (
        SELECT  9 AS qubits,
                ARRAY[0, 1, 2, 3, 4, 5, 6, 7, 8] AS measurements
        ),
        basis AS
        (
        SELECT  eigenstate
        FROM    settings
        CROSS JOIN
                generate_series(0, (1 << qubits) - 1) AS eigenstate
        ),
        gates (opcode, arity, matrix) AS MATERIALIZED
        (
        VALUES
                ('X', 1, ARRAY[
                        [0, 1],
                        [1, 0]
                        ]::COMPLEX[][]),
                ('Y', 1, ARRAY[
                        [0, -(0, 1)],
                        [-(0, 1), 0]
                        ]::COMPLEX[][]),
                ('Z', 1, ARRAY[
                        [1, 0],
                        [0, -1]
                        ]::COMPLEX[][]),
                ('H', 1, ARRAY[
                        [1 / SQRT(2), 1 / SQRT(2)],
                        [1 / SQRT(2), -1 / SQRT(2)]
                        ]::COMPLEX[][])
        UNION ALL
        SELECT  REPEAT('C', arity - 1) || 'X', arity, matrix
        FROM    GENERATE_SERIES(2, 5) arity
        CROSS JOIN LATERAL
                (
                WITH    constants AS
                        (
                        SELECT  (1 << (arity - 1)) - 1 AS mask,
                                (1 << arity) - 1 AS rank
                        )
                SELECT  ARRAY_AGG(cols ORDER BY row) AS matrix
                FROM    constants
                CROSS JOIN LATERAL
                        (
                        SELECT  row, ARRAY_AGG((CASE WHEN row & mask = mask AND col & mask = mask THEN row <> col ELSE row = col END)::INT::COMPLEX ORDER BY col) cols
                        FROM    GENERATE_SERIES(0, rank) row
                        CROSS JOIN
                                GENERATE_SERIES(0, rank) col
                        GROUP BY
                                row
                        ) gate
                ) toffoli

        ),
        initial_state AS
        (
        SELECT  ARRAY_AGG((CASE eigenstate WHEN 0 THEN 1 ELSE 0 END)::COMPLEX ORDER BY eigenstate) AS state
        FROM    basis
        ),
        new_state AS
        (
        SELECT  new_state.state
        FROM    settings
        CROSS JOIN
                initial_state
        CROSS JOIN LATERAL
                (
                WITH    gate AS MATERIALIZED
                        (
                        SELECT  arity, matrix, ARRAY[0] AS inputs
                        FROM    gates
                        WHERE   gates.opcode = 'H'
                        ),
                        identity_qubits AS MATERIALIZED
                        (
                        SELECT  ARRAY_AGG(input ORDER BY input) identity_qubits
                        FROM    gate
                        CROSS JOIN LATERAL
                                (
                                SELECT  input
                                FROM    GENERATE_SERIES(0, qubits - 1) input
                                EXCEPT
                                SELECT  input
                                FROM    UNNEST(inputs) input
                                ) q
                        ),
                        unitary AS
                        (
                        SELECT  circuit_identity_basis | circuit_gate_row_basis AS row,
                                circuit_identity_basis | circuit_gate_col_basis AS col,
                                coefficient
                        FROM    gate
                        CROSS JOIN
                                identity_qubits
                        CROSS JOIN LATERAL
                                (
                                WITH    circuit_gate_basis AS
                                        (
                                        SELECT  gate_basis, circuit_gate_basis
                                        FROM    GENERATE_SERIES(0, (1 << arity) - 1) gate_basis
                                        CROSS JOIN LATERAL
                                                (
                                                SELECT  COALESCE(BIT_OR(1 << inputs[input + 1]), 0) AS circuit_gate_basis
                                                FROM    GENERATE_SERIES(0, arity - 1) input
                                                WHERE   gate_basis & (1 << input) > 0
                                                ) circuit_gate_basis
                                        )
                                SELECT  row.circuit_gate_basis AS circuit_gate_row_basis,
                                        col.circuit_gate_basis AS circuit_gate_col_basis,
                                        matrix[row.gate_basis + 1][col.gate_basis + 1]::COMPLEX AS coefficient
                                FROM    circuit_gate_basis row
                                CROSS JOIN
                                        circuit_gate_basis col
                                ) circuit_gate_basis
                        CROSS JOIN LATERAL
                                (
                                SELECT  circuit_identity_basis
                                FROM    GENERATE_SERIES(0, (1 << (qubits - arity)) - 1) identity_basis
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  COALESCE(BIT_OR(1 << identity_qubit), 0) AS circuit_identity_basis
                                        FROM    UNNEST(identity_qubits) WITH ORDINALITY AS identity_qubits (identity_qubit, input)
                                        WHERE   identity_basis & (1 << (input - 1)::INT) > 0
                                        ) circuit_identity_basis
                                ) circuit_identity_basis
                        WHERE   coefficient <> 0::COMPLEX
                        ),
                        state AS
                        (
                        WITH    state AS
                                (
                                SELECT  (r, i)::COMPLEX amplitude, ordinality - 1 AS eigenstate
                                FROM    UNNEST(state) WITH ORDINALITY state (r, i, ordinality)
                                )
                        SELECT  row, SUM(amplitude * coefficient) AS amplitude
                        FROM    state
                        JOIN    unitary
                        ON      col = eigenstate
                        GROUP BY
                                row
                        )
                SELECT  ARRAY_AGG(amplitude ORDER BY row) state
                FROM    state
                ) new_state
        )
SELECT  eigenstate_bits, probability
FROM    new_state
CROSS JOIN
        settings
CROSS JOIN LATERAL
        (
        WITH    probabilities AS
                (
                SELECT  norm((r, i)::COMPLEX) AS probability, ordinality - 1 AS eigenstate
                FROM    UNNEST(state) WITH ORDINALITY state(r, i, ordinality)
                )
        SELECT  RIGHT(measurement_eigenstate::BIT(36)::TEXT, ARRAY_LENGTH(measurements, 1)) AS eigenstate_bits,
                SUM(probability)::NUMERIC(4, 4) AS probability
        FROM    probabilities
        CROSS JOIN LATERAL
                (
                SELECT  BIT_OR(((eigenstate >> qubit) & 1) << position::INT - 1) AS measurement_eigenstate
                FROM    UNNEST(measurements) WITH ORDINALITY measurements (qubit, position)
                ) measurement
        GROUP BY
                measurement_eigenstate
        ) measurements
WHERE   probability >= 0.0001
ORDER BY
        probability DESC, eigenstate_bits

Now we have all the machinery in place to run our simulation. Let's run an actual circuit!

We will run the algorithm to solve a binary 2×2 Sudoku puzzle. Each row and each column in a 2×2 square should have exactly one 1 and exactly one 0.

Here's the principle behind the Grover's algorithm.

Let's say we have a boolean-valued function which takes N-bit values and gives either 1 or 0 for every one of them. We need to find the values which give the 1 (or, maybe, just one value).

It might seem trivial — we know the function, we know the values, so all we need to do to apply the function for each value and see what it gives us. However, to know all the answers, we would need to apply this function 2^N times. If the value of N is sufficiently large, it will take significant time to run all these repetitions.

This is what Bitcoin miners do all day long. Every Bitcoin transaction header has an empty slot to fill with 0's and 1's in such a way that its (double) SHA-256 hash be lower than a certain threshold. Even though the SHA-256 algorithm is relatively simple and there exist specialized circuits that can apply the hash billions of times per second, it still takes a significant amount of time and electricity to find a good value to put into this slot.

Another example of such a function would be factorization of a product of two sufficiently large primes. It would return 1 if the input is a factor of the product, 0 otherwise. Even though integer division is a relatively simple operation for a modern computer, it is still considered an impossible task to factorize a big enough number, because the number of repetition would be too large. A major part of modern cryptography is based on this assumption.

For some of these functions, we can design a quantum circuit which emulates calling this function. This circuit should flip the relative phase for the eigenstates which are the answers for the functions. Such a circuit is called "quantum oracle".

We cannot measure the relative phase of an eigenstate directly, but there's a circuit which allows to "amplify" the phases. After its application, the eigenstates with the phase flipped get an amplitude with a higher norm (and the rest of the states, of course, get an amplitude with a lower norm). These states get a higher chance to be measured.

It is trivial to code up such a function if we know its solutions. But the whole point of quantum algorithms is to find these solutions in the first place. So we need to tell the quantum algoritm the conditions of the function and let it come up with the answers. How exactly we do this (and if it's possible to do it at all), depends on the function. This is the trickest part of quantum programming.

So, for our Sudoku puzzle we need to do the following:

1. Encode all possible solutions to the Sudoku puzzle as qubit eigenstates. There are 2⁴ = 16 possible solutions, which can be encoded with 4 qubits
2. Create a quantum oracle. It's a circuit which would flip the phase for the states that satisfy the Sudoku condition and leave it intact for those that don't.
   1. The Sudoku condition is "exactly one bit is set in each row and each column". It can be encoded by applying two CNOT gates for every rows and every columns, and having them control the "row and column" qubits.
   2. The qubit is reversed only if exactly one of two control inputs is 1 and other one is 0 in a row or column.
   3. All the row-and-column qubits control the "Sudoku" qubit through a four-way Toffoli gate.
   4. The Sudoku qubit is reversed only if all the conditions hold.
   5. By placing a Z gate on the "Sudoku" qubit, we introduce a phase difference between its |0⟩ and |1⟩ states. We do it to use a trick known as "phase kickback". The control qubit of a CNOT gate causes the target qubit to trigger (switch the amplitudes of |0⟩ and |1⟩ states of the target qubit). But this operation can also have effect on the control qubit. If there was a phase difference between the amplitudes of |0⟩ and |1⟩ states of the target qubit, this phase difference will propagate back to the control qubit after the evolution through the CNOT gate. This is exactly what we want to do: flip the phase on the eigenstates of the control qubits which have a component triggering the Sudoku qubit
   6. We need to uncompute the "cell" qubits and return them into the original eigenstate of |0000⟩. This is done by re-applying the CNOT gates. This will also propagate the phase kickback back to the original 4 qubits in the oracle

   As we can see, the quantum oracle circuit requires more qubits to check all these conditions: 2 for rows, 2 for columns and 1 more for the total condition. These qubits are called the ancilla qubits. In classical computations, we could use the extra bits of memory as a scratchpad and dump them after we are done. But the quantum computation is reversible, so it's impossible to just ignore the ancilla bits. They are not going anywhere, and if they end up entangled with the oracle qubits, they will continue to influence them. This is why we need to bring the state of the ancilla bits back to the original eigenstate (or any other pure state)

3. Apply the phase amplifier circuit to convert the relative phases into amplitudes with bigger norms
4. Copy the circuit and apply it again to the target state. This will allow the phase amplifier to make another pass, and make the difference in the good answers' and bad answers' norms more pronounced
5. Measure the first 4 qubits and isolate the eigenstates which turn up in the measurements most often. These eigenstates are the solutions to the Sudoku puzzle

Notice that we didn't give the circuit the answers to the Sudoku. We gave it the limiting conditions. These is a very fine distinction, but this is exactly what we would need to do for a really complex task where we don't know the answers beforehand.

Qiskit implementation

Here's how we do it in Qiskit. I took this circuit from the textbook, but expanded the oracle and the phase amplifier into the sequences of elementary gates:

from qiskit import QuantumCircuit, QuantumRegister, ClassicalRegister, assemble, Aer, execute
aer = Aer.get_backend("aer_simulator")

creg = ClassicalRegister(4, name="c")
qc = QuantumCircuit(QuantumRegister(9, name="q"), creg)

qc.h(8)
qc.z(8)

qc.h(0)
qc.h(1)
qc.h(2)
qc.h(3)

qc.cx(0, 4)
qc.cx(1, 4)
qc.cx(0, 5)
qc.cx(2, 5)
qc.cx(1, 6)
qc.cx(3, 6)
qc.cx(2, 7)
qc.cx(3, 7)

qc.mct([4, 5, 6, 7], 8)

qc.cx(0, 4)
qc.cx(1, 4)
qc.cx(0, 5)
qc.cx(2, 5)
qc.cx(1, 6)
qc.cx(3, 6)
qc.cx(2, 7)
qc.cx(3, 7)

qc.barrier()

qc.h(0)
qc.h(1)
qc.h(2)
qc.h(3)
qc.x(0)
qc.x(1)
qc.x(2)
qc.x(3)
qc.h(3)

qc.mct([0, 1, 2], 3)

qc.h(3)
qc.x(0)
qc.x(1)
qc.x(2)
qc.x(3)
qc.h(0)
qc.h(1)
qc.h(2)
qc.h(3)

qc.barrier()

qc.cx(0, 4)
qc.cx(1, 4)
qc.cx(0, 5)
qc.cx(2, 5)
qc.cx(1, 6)
qc.cx(3, 6)
qc.cx(2, 7)
qc.cx(3, 7)

qc.mct([4, 5, 6, 7], 8)

qc.cx(0, 4)
qc.cx(1, 4)
qc.cx(0, 5)
qc.cx(2, 5)
qc.cx(1, 6)
qc.cx(3, 6)
qc.cx(2, 7)
qc.cx(3, 7)

qc.barrier()

qc.h(0)
qc.h(1)
qc.h(2)
qc.h(3)
qc.x(0)
qc.x(1)
qc.x(2)
qc.x(3)
qc.h(3)

qc.mct([0, 1, 2], 3)

qc.h(3)
qc.x(0)
qc.x(1)
qc.x(2)
qc.x(3)
qc.h(0)
qc.h(1)
qc.h(2)
qc.h(3)

qc.measure([0, 1, 2, 3], creg)
qobj = assemble(qc)
aer_result = aer.run(qobj).result()
qc.draw()

from qiskit.visualization import plot_histogram
plot_histogram(aer_result.get_counts())

SQL implementation

And here's how we do it in SQL:

WITH    RECURSIVE
        settings AS
        (
        SELECT  9 AS qubits,
                ARRAY[0, 1, 2, 3] AS measurements,
                '
H: 8;
Z: 8;
H: 0;
H: 1;
H: 2;
H: 3;

CX: 0,4;
CX: 1,4;
CX: 0,5;
CX: 2,5;
CX: 1,6;
CX: 3,6;
CX: 2,7;
CX: 3,7;
CCCCX: 4,5,6,7,8;
CX: 0,4;
CX: 1,4;
CX: 0,5;
CX: 2,5;
CX: 1,6;
CX: 3,6;
CX: 2,7;
CX: 3,7;

H: 0;
H: 1;
H: 2;
H: 3;
X: 0;
X: 1;
X: 2;
X: 3;
H: 3;
CCCX: 0,1,2,3;
H: 3;
X: 0;
X: 1;
X: 2;
X: 3;
H: 0;
H: 1;
H: 2;
H: 3;

CX: 0,4;
CX: 1,4;
CX: 0,5;
CX: 2,5;
CX: 1,6;
CX: 3,6;
CX: 2,7;
CX: 3,7;
CCCCX: 4,5,6,7,8;
CX: 0,4;
CX: 1,4;
CX: 0,5;
CX: 2,5;
CX: 1,6;
CX: 3,6;
CX: 2,7;
CX: 3,7;

H: 0;
H: 1;
H: 2;
H: 3;
X: 0;
X: 1;
X: 2;
X: 3;
H: 3;
CCCX: 0,1,2,3;
H: 3;
X: 0;
X: 1;
X: 2;
X: 3;
H: 0;
H: 1;
H: 2;
H: 3;
' AS program
        ),
        basis AS
        (
        SELECT  eigenstate
        FROM    settings
        CROSS JOIN
                generate_series(0, (1 << qubits) - 1) AS eigenstate
        ),
        gates (opcode, arity, matrix) AS MATERIALIZED
        (
        VALUES
                ('X', 1, ARRAY[
                        [0, 1],
                        [1, 0]
                        ]::COMPLEX[][]),
                ('Y', 1, ARRAY[
                        [0, -(0, 1)],
                        [-(0, 1), 0]
                        ]::COMPLEX[][]),
                ('Z', 1, ARRAY[
                        [1, 0],
                        [0, -1]
                        ]::COMPLEX[][]),
                ('H', 1, ARRAY[
                        [1 / SQRT(2), 1 / SQRT(2)],
                        [1 / SQRT(2), -1 / SQRT(2)]
                        ]::COMPLEX[][])
        UNION ALL
        SELECT  REPEAT('C', arity - 1) || 'X', arity, matrix
        FROM    GENERATE_SERIES(2, 5) arity
        CROSS JOIN LATERAL
                (
                WITH    constants AS
                        (
                        SELECT  (1 << (arity - 1)) - 1 AS mask,
                                (1 << arity) - 1 AS rank
                        )
                SELECT  ARRAY_AGG(cols ORDER BY row) AS matrix
                FROM    constants
                CROSS JOIN LATERAL
                        (
                        SELECT  row, ARRAY_AGG((CASE WHEN row & mask = mask AND col & mask = mask THEN row <> col ELSE row = col END)::INT::COMPLEX ORDER BY col) cols
                        FROM    GENERATE_SERIES(0, rank) row
                        CROSS JOIN
                                GENERATE_SERIES(0, rank) col
                        GROUP BY
                                row
                        ) gate
                ) toffoli

        ),
        circuit AS MATERIALIZED
        (
        SELECT  step, parts[1] AS opcode, inputs::INT[]
        FROM    settings
        CROSS JOIN
                REGEXP_SPLIT_TO_TABLE(program, E'\\s*;\\s*') WITH ORDINALITY instructions(instruction, step)
        CROSS JOIN LATERAL
                REGEXP_MATCHES(instruction, E'(\\w+)\\s*:\\s(.*)') parts
        CROSS JOIN LATERAL
                REGEXP_SPLIT_TO_ARRAY(parts[2], E'\\s*,\\s*') inputs
        ),
        evolutions AS
        (
        SELECT  0 AS step, steps, state
        FROM    (
                SELECT  COUNT(*) AS steps
                FROM    circuit
                ) steps
        CROSS JOIN
                (
                SELECT  ARRAY_AGG((CASE eigenstate WHEN 0 THEN 1 ELSE 0 END)::COMPLEX ORDER BY eigenstate) AS state
                FROM    basis
                ) initial_state
        UNION ALL
        SELECT  step + 1, steps, new_state.state
        FROM    evolutions
        CROSS JOIN
                settings
        CROSS JOIN LATERAL
                (
                WITH    circuit_gate AS MATERIALIZED
                        (
                        SELECT  *
                        FROM    circuit
                        WHERE   circuit.step = evolutions.step + 1
                        ),
                        gate AS MATERIALIZED
                        (
                        SELECT  arity, matrix, inputs
                        FROM    circuit_gate
                        JOIN    gates
                        ON      gates.opcode = circuit_gate.opcode
                        ),
                        identity_qubits AS MATERIALIZED
                        (
                        SELECT  ARRAY_AGG(input ORDER BY input) identity_qubits
                        FROM    gate
                        CROSS JOIN LATERAL
                                (
                                SELECT  input
                                FROM    GENERATE_SERIES(0, qubits - 1) input
                                EXCEPT
                                SELECT  input
                                FROM    UNNEST(inputs) input
                                ) q
                        ),
                        unitary AS
                        (
                        SELECT  circuit_identity_basis | circuit_gate_row_basis AS row,
                                circuit_identity_basis | circuit_gate_col_basis AS col,
                                coefficient
                        FROM    gate
                        CROSS JOIN
                                identity_qubits
                        CROSS JOIN LATERAL
                                (
                                WITH    circuit_gate_basis AS
                                        (
                                        SELECT  gate_basis, circuit_gate_basis
                                        FROM    GENERATE_SERIES(0, (1 << arity) - 1) gate_basis
                                        CROSS JOIN LATERAL
                                                (
                                                SELECT  COALESCE(BIT_OR(1 << inputs[input + 1]), 0) AS circuit_gate_basis
                                                FROM    GENERATE_SERIES(0, arity - 1) input
                                                WHERE   gate_basis & (1 << input) > 0
                                                ) circuit_gate_basis
                                        )
                                SELECT  row.circuit_gate_basis AS circuit_gate_row_basis,
                                        col.circuit_gate_basis AS circuit_gate_col_basis,
                                        matrix[row.gate_basis + 1][col.gate_basis + 1]::COMPLEX AS coefficient
                                FROM    circuit_gate_basis row
                                CROSS JOIN
                                        circuit_gate_basis col
                                ) circuit_gate_basis
                        CROSS JOIN LATERAL
                                (
                                SELECT  circuit_identity_basis
                                FROM    GENERATE_SERIES(0, (1 << (qubits - arity)) - 1) identity_basis
                                CROSS JOIN LATERAL
                                        (
                                        SELECT  COALESCE(BIT_OR(1 << identity_qubit), 0) AS circuit_identity_basis
                                        FROM    UNNEST(identity_qubits) WITH ORDINALITY AS identity_qubits (identity_qubit, input)
                                        WHERE   identity_basis & (1 << (input - 1)::INT) > 0
                                        ) circuit_identity_basis
                                ) circuit_identity_basis
                        WHERE   coefficient <> 0::COMPLEX
                        ),
                        state AS
                        (
                        WITH    state AS
                                (
                                SELECT  (r, i)::COMPLEX amplitude, ordinality - 1 AS eigenstate
                                FROM    UNNEST(state) WITH ORDINALITY state (r, i, ordinality)
                                )
                        SELECT  row, SUM(amplitude * coefficient) AS amplitude
                        FROM    state
                        JOIN    unitary
                        ON      col = eigenstate
                        GROUP BY
                                row
                        )
                SELECT  ARRAY_AGG(amplitude ORDER BY row) state
                FROM    state
                ) new_state
        WHERE   step < steps
        )
SELECT  eigenstate_bits, probability
FROM    (
        SELECT  state
        FROM    evolutions
        WHERE   step = steps
        ) state
CROSS JOIN
        settings
CROSS JOIN LATERAL
        (
        WITH    probabilities AS
                (
                SELECT  norm((r, i)::COMPLEX) AS probability, ordinality - 1 AS eigenstate
                FROM    UNNEST(state) WITH ORDINALITY state(r, i, ordinality)
                )
        SELECT  RIGHT(measurement_eigenstate::BIT(36)::TEXT, ARRAY_LENGTH(measurements, 1)) AS eigenstate_bits,
                SUM(probability)::NUMERIC(4, 4) AS probability
        FROM    probabilities
        CROSS JOIN LATERAL
                (
                SELECT  BIT_OR(((eigenstate >> qubit) & 1) << position::INT - 1) AS measurement_eigenstate
                FROM    UNNEST(measurements) WITH ORDINALITY measurements (qubit, position)
                ) measurement
        GROUP BY
                measurement_eigenstate
        ) measurements
WHERE   probability >= 0.0001
ORDER BY
        probability DESC, eigenstate_bits

As we can see, our SQL quantum emulator runs in a fraction of a second and gives back the same answer that Qiskit's emulator does. The majority of experiments measure the first 4 qubits in the states |0110⟩ and |1001⟩, which correspond to the fields and , which are indeed the answers to the binary Sudoku puzzle. It works!

You can view the queries here: https://github.com/quassnoi/explain-extended-2022

I wish you to be in perfect state in this New Year. May you only be entangled with people you love, may all probabilities be in your favor, and may all your measurements come out the way you expect.

Previous New Year posts:

• Happy New 2010 Year!
• Happy New 2011 Year!
• Happy New 2012 Year!
• Happy New 2013 Year!
• Happy New 2014 Year!
• Happy New 2015 Year!
• Happy New 2016 Year!
• Happy New 2017 Year!
• Happy New 2018 Year!
• Happy New 2019 Year!
• Happy New 2020 Year!
• Happy New 2021 Year!

The post Happy New Year: quantum computer emulator in SQL appeared first on EXPLAIN EXTENDED.

In my New Year posts I usually try to recap and summarize the past year. It won't take long this time:

Fuck you, coronavirus!

Now that I've gotten that off my chest, I have to think of something to write about in this New Year's post.

So I was thinking, why not put a face to the name we all hate so much?

Let's use SQL to do some ray tracing and draw a 3D picture of the dreaded virus.

By now, I believe we are all familiar with the picture of the virus. It looks like a ball covered with spikes. The spikes look something like the solar corona, which is what gave the virus its name. They have this distinct triangular shape.

We'll create a sphere covered with several dozens of spikes.

Every spike will be a small pyramid, with an equilateral triangle as a base and isosceles triangles as lateral faces. This means it will be a right pyramid.

The pyramids will be "standing" on their apexes, upside-down. The height of every pyramid will be perpendicular to the sphere surface and continue the sphere's radius at the apex.

Then we will implement the pinhole camera model and use ray tracing algorithms to calculate the lighting of the sphere and the spikes.

Types

3D modeling heavily uses vector algebra. Of course pure SQL offers enough math functions to get around. But functions and routines are not first class citizens in SQL, which means we would have to copy-paste the bulky vector manipulation formulas every time we will need them, which would make our query unwieldy really fast.

This is a good chance to get familiar with PostgreSQL's rich system of custom types and custom operators. It lets users define their own types, create functions to work with them and even overload the operators.

Before we start working with our model, we will create a type for 3D vectors and several functions and operators to work with that type.

CREATE TYPE VEC3 AS (x DOUBLE PRECISION, y DOUBLE PRECISION, z DOUBLE PRECISION);



CREATE OR REPLACE FUNCTION VEC3_ADD (a VEC3, b VEC3)
RETURNS vec3
AS
$$
SELECT (a.x + b.x, a.y + b.y, a.z + b.z)::VEC3;
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE OPERATOR + (FUNCTION = VEC3_ADD, LEFTARG = VEC3, RIGHTARG = VEC3, COMMUTATOR = +);



CREATE OR REPLACE FUNCTION VEC3_NEG(a VEC3)
RETURNS VEC3
AS
$$
SELECT  (-a.x, -a.y, -a.z)::VEC3;
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE OPERATOR - (FUNCTION = VEC3_NEG, RIGHTARG = VEC3);



CREATE OR REPLACE FUNCTION VEC3_SUB(a VEC3, b VEC3)
RETURNS VEC3
AS
$$
SELECT (a.x - b.x, a.y - b.y, a.z - b.z)::VEC3;
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE OPERATOR - (FUNCTION = VEC3_SUB, RIGHTARG = VEC3, LEFTARG = VEC3);



CREATE OR REPLACE FUNCTION VEC3_MUL(a VEC3, k DOUBLE PRECISION)
RETURNS VEC3
AS
$$
SELECT  (a.x * k, a.y * k, a.z * k)::VEC3;
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE OR REPLACE FUNCTION VEC3_MUL(k DOUBLE PRECISION, a VEC3)
RETURNS VEC3
AS
$$
SELECT  VEC3_MUL(a, k);
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE OPERATOR * (FUNCTION = VEC3_MUL, LEFTARG = VEC3, RIGHTARG = DOUBLE PRECISION, COMMUTATOR = *);
CREATE OPERATOR * (FUNCTION = VEC3_MUL, LEFTARG = DOUBLE PRECISION, RIGHTARG = VEC3, COMMUTATOR = *);



CREATE OR REPLACE FUNCTION VEC3_DIV(a VEC3, k DOUBLE PRECISION)
RETURNS VEC3
AS
$$
SELECT  (a * (1 / k))::VEC3
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE OPERATOR / (FUNCTION = VEC3_DIV, LEFTARG = VEC3, RIGHTARG = DOUBLE PRECISION);



CREATE OR REPLACE FUNCTION VEC3_DOT(a VEC3, b VEC3)
RETURNS DOUBLE PRECISION
AS
$$
SELECT  a.x * b.x + a.y * b.y + a.z * b.z;
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE OPERATOR * (FUNCTION = VEC3_DOT, LEFTARG = VEC3, RIGHTARG = VEC3, COMMUTATOR = *);



CREATE OR REPLACE FUNCTION VEC3_MODULE(a VEC3)
RETURNS DOUBLE PRECISION
AS
$$
SELECT  SQRT(a.x^2 + a.y^2 + a.z^2)
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE OPERATOR | (FUNCTION = VEC3_MODULE, RIGHTARG = VEC3);



CREATE OR REPLACE FUNCTION VEC3_NORM(a VEC3)
RETURNS VEC3
AS
$$
SELECT  a / |a
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE OPERATOR || (FUNCTION = VEC3_NORM, RIGHTARG = VEC3);



CREATE OR REPLACE FUNCTION VEC3_CROSS(a VEC3, b VEC3)
RETURNS VEC3
AS
$$
SELECT  (a.y * b.z - a.z * b.y, a.z * b.x - a.x * b.z, a.x * b.y - a.y * b.x)::VEC3
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE OPERATOR ** (FUNCTION = VEC3_CROSS, LEFTARG = VEC3, RIGHTARG = VEC3);



CREATE TYPE SPHERICAL AS (radius DOUBLE PRECISION, theta DOUBLE PRECISION, phi DOUBLE PRECISION);



CREATE OR REPLACE FUNCTION VEC3_TO_SPHERICAL (a VEC3)
RETURNS SPHERICAL
AS
$$
SELECT  (|a, CASE WHEN |a = 0 THEN 0 ELSE ACOS(a.z / |a) END, ATAN2(a.y, a.x))::SPHERICAL
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE CAST (VEC3 AS SPHERICAL) WITH FUNCTION VEC3_TO_SPHERICAL(VEC3) AS ASSIGNMENT;



CREATE OR REPLACE FUNCTION SPHERICAL_TO_VEC3 (a SPHERICAL)
RETURNS VEC3
AS
$$
SELECT  a.radius * (SIN(a.theta) * COS(a.phi), SIN(a.theta) * SIN(a.phi), COS(a.theta))::VEC3
$$
LANGUAGE SQL
IMMUTABLE
RETURNS NULL ON NULL INPUT;

CREATE CAST (SPHERICAL AS VEC3) WITH FUNCTION SPHERICAL_TO_VEC3(SPHERICAL) AS ASSIGNMENT;

The type VEC3 is just a fancy name for a tuple of x, y, and z.

The operators +, -, *, / do scalar arithmetic operations on the vector, they accept a vector and a scalar.

The operators * and ** are dot product and cross product, respectively. They both accept two vectors on the either side, but the first one returns a number, and the second one returns another vector.

The unary operator | calculates the vector module (length), and || normalizes the vector (returns a vector with the same direction and the module of 1).

Finally, we'll create another type, SPHERICAL, which is a representation of a vector in spherical coordinates. SPHERICAL and VEC3 can be cast into each other, as they are just different representations of the same thing. We will need both of them, because some operations are easier in cartesian representation, and some are in spherical.

The apexes

Modeling a sphere is quite easy: all we need is its radius and the vector for its center.

For the spikes, we need to come up with a list of points on the sphere where the apexes will be. Then we'll need to create the base at a certain distance from the apex.

For our model to look close enough to the real thing, we need the apexes to be distributed more or less evenly around the sphere. I'm saying "more or less" because this problem, known as Thomson problem, does not have an exact solution except for some particular cases.

We don't need an exact solution, though, because we're modeling nature and it's a long way from being exact. All we need is a good enough solution. Fortunately, good enough is something nature itself has long since figured out. Turns out, there's a really simple algorithm, called the Fibonacci lattice, which allows to place points on a disc or a sphere so that their average density remains about the same.

The trick is to place the points on a Fermat's spiral at a certain fixed angle between every two adjacent points. This angle, called the golden angle, divides the circle in the golden ratio. So the points having the polar coordinates (k * √i, i * ϑ), where ϑ is the golden angle, will having density quite close to uniform. This is how the sunflower grows its florets. It makes sure that all the seeds will have equal space.

The same principle works for the sphere, except that the flat spiral has to be mapped onto a 3-dimensional surface. To preserve the density, we need to map the spiral in such a way that the flat area becomes proportional to the sphere area. We will need to use the cylindrical equal-area projection for that.

The polar coordinates on the plane will map to these spherical coordinates: (1, ACOS(2 * i / (N + 1) - 1), i * ϑ), where N is the number of points.

Let's build the set of 40 apexes for our future spikes on the unit sphere:

WITH    spikes AS
        (
        SELECT  i, spherical.*
        FROM    (
                SELECT  40 AS spikes
                ) constants
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, spikes - 1) AS i
        CROSS JOIN LATERAL
                (
                SELECT  ACOS(2 * i / spikes::DOUBLE PRECISION - 1) AS theta,
                        PI() * (3 - SQRT(5)) * i AS phi
                ) spherical
        )
SELECT  *
FROM    spikes;

Let's see how good our distribution is. To do this, we'll take a hemisphere with the center in each apex, and see how many spikes are located on this hemisphere. If the distribution is uniform, every hemisphere should contain half of all the spikes.

All the points on the hemisphere are within the straight line distance of √2 from its center. So we'll just have to convert the spherical coordinates into cartesian (using our cool types and conversion operators) and select all of them which are within the distance. The distance between two apexes is the module of their vector difference.

Let's make it into a query:

WITH    spikes AS
        (
        SELECT  i, spherical.*
        FROM    (
                SELECT  40 AS spikes
                ) constants
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, spikes - 1) AS i
        CROSS JOIN LATERAL
                (
                SELECT  ACOS(2 * i / spikes::DOUBLE PRECISION - 1) AS theta,
                        PI() * (3 - SQRT(5)) * i AS phi
                ) spherical
        )
SELECT  MIN(hemisphere_neighbors), MAX(hemisphere_neighbors)
FROM    spikes s1
CROSS JOIN LATERAL
        (
        SELECT  COUNT(*) hemisphere_neighbors
        FROM    spikes s2
        WHERE   |((1, s1.theta, s1.phi)::SPHERICAL::VEC3 - (1, s2.theta, s2.phi)::SPHERICAL::VEC3) <= SQRT(2)
        ) s2

It looks like every hemisphere contains from 18 to 21 apexes (out of 40 total) which is just about all right for our purpose.

The spikes

We have our apexes and we need to build the spike models from them. Every spike will be defined by 4 triangles: the base and the three faces of the pyramid. Every triangle will be defined as a set of three vectors, its vertices. So we'll have 4 records, three fields in each, for every spike.

The easiest way to build 40 spikes is to build one "prototype" spike and then clone it.

For the reasons that will become obvious later, it's best to build the prototype spike on the apex with both inclination and azimuth being zero: (1, 0, 0) in spherical coordinates. Note that this apex is not on our list. It is just a prototype, which will not make it to the final model.

We'll make our spikes 30% of the sphere's radius long and 10% wide. This way, the will be big enough to look cool on an ASCII art page, while still preserving that distinctive triangular shape.

The model apex has cartesian coordinates (0, 0, 1) which means that its radius is the Z-axis. The altitude (or height) dropped from the apex to the base should continue the radius, and its length is 0.3, so the center of the base will have the coordinates (0, 0, 1.3). The vertices of the base have to be at the distance of 0.1 from the center of the base and should be 120 degrees apart from each other. The base is perpendicular to the height (which is the Z-axis), so it's parallel to the XY plane.

It means that the Z component of the all the base vertices will be the same (i.e. 1.3), and the X and Y components will be points on the 0.1 circle around the origin, at 0, 120 and 240 degrees.

Let's create the prototype spike:

WITH    spike_parameters AS
        (
        SELECT  0.3 AS height, 0.1 AS width
        ),
        spike_origin_vertices AS
        (
        SELECT  1 AS j, (0, 0, 1)::VEC3 AS origin_vertex
        UNION ALL
        SELECT  i + 2 AS j, (0, 0, 1 + height)::VEC3 + (SIN(2 * PI() * i / 3), COS(2 * PI() * i / 3), 0)::VEC3 * width AS origin_vertex
        FROM    spike_parameters
        CROSS JOIN
                GENERATE_SERIES(0, 2) i
        )
SELECT  *
FROM    spike_origin_vertices

Barring the rounding errors, the vertices look about right: the apex is on the sphere, the base vertices are 0.1 units away from the Z-axis, all having the same 1.3 units Z component.

Now we have to take this prototype and graft it on every apex in the Fibonacci lattice we had created earlier. We also need to make sure the base is continuing the radius at the apex, and adjust all the angles accordingly.

One way to do that is to rotate the prototype about the center of the sphere, so that prototype apex becomes the new apex. It's doable, but the formula for calculating this rotation (the Rodrigues' rotation formula) is very bulky, and we would also need an extra step to calculate the axis of the rotation.

However, we can make the rotation much easier, if we do it in two steps. Every apex is a point on the unit sphere, so it will have the radius 1, and some inclination and azimuth: (1, θ, φ). Now, remember how we located our prototype apex at (1, 0, 0)? This means that we can rotate it in two easy, successive steps: first, about the Y-axis by θ (the inclination), then about the Z-axis by φ (the azimuth): (1, 0, 0) → (1, θ, 0) → (1, θ, φ). The rest of the vertices will tag along.

A rotation about a basis axis is much easier (since it's effectively 2-d), and we already know the angles (they come straight from the Fibonacci lattice formula).

Here's the query to clone all the vertices:

WITH    spikes AS
        (
        SELECT  i, spherical.*
        FROM    (
                SELECT  40 AS spikes
                ) constants
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, spikes - 1) AS i
        CROSS JOIN LATERAL
                (
                SELECT  ACOS(2 * i / spikes::DOUBLE PRECISION - 1) AS theta,
                        PI() * (3 - SQRT(5)) * i AS phi
                ) spherical
        ),
        spike_parameters AS
        (
        SELECT  0.3 AS height, 0.1 AS width
        ),
        spike_origin_vertices AS
        (
        SELECT  1 AS j, (0, 0, 1)::VEC3 AS origin_vertex
        UNION ALL
        SELECT  i + 2 AS j, (0, 0, 1 + height)::VEC3 + (SIN(2 * PI() * i / 3), COS(2 * PI() * i / 3), 0)::VEC3 * width AS origin_vertex
        FROM    spike_parameters
        CROSS JOIN
                GENERATE_SERIES(0, 2) i
        ),
        spike_vertices AS
        (
        SELECT  i, rotated
        FROM    spikes
        CROSS JOIN
                spike_origin_vertices
        CROSS JOIN LATERAL
                (
                SELECT  ((origin_vertex).x * COS(theta) + (origin_vertex).z * SIN(theta), (origin_vertex).y, (origin_vertex).z * COS(theta) - (origin_vertex).x * SIN(theta))::VEC3 AS rotated_y
                ) y
        CROSS JOIN LATERAL
                (
                SELECT  ((rotated_y).x * COS(phi) - (rotated_y).y * SIN(phi), (rotated_y).y * COS(phi) + (rotated_y).x * SIN(phi), (rotated_y).z)::VEC3 AS rotated
                ) rotated
        )
SELECT  *
FROM    spike_vertices

For every spike, we have a set of 4 vertices that define it.

Let's combine these vertices into triangles. A triangular pyramid has this nice property that every three of its vertices make a face, so we just need to take all 3-combinations of 4 vertices.

Here's the query:

WITH    spikes AS
        (
        SELECT  i, spherical.*
        FROM    (
                SELECT  40 AS spikes
                ) constants
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, spikes - 1) AS i
        CROSS JOIN LATERAL
                (
                SELECT  ACOS(2 * i / spikes::DOUBLE PRECISION - 1) AS theta,
                        PI() * (3 - SQRT(5)) * i AS phi
                ) spherical
        ),
        spike_parameters AS
        (
        SELECT  0.3 AS height, 0.1 AS width
        ),
        spike_origin_vertices AS
        (
        SELECT  1 AS j, (0, 0, 1)::VEC3 AS origin_vertex
        UNION ALL
        SELECT  i + 2 AS j, (0, 0, 1 + height)::VEC3 + (SIN(2 * PI() * i / 3), COS(2 * PI() * i / 3), 0)::VEC3 * width AS origin_vertex
        FROM    spike_parameters
        CROSS JOIN
                GENERATE_SERIES(0, 2) i
        ),
        spike_vertices AS
        (
        SELECT  i, j, rotated
        FROM    spikes
        CROSS JOIN
                spike_origin_vertices
        CROSS JOIN LATERAL
                (
                SELECT  ((origin_vertex).x * COS(theta) + (origin_vertex).z * SIN(theta), (origin_vertex).y, (origin_vertex).z * COS(theta) - (origin_vertex).x * SIN(theta))::VEC3 AS rotated_y
                ) y
        CROSS JOIN LATERAL
                (
                SELECT  ((rotated_y).x * COS(phi) - (rotated_y).y * SIN(phi), (rotated_y).y * COS(phi) + (rotated_y).x * SIN(phi), (rotated_y).z)::VEC3 AS rotated
                ) rotated
        ),
        spike_triangles AS
        (
        SELECT  i, sv1.rotated AS one, sv2.rotated AS two, sv3.rotated AS three
        FROM    spike_vertices sv1
        JOIN    spike_vertices sv2
        USING   (i)
        JOIN    spike_vertices sv3
        USING   (i)
        WHERE   sv2.j > sv1.j
                AND sv3.j > sv2.j
        )
SELECT  *
FROM    spike_triangles

Ray tracing

For the ray tracing, we will use the most simple mapping: the pinhole camera model. Every pixel on the resulting image would be lit by exactly one point on the surface of the model. There are no lenses, which means there are no geometric distortions on the image: all straight lines map to straight lines.

In a real pinhole camera, the image plane is located behind the pinhole, and the image is reversed. However, we're dealing with a mathematical model, so we are free to locate the image in front of the pinhole (or the viewpoint). All the mathematics stay the same, but this way, the image is not reversed.

The brightness of every pixel on our image is defined by intensity of light which the corresponding point on the model surface reflects towards this pixel.

For our model, we will be using a single source of light, located quite far away from the scene. We will also assume some amount of ambient light, less bright than the main source, so that we could see the parts of our model which are not being directly illuminated by the source.

Here's what we need to do to calculate the brightness of every pixel:

1. Define if the pixel ray intersects any part of the model. If it doesn't, there is nothing to reflect the light, and the pixel is completely black
2. If the pixel ray intersects several parts of the model, find out which one is the closest. Since it's blocking the light from the other surfaces, the closest surface will be the only one which defines the pixel intensity
3. Find out the amount of light reflected by surface at the intersection point in the direction of the pixel ray.

Intersections

For every pixel, we know the parameters of the pixel ray: it can be defined by a unit vector from the origin (the viewpoint) towards the pixel. We have two types of objects in our model: spheres (one sphere, actually) and triangles. For every type of object, we know its parameters: the radius and the center for the sphere, and the coordinates of all the vertices for the triangles.

So we have everything we need to implement two different algorithms: intersection of a ray with a sphere and intersection of a ray with a triangle.

Intersections with the sphere

To figure out whether or not the ray intersects with the sphere, we need to calculate the distances from the center of the sphere to the ray. This distance is equal to the length of the perpendicular dropped from the center of the sphere. We need to see if its length is more, equal or less than the sphere's radius. If it's more, then the ray does not intersect the sphere. If it's equal, the ray touches the sphere (intersects it in exactly one point). Finally, if it's less, then the ray intersects the sphere in two points.

The intersection point can be presented in the vector form as the origin point (V) plus pixel unit vector (P) times the distance from the viewpoint (d): (V̅ + d * P̅). The difference between this vector and the sphere center vector is another vector, whose module should be equal to the radius of the sphere.

This way, we get a quadratic equation in regard to d: |V̅ + d * P̅ - C̅|2 = R2, or d2 + 2 * d * P̅(V̅ - C̅) + (|V̅ - C̅|2 - R2) = 0

We know that if the discriminant of this equation is less than zero, there is no intersection. We can use this fact to make our query a little bit more efficient: we will calculate the determinant first (we need to do that anyway to solve the equation), and just skip the rows where it's less than zero.

For the pixels whose rays do intersect the sphere, we need to find the closest intersection, because this is the only one which we will need. The farthest one, first, will face away from the pixel, and, second, will be blocked by the closest one. This way we can optimize our query yet a little bit.

Let's find all the intersection points for every pixel.

We'll generate a grid of pixels first, at the distance of 100 units from the origin. For each pixel, we'll calculate a unit vector from the viewpoint.

Then we will find the discriminant of the quadratic equation above and throw away the pixels with the negative discriminant.

We then will solve the quadratic equation above, throwing away all the negative results (not that there's gonna be any). A negative result would mean an intersection behind the pixel grid, and all of the sphere is all in front of the grid. But we will want to keep our query tidy. Every solution would represent the distance to the intersection point.

Finally, if there's more that one solution for a pixel, we select the smallest one, because this one would be the closest intersection.

Here's the query (limiting the results for demonstration purposes):

WITH    origin AS
        (
        SELECT  (0, 0, 0)::VEC3 AS origin
        ),
        sphere AS
        (
        SELECT 120 AS radius, (0, 0, 300)::VEC3 AS center
        ),
        pixels AS
        (
        SELECT  *
        FROM    GENERATE_SERIES(-70, 70) x
        CROSS JOIN
                GENERATE_SERIES(-70, 70) y
        CROSS JOIN LATERAL
                (
                SELECT  ||((x, y, 100)::VEC3) AS pixel_unit
                ) pixel_unit
        ),
        sphere_intersection_coefficients AS
        (
        SELECT  *
        FROM    pixels
        CROSS JOIN
                sphere
        CROSS JOIN
                origin
        CROSS JOIN LATERAL
                (
                SELECT  1 AS a, 2 * pixel_unit * (origin - center) AS b, (|(origin - center))^2 - radius^2 AS c
                ) q3
        CROSS JOIN LATERAL
                (
                SELECT b ^ 2 - 4 * a * c AS discriminant
                ) q4
        WHERE   discriminant > 0
        ),
        sphere_intersections AS
        (
        SELECT  x, y, pixel_unit, intersection_distance
        FROM    sphere_intersection_coefficients
        CROSS JOIN LATERAL
                (
                SELECT  t AS intersection_distance
                FROM    (VALUES (1), (-1)) q (sign)
                CROSS JOIN LATERAL
                        (
                        SELECT  (-b + SQRT(discriminant) * sign) / (2 * a) AS t
                        WHERE   discriminant > 0
                        ) q2
                WHERE   t > 0
                ORDER BY
                        t
                LIMIT   1
                ) q
        )
SELECT  *
FROM    sphere_intersections
LIMIT 10;

We got the pixel unit and distance to every intersection with the sphere, which is enough to reconstruct the cartesian coordinates for the intersection points. We will need them in the future.

Intersections with the triangles

Remember the spikes? We have 40 spikes, each one made of 4 triangles. This means we have 160 triangles. Same as with the sphere, we will need to find the intersections of pixel rays with every one of these triangles.

There is quite an efficient algorithm for finding the intersections of a ray with a triangle. It is called Möller–Trumbore intersection algorithm.

Those interested in the math and principles behind it can read its detailed description on Scratchapixel: Möller-Trumbore algorithm. Scratchapixel is an awesome online book about computer graphics in general and ray tracing in particular.

I will just copy the implementation from Wikipedia and adapt it for SQL. This is quite a straightforward algorithm, with no recursion and almost no branching. It translates to SQL very easily. It's a little bit heavy on math, though.

When we first constructed our spikes, we had modeled them around a unit sphere. This time, we will have to model them around the actual sphere. This is very easy using the vector algebra. We will multiply every spike vertex vector by the radius (enlargement) and add the sphere center vector to it (translation). This is doable just with our basic vector operators, without any fancy matrix calculations. I'll add one more field to the CTE called spike_vertices to do that.

Now that we know the pixel ray parameters and spike parameters, we can just feed them to the algorithm and get the list of intersections back. As in the previous case, it will return us the list of distances to the intersection for every pixel.

Note that unlike the sphere, a ray can have zero, one or infinity intersections with a triangle. Infinity means that the ray lies in the same plane as the triangle and the intersection is not a point, but a line segment. Such triangles would be invisible, as they are facing the viewer with their infinitely thin, non-existing third dimension. We will have to throw away cases like that, which is something that the algorithm takes care of.

For the demonstration purposes, we'll take just one triangle out of just one spike.

Here's the query:

WITH    origin AS
        (
        SELECT  (0, 0, 0)::VEC3 AS origin
        ),
        pixels AS
        (
        SELECT  *
        FROM    GENERATE_SERIES(-70, 70) x
        CROSS JOIN
                GENERATE_SERIES(-70, 70) y
        CROSS JOIN LATERAL
                (
                SELECT  ||((x, y, 100)::VEC3) AS pixel_unit
                ) pixel_unit
        ),
        sphere AS
        (
        SELECT 120 AS radius, (0, 0, 300)::VEC3 AS center
        ),
        spikes AS
        (
        SELECT  i, spherical.*
        FROM    (
                SELECT  40 AS spikes
                ) constants
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, spikes - 1) AS i
        CROSS JOIN LATERAL
                (
                SELECT  ACOS(2 * i / spikes::DOUBLE PRECISION - 1) AS theta,
                        PI() * (3 - SQRT(5)) * i AS phi
                ) spherical
        ),
        spike_parameters AS
        (
        SELECT  0.3 AS height, 0.1 AS width
        ),
        spike_origin_vertices AS
        (
        SELECT  1 AS j, (0, 0, 1)::VEC3 AS origin_vertex
        UNION ALL
        SELECT  i + 2 AS j, (0, 0, 1 + height)::VEC3 + (SIN(2 * PI() * i / 3), COS(2 * PI() * i / 3), 0)::VEC3 * width AS origin_vertex
        FROM    spike_parameters
        CROSS JOIN
                GENERATE_SERIES(0, 2) i
        ),
        spike_vertices AS
        (
        SELECT  i, j, vertex
        FROM    spikes
        CROSS JOIN
                spike_origin_vertices
        CROSS JOIN LATERAL
                (
                SELECT  ((origin_vertex).x * COS(theta) + (origin_vertex).z * SIN(theta), (origin_vertex).y, (origin_vertex).z * COS(theta) - (origin_vertex).x * SIN(theta))::VEC3 AS rotated_y
                ) y
        CROSS JOIN LATERAL
                (
                SELECT  ((rotated_y).x * COS(phi) - (rotated_y).y * SIN(phi), (rotated_y).y * COS(phi) + (rotated_y).x * SIN(phi), (rotated_y).z)::VEC3 AS rotated
                ) rotated
        CROSS JOIN LATERAL
                (
                SELECT  (sphere.radius * rotated) + sphere.center AS vertex
                FROM    sphere
                ) mapped
        ),
        spike_triangles AS
        (
        SELECT  i,
                sv1.j + sv2.j + sv3.j - 5 AS j,
                sv1.vertex AS one, sv2.vertex AS two, sv3.vertex AS three
        FROM    spike_vertices sv1
        JOIN    spike_vertices sv2
        USING   (i)
        JOIN    spike_vertices sv3
        USING   (i)
        WHERE   sv2.j > sv1.j
                AND sv3.j > sv2.j
        ),
        spike_intersections AS
        (
        SELECT  i, j, one, two, three, x, y, pixel_unit, intersection_distance
        FROM    pixels
        CROSS JOIN
                origin
        CROSS JOIN
                spike_triangles
        CROSS JOIN LATERAL
                (
                SELECT  two - one AS edge1, three - one AS edge2
                ) edges
        CROSS JOIN LATERAL
                (
                SELECT  pixel_unit ** edge2 AS p_vector
                ) normal
        CROSS JOIN LATERAL
                (
                SELECT  edge1 * p_vector AS determinant
                ) det
        CROSS JOIN LATERAL
                (
                SELECT  origin - one AS t_vector
                ) t
        CROSS JOIN LATERAL
                (
                SELECT  (t_vector * p_vector) / determinant AS u
                WHERE   determinant NOT BETWEEN -1e-8 AND 1e-8
                ) u
        CROSS JOIN LATERAL
                (
                SELECT  t_vector ** edge1 AS q_vector
                WHERE   u BETWEEN 0 AND 1
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  (pixel_unit * q_vector) / determinant AS v
                ) v
        CROSS JOIN LATERAL
                (
                SELECT  (edge2 * q_vector) / determinant AS intersection_distance
                WHERE   v >= 0 AND u + v <= 1
                ) intersection
        )
SELECT  *
FROM    spike_intersections
WHERE   i = 39
        AND j = 1;

This is just one triangle from the farthest spike from the point of view, and we only got seven intersections (which means that if this triangle is visible at all and not blocked by something else, it will occupy only 7 pixels on the screen). But if we look at the values of x and y of the intersections, we can see that it will have a distinctive triangular shape.

Combining the intersections

For each pixel, we have a list of its ray intersections with different objects (the sphere and the triangles). It is possible that one ray intersects with different objects. In this case, we should get rid of the duplicates and just leave one intersection, the closest one to the viewpoint. This is a usual max-row-per-group task. We can do that by combining the queries with UNION ALL, assigning every row a ROW_NUMBER() with partition by the pixel and ordering by the distance, and getting rid of all records with the row numbers higher than 1.

For demonstration purposes, we will just list the intersection for a narrow 10x2 rectangle in the center of the screen. We will also output the origin of each intersection (sphere or spike triangle), and, if the latter, which spike and which triangle.

WITH    origin AS
        (
        SELECT  (0, 0, 0)::VEC3 AS origin
        ),
        pixels AS
        (
        SELECT  *
        FROM    GENERATE_SERIES(-5, 4) x
        CROSS JOIN
                GENERATE_SERIES(-5, -4) y
        CROSS JOIN LATERAL
                (
                SELECT  ||((x, y, 100)::VEC3) AS pixel_unit
                ) pixel_unit
        ),
        sphere AS
        (
        SELECT 120 AS radius, (0, 0, 300)::VEC3 AS center
        ),
        spikes AS
        (
        SELECT  i, spherical.*
        FROM    (
                SELECT  40 AS spikes
                ) constants
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, spikes - 1) AS i
        CROSS JOIN LATERAL
                (
                SELECT  ACOS(2 * i / spikes::DOUBLE PRECISION - 1) AS theta,
                        PI() * (3 - SQRT(5)) * i AS phi
                ) spherical
        ),
        spike_parameters AS
        (
        SELECT  0.3 AS height, 0.1 AS width
        ),
        spike_origin_vertices AS
        (
        SELECT  1 AS j, (0, 0, 1)::VEC3 AS origin_vertex
        UNION ALL
        SELECT  i + 2 AS j, (0, 0, 1 + height)::VEC3 + (SIN(2 * PI() * i / 3), COS(2 * PI() * i / 3), 0)::VEC3 * width AS origin_vertex
        FROM    spike_parameters
        CROSS JOIN
                GENERATE_SERIES(0, 2) i
        ),
        spike_vertices AS
        (
        SELECT  i, j, vertex
        FROM    spikes
        CROSS JOIN
                spike_origin_vertices
        CROSS JOIN LATERAL
                (
                SELECT  ((origin_vertex).x * COS(theta) + (origin_vertex).z * SIN(theta), (origin_vertex).y, (origin_vertex).z * COS(theta) - (origin_vertex).x * SIN(theta))::VEC3 AS rotated_y
                ) y
        CROSS JOIN LATERAL
                (
                SELECT  ((rotated_y).x * COS(phi) - (rotated_y).y * SIN(phi), (rotated_y).y * COS(phi) + (rotated_y).x * SIN(phi), (rotated_y).z)::VEC3 AS rotated
                ) rotated
        CROSS JOIN LATERAL
                (
                SELECT  (sphere.radius * rotated) + sphere.center AS vertex
                FROM    sphere
                ) mapped
        ),
        spike_triangles AS
        (
        SELECT  i,
                sv1.j + sv2.j + sv3.j - 5 AS j,
                sv1.vertex AS one, sv2.vertex AS two, sv3.vertex AS three
        FROM    spike_vertices sv1
        JOIN    spike_vertices sv2
        USING   (i)
        JOIN    spike_vertices sv3
        USING   (i)
        WHERE   sv2.j > sv1.j
                AND sv3.j > sv2.j
        ),
        spike_intersections AS
        (
        SELECT  i, j, one, two, three, x, y, pixel_unit, intersection_distance
        FROM    pixels
        CROSS JOIN
                origin
        CROSS JOIN
                spike_triangles
        CROSS JOIN LATERAL
                (
                SELECT  two - one AS edge1, three - one AS edge2
                ) edges
        CROSS JOIN LATERAL
                (
                SELECT  pixel_unit ** edge2 AS p_vector
                ) normal
        CROSS JOIN LATERAL
                (
                SELECT  edge1 * p_vector AS determinant
                ) det
        CROSS JOIN LATERAL
                (
                SELECT  origin - one AS t_vector
                ) t
        CROSS JOIN LATERAL
                (
                SELECT  (t_vector * p_vector) / determinant AS u
                WHERE   determinant NOT BETWEEN -1e-8 AND 1e-8
                ) u
        CROSS JOIN LATERAL
                (
                SELECT  t_vector ** edge1 AS q_vector
                WHERE   u BETWEEN 0 AND 1
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  (pixel_unit * q_vector) / determinant AS v
                ) v
        CROSS JOIN LATERAL
                (
                SELECT  (edge2 * q_vector) / determinant AS intersection_distance
                WHERE   v >= 0 AND u + v <= 1
                ) intersection
        ),
        sphere_intersection_coefficients AS
        (
        SELECT  *
        FROM    pixels
        CROSS JOIN
                sphere
        CROSS JOIN
                origin
        CROSS JOIN LATERAL
                (
                SELECT  1 AS a, 2 * pixel_unit * (origin - center) AS b, (|(origin - center))^2 - radius^2 AS c
                ) q3
        CROSS JOIN LATERAL
                (
                SELECT b ^ 2 - 4 * a * c AS discriminant
                ) q4
        WHERE   discriminant > 0
        ),
        sphere_intersections AS
        (
        SELECT  x, y, pixel_unit, intersection_distance
        FROM    sphere_intersection_coefficients
        CROSS JOIN LATERAL
                (
                SELECT  t AS intersection_distance
                FROM    (VALUES (1), (-1)) q (sign)
                CROSS JOIN LATERAL
                        (
                        SELECT  (-b + SQRT(discriminant) * sign) / (2 * a) AS t
                        WHERE   discriminant > 0
                        ) q2
                WHERE   t > 0
                ORDER BY
                        t
                LIMIT   1
                ) q
        ),
        closest_intersections AS
        (
        SELECT  object, i, j, x, y, pixel_unit, intersection_distance
        FROM    (
                SELECT  q.*,
                        ROW_NUMBER() OVER (PARTITION BY x, y ORDER BY intersection_distance) rn
                FROM    (
                        SELECT  'sphere' AS object, 0 AS i, 0 AS j, x, y, pixel_unit, intersection_distance
                        FROM    sphere_intersections
                        UNION ALL
                        SELECT  'spike', i, j, x, y, pixel_unit, intersection_distance
                        FROM    spike_intersections
                        ) q
                ) q
        WHERE   rn = 1
        )
SELECT  *
FROM    closest_intersections
ORDER BY
        x, y

We see that some pixels show the intersection with the sphere, and some with the triangle 4 of the spike 0. When the intersection is with the spike, the distance is lower: the spike protrudes out of the sphere and the pixel ray hits it earlier than it would the sphere.

Reflections

To calculate the pixel intensity at each intersection we will be using the Phong reflection model.

This is a simple, yet quite realistic model which accounts for three kinds of reflection, each one adding to another.

• The ambient reflection is the same for all intersections. It's either there, if there is an intersection, or not. It simulates the scattered light from the walls, ceiling, other objects and stuff like this, which is always there and which is dim and uniform enough to bother with its exact calculation.
• The diffuse reflection, or Lambertian reflection, accounts for the light from the main source scattered, or diffusely reflected by the surface. By the Lambert's cosine law, the amount of scattered light which hits the pixel does not depend on the view point, but it does depend on the angle between the object's surface normal at this point and the source of light.
• The specular reflection is responsible for the "shiny" part of the reflection. Dr. Bui Tuong Phong, the author of the model, had noticed that some glossy surfaces, like marble or still water, behave somewhat in between the rough surfaces and perfect mirrors. They don't reflect all the light at the angle equal to the angle of incidence, but the closer the angle of reflection to the angle of incidence is, the more light they do reflect. A reflection of a lamp on a bowling ball looks larger and smoother than on a polished metal ball.

  The Phong model treats this component of reflected light as proportional to (R̅ ⋅ V̅)α, where R̅ is the reflection unit vector, V̅ is the pixel unit vector and α is the shininess constant. The shinier the object, the larger is this constant, the more sharp is the peak. For a perfect mirror, this component is infinite, all the light is reflected strictly at the angle equal to the angle of incidence, and it looks like a shiny dot on the object. For an object which is merely glossy, the reflection of the light source looks like a blurred spot

Unlike the previous term, this one does not depend explicitly on the surface normal and the direction to the light source, but it does depend implicitly on them, because R̅ depends on both these terms.

The formula for the intensity of light at each point is A + L * Il * (L̅ ⋅ N̅) + S * Is * (R̅ ⋅ V̅)α. A, L and S are reflection constants, I_l and I_s are light intensities for diffuse and specular reflection. They are different because the prevailing kind of reflection can depend on the light's color and polarization.

So, to calculate the intensity of reflected light at each pixel, we need to know the light intensities and the surface parameters (ambient reflection constant, Lambertian reflection constant, specular reflection constant and shininess). We also need to know the direction between the pixel intersection point and the light source. Finally, we need to know the direction of the vector normal to the surface at this point.

The surface parameters we will specify next to the rest of fields in the sphere and spike CTEs. For the light source, we will create another single-record CTE with the parameters (position and intensities).

As for the normal vectors in the intersection points, it makes sense to calculate them at the same time we calculate the intersections. They need to be calculated differently for the sphere and for the triangles.

Strictly speaking, there are two normal vectors to each surface point, differing in sign (because the surface has two sides). However, we can only see one side at each time. This means that we will have to select one normal vector from two, namely the one which gives the positive dot product with the point of view.

This is a problem for the triangles, but this is not a problem for the sphere because of its geometry. We will always have to select the normal vector pointing outside the sphere. A vector pointing inside the sphere would never give a positive dot product with the point of view, because this intersection would be the farthest one of the two, and would be blocked by another one. After all, we can never see the inside of a sphere if we are located outside!

For the sphere, the normal vector is just the unit vector from the center of the sphere to the intersection point.

For a triangle, the normal vector is a unit vector for a cross product of any two of its sides. The sign will have to be selected as described above.

The reflection unit vector can be calculated as 2 * (L̅ ⋅ N̅) * N̅ - L̅.

Let's calculate all these parameters and substitute them into Phong's formula. As before, we'll be only doing this for a small section of the grid.

WITH    origin AS
        (
        SELECT  (0, 0, 0)::VEC3 AS origin
        ),
        light AS
        (
        SELECT  (-5000, 5000, -5000)::VEC3 AS position, 0.7 AS specular, 0.6 AS lambertian
        ),
        pixels AS
        (
        SELECT  *
        FROM    GENERATE_SERIES(-5, 4) x
        CROSS JOIN
                GENERATE_SERIES(-5, -4) y
        CROSS JOIN LATERAL
                (
                SELECT  ||((x, y, 100)::VEC3) AS pixel_unit
                ) pixel_unit
        ),
        sphere AS
        (
        SELECT 120 AS radius, (0, 0, 300)::VEC3 AS center, 0.7 AS specular, 0.6 AS lambertian, 1 AS shininess
        ),
        spikes AS
        (
        SELECT  i, spherical.*
        FROM    (
                SELECT  40 AS spikes
                ) constants
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, spikes - 1) AS i
        CROSS JOIN LATERAL
                (
                SELECT  ACOS(2 * i / spikes::DOUBLE PRECISION - 1) AS theta,
                        PI() * (3 - SQRT(5)) * i AS phi
                ) spherical
        ),
        spike_parameters AS
        (
        SELECT  0.3 AS height, 0.1 AS width, 2 AS specular, 4 AS lambertian, 4 AS shininess
        ),
        spike_origin_vertices AS
        (
        SELECT  1 AS j, (0, 0, 1)::VEC3 AS origin_vertex
        UNION ALL
        SELECT  i + 2 AS j, (0, 0, 1 + height)::VEC3 + (SIN(2 * PI() * i / 3), COS(2 * PI() * i / 3), 0)::VEC3 * width AS origin_vertex
        FROM    spike_parameters
        CROSS JOIN
                GENERATE_SERIES(0, 2) i
        ),
        spike_vertices AS
        (
        SELECT  i, j, vertex
        FROM    spikes
        CROSS JOIN
                spike_origin_vertices
        CROSS JOIN LATERAL
                (
                SELECT  ((origin_vertex).x * COS(theta) + (origin_vertex).z * SIN(theta), (origin_vertex).y, (origin_vertex).z * COS(theta) - (origin_vertex).x * SIN(theta))::VEC3 AS rotated_y
                ) y
        CROSS JOIN LATERAL
                (
                SELECT  ((rotated_y).x * COS(phi) - (rotated_y).y * SIN(phi), (rotated_y).y * COS(phi) + (rotated_y).x * SIN(phi), (rotated_y).z)::VEC3 AS rotated
                ) rotated
        CROSS JOIN LATERAL
                (
                SELECT  (sphere.radius * rotated) + sphere.center AS vertex
                FROM    sphere
                ) mapped
        ),
        spike_triangles AS
        (
        SELECT  i,
                sv1.j + sv2.j + sv3.j - 5 AS j,
                sv1.vertex AS one, sv2.vertex AS two, sv3.vertex AS three
        FROM    spike_vertices sv1
        JOIN    spike_vertices sv2
        USING   (i)
        JOIN    spike_vertices sv3
        USING   (i)
        WHERE   sv2.j > sv1.j
                AND sv3.j > sv2.j
        ),
        spike_intersections AS
        (
        SELECT  i, j, x, y, pixel_unit, intersection_distance, intersection, intersection_norm_unit,
                specular, lambertian, shininess
        FROM    pixels
        CROSS JOIN
                origin
        CROSS JOIN
                spike_triangles
        CROSS JOIN
                spike_parameters
        CROSS JOIN LATERAL
                (
                SELECT  two - one AS edge1, three - one AS edge2
                ) edges
        CROSS JOIN LATERAL
                (
                SELECT  pixel_unit ** edge2 AS p_vector
                ) normal
        CROSS JOIN LATERAL
                (
                SELECT  edge1 * p_vector AS determinant
                ) det
        CROSS JOIN LATERAL
                (
                SELECT  origin - one AS t_vector
                ) t
        CROSS JOIN LATERAL
                (
                SELECT  (t_vector * p_vector) / determinant AS u
                WHERE   determinant NOT BETWEEN -1e-8 AND 1e-8
                ) u
        CROSS JOIN LATERAL
                (
                SELECT  t_vector ** edge1 AS q_vector
                WHERE   u BETWEEN 0 AND 1
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  (pixel_unit * q_vector) / determinant AS v
                ) v
        CROSS JOIN LATERAL
                (
                SELECT  (edge2 * q_vector) / determinant AS intersection_distance
                WHERE   v >= 0 AND u + v <= 1
                ) distance
        CROSS JOIN LATERAL
                (
                SELECT  origin + (pixel_unit * intersection_distance) AS intersection
                FROM    origin
                ) intersection
        CROSS JOIN LATERAL
                (
                SELECT  ||(edge1 ** edge2) AS intersection_norm_unit_oriented
                ) norm_oriented
        CROSS JOIN LATERAL
                (
                SELECT  intersection_norm_unit_oriented * -SIGN(pixel_unit * intersection_norm_unit_oriented) AS intersection_norm_unit
                ) norm
        ),
        sphere_intersection_coefficients AS
        (
        SELECT  *
        FROM    pixels
        CROSS JOIN
                sphere
        CROSS JOIN
                origin
        CROSS JOIN LATERAL
                (
                SELECT  1 AS a, 2 * pixel_unit * (origin - center) AS b, (|(origin - center))^2 - radius^2 AS c
                ) q3
        CROSS JOIN LATERAL
                (
                SELECT b ^ 2 - 4 * a * c AS discriminant
                ) q4
        WHERE   discriminant > 0
        ),
        sphere_intersections AS
        (
        SELECT  x, y, pixel_unit, intersection_distance, intersection, intersection_norm_unit,
                specular, lambertian, shininess
        FROM    sphere_intersection_coefficients
        CROSS JOIN LATERAL
                (
                SELECT  t AS intersection_distance
                FROM    (VALUES (1), (-1)) q (sign)
                CROSS JOIN LATERAL
                        (
                        SELECT  (-b + SQRT(discriminant) * sign) / (2 * a) AS t
                        WHERE   discriminant > 0
                        ) q2
                WHERE   t > 0
                ORDER BY
                        t
                LIMIT   1
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  origin + (pixel_unit * intersection_distance) AS intersection
                FROM    origin
                ) intersection
        CROSS JOIN LATERAL
                (
                SELECT  ||(intersection - center) AS intersection_norm_unit
                FROM    sphere
                ) q3
        ),
        closest_intersections AS
        (
        SELECT  *
        FROM    (
                SELECT  q.*,
                        ROW_NUMBER() OVER (PARTITION BY x, y ORDER BY intersection_distance) rn
                FROM    (
                        SELECT  'sphere' AS object, 0 AS i, 0 AS j, *
                        FROM    sphere_intersections
                        UNION ALL
                        SELECT  'spike', *
                        FROM    spike_intersections
                        ) q
                ) q
        WHERE   rn = 1
        ),
        intensities AS
        (
        SELECT  object, i, j, x, y, intersection, intersection_norm_unit, intensity
        FROM    closest_intersections
        CROSS JOIN
                light
        CROSS JOIN LATERAL
                (
                SELECT  ||(light.position - intersection) AS light_unit
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  2 * (light_unit * intersection_norm_unit) * intersection_norm_unit - light_unit AS reflection_unit
                ) q2
        CROSS JOIN LATERAL
                (
                SELECT  GREATEST(light_unit * intersection_norm_unit, 0) AS norm_term
                ) q3
        CROSS JOIN LATERAL
                (
                SELECT  0.125 +
                        closest_intersections.lambertian * light.lambertian * norm_term +
                        closest_intersections.specular * light.specular * CASE WHEN norm_term > 0 THEN GREATEST(reflection_unit * -pixel_unit, 0) ^ shininess ELSE 0 END +
                        0 AS intensity
                ) q4
        )
SELECT  *
FROM    intensities
ORDER BY
        x, y

We can see that the normal vector to the spike triangle is the same for all the points. This is expected for a flat surface. The intensity values differ ever so slightly for the points on the spike triangle. Also, the spike is much more shiny than the sphere, because of the way the constants are picked (for aesthetic reasons).

The picture

Now that we have all the pieces together, let's build the picture of the dreaded coronavirus.

Note that this query takes quite a time (about 3 minutes on my machine), but hey, that's ray tracing in SQL! As Dr. Kajiya had put it in one of his papers, "ray tracing isn't too slow; computers are too slow".

Here's the query:

WITH    origin AS
        (
        SELECT  (0, 0, 0)::VEC3 AS origin
        ),
        light AS
        (
        SELECT  (-5000, 5000, -5000)::VEC3 AS position, 0.7 AS specular, 0.6 AS lambertian
        ),
        pixels AS
        (
        SELECT  *
        FROM    GENERATE_SERIES(-70, 70) x
        CROSS JOIN
                GENERATE_SERIES(-70, 70) y
        CROSS JOIN LATERAL
                (
                SELECT  ||((x, y, 100)::VEC3) AS pixel_unit
                ) pixel_unit
        ),
        sphere AS
        (
        SELECT 120 AS radius, (0, 0, 300)::VEC3 AS center, 0.7 AS specular, 0.6 AS lambertian, 1 AS shininess
        ),
        spikes AS
        (
        SELECT  i, spherical.*
        FROM    (
                SELECT  40 AS spikes
                ) constants
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, spikes - 1) AS i
        CROSS JOIN LATERAL
                (
                SELECT  ACOS(2 * i / spikes::DOUBLE PRECISION - 1) AS theta,
                        PI() * (3 - SQRT(5)) * i AS phi
                ) spherical
        ),
        spike_parameters AS
        (
        SELECT  0.3 AS height, 0.1 AS width, 2 AS specular, 4 AS lambertian, 4 AS shininess
        ),
        spike_origin_vertices AS
        (
        SELECT  1 AS j, (0, 0, 1)::VEC3 AS origin_vertex
        UNION ALL
        SELECT  i + 2 AS j, (0, 0, 1 + height)::VEC3 + (SIN(2 * PI() * i / 3), COS(2 * PI() * i / 3), 0)::VEC3 * width AS origin_vertex
        FROM    spike_parameters
        CROSS JOIN
                GENERATE_SERIES(0, 2) i
        ),
        spike_vertices AS
        (
        SELECT  i, j, vertex
        FROM    spikes
        CROSS JOIN
                spike_origin_vertices
        CROSS JOIN LATERAL
                (
                SELECT  ((origin_vertex).x * COS(theta) + (origin_vertex).z * SIN(theta), (origin_vertex).y, (origin_vertex).z * COS(theta) - (origin_vertex).x * SIN(theta))::VEC3 AS rotated_y
                ) y
        CROSS JOIN LATERAL
                (
                SELECT  ((rotated_y).x * COS(phi) - (rotated_y).y * SIN(phi), (rotated_y).y * COS(phi) + (rotated_y).x * SIN(phi), (rotated_y).z)::VEC3 AS rotated
                ) rotated
        CROSS JOIN LATERAL
                (
                SELECT  (sphere.radius * rotated) + sphere.center AS vertex
                FROM    sphere
                ) mapped
        ),
        spike_triangles AS
        (
        SELECT  sv1.vertex AS one, sv2.vertex AS two, sv3.vertex AS three
        FROM    spike_vertices sv1
        JOIN    spike_vertices sv2
        USING   (i)
        JOIN    spike_vertices sv3
        USING   (i)
        WHERE   sv2.j > sv1.j
                AND sv3.j > sv2.j
        ),
        spike_intersections AS
        (
        SELECT  x, y, pixel_unit, intersection_distance, intersection, intersection_norm_unit,
                specular, lambertian, shininess
        FROM    pixels
        CROSS JOIN
                origin
        CROSS JOIN
                spike_triangles
        CROSS JOIN
                spike_parameters
        CROSS JOIN LATERAL
                (
                SELECT  two - one AS edge1, three - one AS edge2
                ) edges
        CROSS JOIN LATERAL
                (
                SELECT  pixel_unit ** edge2 AS p_vector
                ) normal
        CROSS JOIN LATERAL
                (
                SELECT  edge1 * p_vector AS determinant
                ) det
        CROSS JOIN LATERAL
                (
                SELECT  origin - one AS t_vector
                ) t
        CROSS JOIN LATERAL
                (
                SELECT  (t_vector * p_vector) / determinant AS u
                WHERE   determinant NOT BETWEEN -1e-8 AND 1e-8
                ) u
        CROSS JOIN LATERAL
                (
                SELECT  t_vector ** edge1 AS q_vector
                WHERE   u BETWEEN 0 AND 1
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  (pixel_unit * q_vector) / determinant AS v
                ) v
        CROSS JOIN LATERAL
                (
                SELECT  (edge2 * q_vector) / determinant AS intersection_distance
                WHERE   v >= 0 AND u + v <= 1
                ) distance
        CROSS JOIN LATERAL
                (
                SELECT  origin + (pixel_unit * intersection_distance) AS intersection
                FROM    origin
                ) intersection
        CROSS JOIN LATERAL
                (
                SELECT  ||(edge1 ** edge2) AS intersection_norm_unit_oriented
                ) norm_oriented
        CROSS JOIN LATERAL
                (
                SELECT  intersection_norm_unit_oriented * -SIGN(pixel_unit * intersection_norm_unit_oriented) AS intersection_norm_unit
                ) norm
        ),
        sphere_intersection_coefficients AS
        (
        SELECT  *
        FROM    pixels
        CROSS JOIN
                sphere
        CROSS JOIN
                origin
        CROSS JOIN LATERAL
                (
                SELECT  1 AS a, 2 * pixel_unit * (origin - center) AS b, (|(origin - center))^2 - radius^2 AS c
                ) q3
        CROSS JOIN LATERAL
                (
                SELECT b ^ 2 - 4 * a * c AS discriminant
                ) q4
        WHERE   discriminant > 0
        ),
        sphere_intersections AS
        (
        SELECT  x, y, pixel_unit, intersection_distance, intersection, intersection_norm_unit,
                specular, lambertian, shininess
        FROM    sphere_intersection_coefficients
        CROSS JOIN LATERAL
                (
                SELECT  t AS intersection_distance
                FROM    (VALUES (1), (-1)) q (sign)
                CROSS JOIN LATERAL
                        (
                        SELECT  (-b + SQRT(discriminant) * sign) / (2 * a) AS t
                        WHERE   discriminant > 0
                        ) q2
                WHERE   t > 0
                ORDER BY
                        t
                LIMIT   1
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  origin + (pixel_unit * intersection_distance) AS intersection
                FROM    origin
                ) intersection
        CROSS JOIN LATERAL
                (
                SELECT  ||(intersection - center) AS intersection_norm_unit
                FROM    sphere
                ) q3
        ),
        closest_intersections AS
        (
        SELECT  *
        FROM    (
                SELECT  q.*,
                        ROW_NUMBER() OVER (PARTITION BY x, y ORDER BY intersection_distance) rn
                FROM    (
                        SELECT  *
                        FROM    sphere_intersections
                        UNION ALL
                        SELECT  *
                        FROM    spike_intersections
                        ) q
                ) q
        WHERE   rn = 1
        ),
        intensities AS
        (
        SELECT  x, y, intensity
        FROM    closest_intersections
        CROSS JOIN
                light
        CROSS JOIN LATERAL
                (
                SELECT  ||(light.position - intersection) AS light_unit
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  2 * (light_unit * intersection_norm_unit) * intersection_norm_unit - light_unit AS reflection_unit
                ) q2
        CROSS JOIN LATERAL
                (
                SELECT  GREATEST(light_unit * intersection_norm_unit, 0) AS norm_term
                ) q3
        CROSS JOIN LATERAL
                (
                SELECT  0.125 +
                        closest_intersections.lambertian * light.lambertian * norm_term +
                        closest_intersections.specular * light.specular * CASE WHEN norm_term > 0 THEN GREATEST(reflection_unit * -pixel_unit, 0) ^ shininess ELSE 0 END +
                        0 AS intensity
                ) q4
        )
SELECT  STRING_AGG(dot, '' ORDER BY x) AS picture
FROM    pixels p
LEFT JOIN
        intensities i
USING   (x, y)
CROSS JOIN LATERAL
        (
        SELECT  ' .:—=+*#%@' AS palette
        ) palette
CROSS JOIN LATERAL
        (
        SELECT  SUBSTRING(palette, LEAST(FLOOR(COALESCE(intensity, 0) * LENGTH(palette))::INT + 1, LENGTH(palette)), 1) AS dot
        ) dots
GROUP BY
        y
ORDER BY
        y

You can view the queries here: https://github.com/quassnoi/explain-extended-2021

In the New Year, I wish all of us to stay safe, get vaccinated and not let the filthy beast affect our lives anymore!

Previous New Year posts:

• Happy New 2010 Year!
• Happy New 2011 Year!
• Happy New 2012 Year!
• Happy New 2013 Year!
• Happy New 2014 Year!
• Happy New 2015 Year!
• Happy New 2016 Year!
• Happy New 2017 Year!
• Happy New 2018 Year!
• Happy New 2019 Year!
• Happy New 2020 Year!

The post Happy New Year: 3D picture of the coronavirus in SQL appeared first on EXPLAIN EXTENDED.

One of its most beautiful places is the Everglades: the endless sea of grass, extending to the horizon, as far as the eye can see and beyond, and teeming with life.

There are all kinds of animals there. Herons, egrets, anhingas; fish, turtles, snakes; otters, skunks, small rodents; and of course, the king of Florida's wetlands, the American alligator.

The alligator is a well-oiled killing machine. It's motionless and extremely energy efficient when resting, but deadly fast when hunting. When the alligator is hunting, its eyes instantly track the faintest motion — and a fierce jump in any direction will follow immediately.

I was on a guided tour, and the tour guide mentioned that the alligators have binocular vision. The fields of view of their two eyes overlap, giving the predator the ability to estimate the direction and the distance to its prey more accurately.

We humans also have binocular vision. It allows us to see the world in three dimensions. When we are looking at an object with our two eyes, each eye sees it at a slightly different angle. The closer the thing is to us, the more the difference. This effect is called binocular parallax, and our brain can use it to estimate the distance to the object.

There are ways to trick the brain into believing something is 3D while it's not. To do this, we need a way to project a different image into each eye. There are lots of ways to do that: think holograms, polarized glasses, tilt cards, and many more.

Most of those methods require special equipment and materials, either to see or to produce the image. Maybe even both.

However, there is a way to see a three-dimensional image even on a simple piece of paper (or a plain LCD monitor without any 3D capabilities). It is technically called an autostereogram but most people know them as Magic Eye pictures.

A picture like this looks like repeating patterns of random dots or characters. The frequency of the patterns encodes the three-dimensional image: the close is the part of the image to the observer, the more frequent are the patterns.

It takes some effort to see the depth in what at first seems to be a random dot pattern. Not everyone can do that on the first try. There are lots of resources online which teach how to do that. The good thing is it's like riding a bicycle: once you got it right for the first time, there's no going back, it's always there with you.

Ever since I was a kid, I have been fascinated by the stereograms. So the moment I heard the words "binocular vision" from the tour guide, I instantly knew what would this New Year post be about.

Let's make a stereogram in PostgreSQL!

Stereogram theory

ASCII characters are almost as good as the color dots for the stereograms. All you have to do is combine them into repeating patterns. It does not matter if it's color dots or symbols. Ideally, they should be random and have different shapes within the pattern, but it's not necessary.

The go-to resource on ASCII stereograms is Gary Beene's page. There is a detailed description of the principles behind them, some examples, and even a code snippet in PowerBASIC to generate them. The only thing we need is translating it from PowerBASIC to SQL.

To create a stereogram, we first need an image to encode. Let's generate a simple depth mask: two overlapping rectangles. A larger number means closer to us.

WITH    parameters AS
        (
        SELECT  *
        FROM    (
                VALUES
                (112, 50)
                ) v (width, height)
        ),
        mask AS
        (
        SELECT  x, y,
                CASE
                WHEN x BETWEEN 40 AND 90 AND y BETWEEN 15 AND 35 THEN 4
                WHEN x BETWEEN 15 AND 65 AND y BETWEEN 10 AND 30 THEN 2
                ELSE 0 END AS depth
        FROM    parameters
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, height - 1) y
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, width - 1) x
        )
SELECT  STRING_AGG(COALESCE(depth::TEXT, ' '), '' ORDER BY x)
FROM    parameters
CROSS JOIN LATERAL
        GENERATE_SERIES(0, height - 1) y
CROSS JOIN LATERAL
        GENERATE_SERIES(0, width - 1) x
JOIN    mask
USING   (x, y)
GROUP BY
        y
ORDER BY
        y;

The depth map here is quite simple, and it's easy to see what's going on here: zero means the background level, two is closer to us, four is yet closer.

Random patterns

Now, let's generate a character map of random patterns. Every pattern will be 16 characters wide, and there will be 7 of them.

We will be seeding our random number generator with a constant value so that the random patterns will be the same between query invocations. In a short while, this will allow us to see what's happening with the 3D pictures. For now, we'll generate a plain background.

To do this, we will first generate our "lines" with a GENERATE_SERIES function and then, for each line, call another GENERATE_SERIES to generate a random pattern for each line.

Note that we are using 0 * y instead of just 0 as a first parameter to GENERATE_SERIES. It is because PostgreSQL optimizer is trying to be smart and will only invoke the lateral query once if it sees that it does not depend on the outermost query.

Usually, it's a good thing, but in this case, we do need the engine to invoke that RANDOM for each separate value of y, and that's how we introduce that external dependency (which does not do that much as it's a multiplication by zero).

Here's what our background looks like:

SELECT  SETSEED(0.20191231);

WITH    RECURSIVE
        parameters AS
        (
        SELECT  *
        FROM    (
                VALUES
                (16, 7, 50)
                ) v (pattern_length, pattern_count, height)
        CROSS JOIN LATERAL
                (
                SELECT  pattern_length * pattern_count AS width
                ) q
        ),
        patterns AS
        (
        SELECT  y, pattern
        FROM    parameters
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, height - 1) y
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(CHR(FLOOR(RANDOM() * 94)::INT + 33), '')
                FROM    GENERATE_SERIES(0 * y, pattern_length - 1) x
                ) q (pattern)
        )
SELECT  REPEAT(pattern, pattern_count) AS line
FROM    patterns
CROSS JOIN
        parameters
ORDER BY
        y;

At first glance, it looks completely random, but if you look close enough, you see that the patterns are repeating horizontally every 16 characters.

Adding depth

Now, the fun part. On the background plane, our patterns are 16 characters long. We need them to have a shorter period where the depth map value is larger than 0. It means that they have to be 14 characters long on the background rectangle (which has the depth of 2) and 12 characters long on the foreground rectangle (which has the depth of 4).

The algorithm on Gary's page describes how to do that.

We will have to convert our query to a recursive one so that we can keep track of the pattern string for each line as we build it.

On each step, we will calculate the period of our pattern (according to the value of the depth map at this point), then we will take N'th character from the end of the string we've built so far (where N is the current value of the period) and append it to the end of the line.

The pattern will stay periodic as long as the depth remains the same.

When the depth increases, the pattern will lose some of its symbols (just in the right place).

When the depth decreases, the pattern will take the extra symbols from its beginning.

It will not be the original pattern anymore, of course, but it was random in the first place, so it does not matter much, as long as it's still periodic.

To have a better look at this algorithm, let's replace our random characters with hexadecimal digits and see what's going on here:

SELECT  SETSEED(0.20191231);

WITH    RECURSIVE
        parameters AS
        (
        SELECT  *
        FROM    (
                VALUES
                (16, 7, 50)
                ) v (pattern_length, pattern_count, height)
        CROSS JOIN LATERAL
                (
                SELECT  pattern_length * pattern_count AS width
                ) q
        ),
        patterns AS
        (
        SELECT  y, pattern
        FROM    parameters
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, height - 1) y
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(CHR(CASE WHEN x < 10 THEN x + 48 ELSE x + 55 END), '')
                FROM    GENERATE_SERIES(0 * y, pattern_length - 1) x
                ) q (pattern)
        ),
        mask AS
        (
        SELECT  x, y,
                CASE
                WHEN x BETWEEN 40 AND 90 AND y BETWEEN 15 AND 35 THEN 4
                WHEN x BETWEEN 15 AND 65 AND y BETWEEN 10 AND 30 THEN 2
                ELSE 0 END AS depth
        FROM    parameters
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, height - 1) y
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, width - 1) x
        ),
        lines AS
        (
        SELECT  0 AS x, y, pattern AS line
        FROM    patterns
        CROSS JOIN
                parameters
        UNION ALL
        SELECT  x + 1, y, line || LEFT(RIGHT(line, pattern_length - depth), 1)
        FROM    lines
        JOIN    mask
        USING   (x, y)
        CROSS JOIN
                parameters
        WHERE   x < width
        )
SELECT  SUBSTR(line, pattern_length + 1, width) AS line
FROM    lines
CROSS JOIN
        parameters
WHERE   x = width
ORDER BY
        y;

The first ten lines are perfectly periodic from 0 to F.

On line 11 and below, however, something weird is happening: starting from the very first period, the F and the 0 vanish from the pattern.

It goes on for some time from 1 to E (with a period of 14), and then it makes recovery of sorts and gets its original length of 16 back.

However, it's not that beautiful series of consecutive digits anymore. The 0 and the F are gone forever, and 8 and 9 are inserted back into the pattern instead. It is the artifact of the algorithm. However, if we go back to the random sequences of characters, it does not matter much.

You have probably noticed that the shapes of the rectangles are discernible in the picture above. It is because it already is a complete, functioning stereogram!

But the fact that the patterns are repeating vertically, as well as horizontally, confuses the brain. The picture works much better with a random pattern of dots for every line.

Final result

Let's combine the last two queries and see the result:

SELECT  SETSEED(0.20191231);

WITH    RECURSIVE
        parameters AS
        (
        SELECT  *
        FROM    (
                VALUES
                (16, 7, 50)
                ) v (pattern_length, pattern_count, height)
        CROSS JOIN LATERAL
                (
                SELECT  pattern_length * pattern_count AS width
                ) q
        ),
        patterns AS
        (
        SELECT  y, pattern
        FROM    parameters
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, height - 1) y
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(CHR(FLOOR(RANDOM() * 94)::INT + 33), '')
                FROM    GENERATE_SERIES(0 * y, pattern_length - 1) x
                ) q (pattern)
        ),
        mask AS
        (
        SELECT  x, y,
                CASE
                WHEN x BETWEEN 40 AND 90 AND y BETWEEN 15 AND 35 THEN 4
                WHEN x BETWEEN 15 AND 65 AND y BETWEEN 10 AND 30 THEN 2
                ELSE 0 END AS depth
        FROM    parameters
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, height - 1) y
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, width - 1) x
        ),
        lines AS
        (
        SELECT  0 AS x, y, pattern AS line
        FROM    patterns
        CROSS JOIN
                parameters
        UNION ALL
        SELECT  x + 1, y, line || LEFT(RIGHT(line, pattern_length - depth), 1)
        FROM    lines
        JOIN    mask
        USING   (x, y)
        CROSS JOIN
                parameters
        WHERE   x < width
        )
SELECT  SUBSTR(line, pattern_length + 1, width) AS line
FROM    lines
CROSS JOIN
        parameters
WHERE   x = width
ORDER BY
        y;

If you're still having problems seeing the 3D image, move the mouse cursor over the picture and back to highlight the differences between the original background and the stereogram.

Hidden message

I've made one more stereogram, with a hidden message. Can you see what it is?

SELECT  SETSEED(0.20191231);

WITH    RECURSIVE
        parameters AS
        (
        SELECT  *
        FROM    (
                VALUES
                (16, 7, 50)
                ) v (pattern_length, pattern_count, height)
        CROSS JOIN LATERAL
                (
                SELECT  pattern_length * pattern_count AS width
                ) q
        ),
        patterns AS
        (
        SELECT  y, pattern
        FROM    parameters
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, height - 1) y
        CROSS JOIN LATERAL
                (
                SELECT  STRING_AGG(CHR(FLOOR(RANDOM() * 94)::INT + 33), '')
                FROM    GENERATE_SERIES(0 * y, pattern_length - 1) x
                ) q (pattern)
        ),
        mask AS
        (
        SELECT  x, y,
                CASE
                WHEN y > 4
                     AND SUBSTRING(SPLIT_PART(banner, E'\n',
                                   FLOOR(y / 4)::INT), FLOOR(x / 12)::INT, 1) = '*'
                THEN 2
                ELSE 0
                END depth
        FROM    parameters
        CROSS JOIN
                (
                SELECT  CONVERT_FROM(DECODE('
                KioqICoqKgogICogKiAqCioqKiAqICoK
                KiAgICogKgoqKiogKioqCgoqKiogKioqCiAgKiAqICoK
                KioqICogKgoqICAgKiAqCioqKiAqKioK',
                'BASE64'), 'UTF8') AS banner
                ) q
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, height - 1) y
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, width - 1) x
        ),
        lines AS
        (
        SELECT  0 AS x, y, pattern AS line
        FROM    patterns
        CROSS JOIN
                parameters
        UNION ALL
        SELECT  x + 1, y, line || LEFT(RIGHT(line, pattern_length - depth), 1)
        FROM    lines
        JOIN    mask
        USING   (x, y)
        CROSS JOIN
                parameters
        WHERE   x < width
        )
SELECT  SUBSTR(line, pattern_length + 1, width) AS line
FROM    lines
CROSS JOIN
        parameters
WHERE   x = width
ORDER BY
        y;

You can view the queries here: https://github.com/quassnoi/explain-extended-2020

Let your life in this New Year be full and solid!

Previous New Year posts:

• Happy New 2010 Year!
• Happy New 2011 Year!
• Happy New 2012 Year!
• Happy New 2013 Year!
• Happy New 2014 Year!
• Happy New 2015 Year!
• Happy New 2016 Year!
• Happy New 2017 Year!
• Happy New 2018 Year!
• Happy New 2019 Year!

The post Happy New Year: a stereogram in SQL appeared first on EXPLAIN EXTENDED.

As I was preparing this year's entry, I found myself all out of ideas. What image shall we be generating this year? A cat? A pig? A winter scenery? It's all doable in SQL (or course), however I couldn't make myself pick anything in particular. And frankly speaking, even with all the power of SQL at my hands, I'm a lousy artist.

Then an idea struck me. Why try and create art when there's so many excellent artists out there on the Internet, whose work I could just steal put to fair use? And my phone camera makes better pictures than I could ever aspire to create myself.

Images that come out of the camera or from the Internet are all digital and compressed. Digital is of course a good thing when it comes to computer processing, but compressed is a challenge. And challenges is something I like.

So this year, we will be creating a GIF decoder in SQL.

GIF is one of the earliest compressed image storage formats, famous for its early adoption by the World Wide Web and for being named with an acronym no one can agree how to pronounce correctly. At its core lies LZW, a lossless compression algorithm which uses dictionary tables to encode repeating patterns of data. GIF is not the best format out there, of course, and it has fallen out of use in the last years (or even decades). Its algorithm, however, is not particularly memory or CPU intensive and as the image compression algorithms go, its implementation is quite simple. In other words, it makes a perfect SQL exercise.

To become familiar with GIF, I used an excellent GIF tutorial called What's in a GIF? by Eric. S Raymond and Mike Flickinger. This tutorial is built around explanation of contents of a tiny sample GIF file:

and this file is what we will be using during the first part of our journey.

Let's get started!

Feeding a binary file to the database

As usual, we will be using the latest version of PostgreSQL for this, because of its excellent support of recursive queries, rowset generators, lateral joins and complex types. The first is an absolute must for anything beyond the most simple tasks, the rest make even the complex tasks easier and more readable.

Unfortunately, we cannot easily use binary files with psql directly. We will have to recourse to a little bit of shell magic: we will hex-encode the file and pipe it to the \copy command, which will insert its contents into a temporary table.

We will need two scripts for that:

pre.sql

CREATE TEMPORARY TABLE
        input
        (
        data TEXT
        );

01.bin.sql

WITH    RECURSIVE
        bin AS
        (
        SELECT  DECODE(data, 'HEX') AS buffer
        FROM    input
        )
SELECT  ENCODE(buffer, 'HEX')
FROM    bin;

, which we will be calling in a single shell command:

cat sample_1.gif | od -A n -vt x1 | tr -d '\n ' | psql -f pre.sql -c "COPY input FROM stdin" -f 01.bin.sql

Here we see the hex-encoded content of the sample GIF file in all its entirety. Note that we have put it into a BYTEA field in a temporary table. This will allow working with the parts of the file more easily than if it would be a hex-encoded string.

Let's try and parse the header of the file. To do this, we will be using BIT operators and BYTEA functions:

02.header.sql

WITH    RECURSIVE
        bin AS
        (
        SELECT  DECODE(data, 'HEX') AS buffer
        FROM    input
        ),
        header AS
        (
        SELECT  h.*, io.*
        FROM    bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, 10)::BIT(8) AS flags
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CONVERT_FROM(SUBSTR(buffer, 1, 6), 'LATIN1') AS version,
                        (GET_BYTE(buffer, 7)::BIT(8) || GET_BYTE(buffer, 6)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, 9)::BIT(8) || GET_BYTE(buffer, 8)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = B'1' AS gtc,
                        (flags << 1)::BIT(3)::INT + 1 AS depth,
                        (flags << 4) = B'1' AS color_sort,
                        2 << (flags << 5)::BIT(3)::INT AS gct_size,
                        GET_BYTE(buffer, 11)::INT AS bci,
                        GET_BYTE(buffer, 12)::INT AS aspect
                ) h
        CROSS JOIN LATERAL
                (
                SELECT  13 + 3 * gct_size AS blocks_offset
                ) io
        )
SELECT  *
FROM    header;

This is some basic information about the GIF file.

version, width and height are self-explanatory.

gct,depth, color_sort and gct_size are related to the image palette. The GIF standard allows encoding any colors from 24-bit RGB color space, but no more than 256 of them at the same time. The set of 24-bit colors used within a single image is called a palette. Palettes are the reason GIF pictures look grainy on the screen: there are just not enough colors available in the palette to encode the smooth transitions. It was enough for most screens back in the day, though, and this is more than enough for the sample file which only has four colors.

To define a palette, we need to know how many different colors are used in the file and what are their values in the color space. This information is stored in a color table. Static GIF files mostly use a Global Color Table for that. There can also be local color tables, but we won't be dealing with them in this post. So, gct is a flag showing if a Global Color Table is being used (spoiler: it is). depth is the number of colors in the palette (as a base-2 logarithm, so 2 means 2² = 4 colors). color_sort is a legacy flag which, if set, means that the colors are sorted from the "most important" to the "least important", so if a decoder or a rendering device is really short on memory, it can choose to only display a selected subset of colors. Finally, gct_size is the size of the Global Color Table in 3-byte chunks, one per color.

The rest of header fields deals with interlacing, pixel aspect ratio and canvas color. Those are of no importance to us.

The Global Color Table immediately follows the header, and we will get back to it later. Now we have to find where the real fun begins. And blocks_offset is a calculated field which brings us past the GCT into the most interesting parts of the file.

Blocks

Data in a GIF file is composed of blocks. Those can be images, captions, comments and the designers have even left room for other extensions, although the chances are low we will see new versions of the GIF standard any time soon. We are interested in image blocks, and we have to local the first image block to work with it.

An image block always goes after a so called Graphic Extension block. All extension blocks start with a value 0x21 (called extension introducer), followed by a byte-long extension type and a byte-long block length. Every extension block ends with a value of 0x00. The image block itself starts with a value 0x2C.

To track the beginning of an image block, we need to recursively read through all the blocks, skipping over those which start with 0x21, and stopping when we find a 0x2C.

To do that, we will use a recursive CTE, which would track the offsets, starting from the first one past the Global Color Table, look at the value at the offset, and if this value is 0x21, skip to the next block (block length at offset + 2, plus 3-byte header plus 1-byte terminator). This stops when the value at the offset is 0x2C (image data).

03.image.sql

WITH    RECURSIVE
        bin AS
        (
        SELECT  DECODE(data, 'HEX') AS buffer
        FROM    input
        ),
        header AS
        (
        SELECT  h.*, io.*
        FROM    bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, 10)::BIT(8) AS flags
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CONVERT_FROM(SUBSTR(buffer, 1, 6), 'LATIN1') AS version,
                        (GET_BYTE(buffer, 7)::BIT(8) || GET_BYTE(buffer, 6)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, 9)::BIT(8) || GET_BYTE(buffer, 8)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = B'1' AS gct,
                        (flags << 1)::BIT(3)::INT + 1 AS depth,
                        (flags << 4) = B'1' AS color_sort,
                        2 << (flags << 5)::BIT(3)::INT AS gct_size,
                        GET_BYTE(buffer, 11)::INT AS bci,
                        GET_BYTE(buffer, 12)::INT AS aspect
                ) h
        CROSS JOIN LATERAL
                (
                SELECT  13 + 3 * gct_size AS blocks_offset
                ) io
        ),
        blocks AS
        (
        SELECT  blocks_offset AS current,
                GET_BYTE(buffer, blocks_offset) AS intro
        FROM    bin
        CROSS JOIN
                header
        UNION ALL
        SELECT  current,
                GET_BYTE(buffer, current) AS intro
        FROM    (
                SELECT  current AS previous,
                        intro AS previous_intro
                FROM    blocks
                ) b
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  previous + GET_BYTE(buffer, previous + 2) + 4 AS current
                ) q
        WHERE   previous_intro = x'21'::INT
        ),
        image_offset AS
        (
        SELECT  current AS image_offset
        FROM    blocks
        WHERE   intro = x'2C'::INT
        )
SELECT  *
FROM    image_offset;

This means the image block starts 33 bytes into the file.

Image header

The image header also has a fixed format easily retrievable using a little bit of BIT and BYTEA functionality:

04.image-header.sql

WITH    RECURSIVE
        bin AS
        (
        SELECT  DECODE(data, 'HEX') AS buffer
        FROM    input
        ),
        header AS
        (
        SELECT  h.*, io.*
        FROM    bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, 10)::BIT(8) AS flags
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CONVERT_FROM(SUBSTR(buffer, 1, 6), 'LATIN1') AS version,
                        (GET_BYTE(buffer, 7)::BIT(8) || GET_BYTE(buffer, 6)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, 9)::BIT(8) || GET_BYTE(buffer, 8)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = B'1' AS gct,
                        (flags << 1)::BIT(3)::INT + 1 AS depth,
                        (flags << 4) = B'1' AS color_sort,
                        2 << (flags << 5)::BIT(3)::INT AS gct_size,
                        GET_BYTE(buffer, 11)::INT AS bci,
                        GET_BYTE(buffer, 12)::INT AS aspect
                ) h
        CROSS JOIN LATERAL
                (
                SELECT  13 + 3 * gct_size AS blocks_offset
                ) io
        ),
        blocks AS
        (
        SELECT  blocks_offset AS current,
                GET_BYTE(buffer, blocks_offset) AS intro
        FROM    bin
        CROSS JOIN
                header
        UNION ALL
        SELECT  current,
                GET_BYTE(buffer, current) AS intro
        FROM    (
                SELECT  current AS previous,
                        intro AS previous_intro
                FROM    blocks
                ) b
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  previous + GET_BYTE(buffer, previous + 2) + 4 AS current
                ) q
        WHERE   previous_intro = x'21'::INT
        ),
        image_offset AS
        (
        SELECT  current AS image_offset
        FROM    blocks
        WHERE   intro = x'2C'::INT
        ),
        image_header AS
        (
        SELECT  q.*, l.*, l2.*
        FROM    image_offset
        CROSS JOIN
                bin
        CROSS JOIN
                header h
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_offset + 9)::BIT(8) AS flags
                ) f
        CROSS JOIN LATERAL
                (
                SELECT  (GET_BYTE(buffer, image_offset + 2)::BIT(8) || GET_BYTE(buffer, image_offset + 1)::BIT(8))::BIT(16)::INT AS left,
                        (GET_BYTE(buffer, image_offset + 4)::BIT(8) || GET_BYTE(buffer, image_offset + 3)::BIT(8))::BIT(16)::INT AS top,
                        (GET_BYTE(buffer, image_offset + 6)::BIT(8) || GET_BYTE(buffer, image_offset + 5)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, image_offset + 8)::BIT(8) || GET_BYTE(buffer, image_offset + 7)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = b'1' AS has_lct,
                        (flags << 1)::BIT(1) = b'1' AS interlace,
                        (flags << 2)::BIT(1) = b'1' AS sort
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CASE WHEN has_lct THEN 2 << (flags << 5)::BIT(3)::INT ELSE 0 END AS lct_size
                ) l
        CROSS JOIN LATERAL
                (
                SELECT  image_offset + 10 + lct_size AS image_data_offset
                ) l2
        )
SELECT  *
FROM    image_header;

We have width and height here, the same ones as in the file header (not surprising as this is the only image in the file). The rest of the header fields is also of no immediate use to us (please refer to the tutorial if you're interested in them). They, however, let us calculate image_data_offset, the offset to the actual image data.

Image data

Compressed image data is composed of multiple data sub-blocks within the image block. Every image data sub-block is 1 to 255 bytes long, and it starts with the block length and ends with 0x00, the block terminator. Before the first sub-block, there is one more value, the minimum code size. This is something that LZW compression uses, and we will need this field a little bit later. We need to obtain the compressed image data in a single BYTEA field. To do this, we extract the data from the sub-blocks recursively, ending the recursion when we come across a zero-length sub-block, then concatenated all the extracted chunks of data together:

05.lzw-data.sql

WITH    RECURSIVE
        bin AS
        (
        SELECT  DECODE(data, 'HEX') AS buffer
        FROM    input
        ),
        header AS
        (
        SELECT  h.*, io.*
        FROM    bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, 10)::BIT(8) AS flags
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CONVERT_FROM(SUBSTR(buffer, 1, 6), 'LATIN1') AS version,
                        (GET_BYTE(buffer, 7)::BIT(8) || GET_BYTE(buffer, 6)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, 9)::BIT(8) || GET_BYTE(buffer, 8)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = B'1' AS gct,
                        (flags << 1)::BIT(3)::INT + 1 AS depth,
                        (flags << 4) = B'1' AS color_sort,
                        2 << (flags << 5)::BIT(3)::INT AS gct_size,
                        GET_BYTE(buffer, 11)::INT AS bci,
                        GET_BYTE(buffer, 12)::INT AS aspect
                ) h
        CROSS JOIN LATERAL
                (
                SELECT  13 + 3 * gct_size AS blocks_offset
                ) io
        ),
        blocks AS
        (
        SELECT  blocks_offset AS current,
                GET_BYTE(buffer, blocks_offset) AS intro
        FROM    bin
        CROSS JOIN
                header
        UNION ALL
        SELECT  current,
                GET_BYTE(buffer, current) AS intro
        FROM    (
                SELECT  current AS previous,
                        intro AS previous_intro
                FROM    blocks
                ) b
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  previous + GET_BYTE(buffer, previous + 2) + 4 AS current
                ) q
        WHERE   previous_intro = x'21'::INT
        ),
        image_offset AS
        (
        SELECT  current AS image_offset
        FROM    blocks
        WHERE   intro = x'2C'::INT
        ),
        image_header AS
        (
        SELECT  q.*, l.*, l2.*
        FROM    image_offset
        CROSS JOIN
                bin
        CROSS JOIN
                header h
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_offset + 9)::BIT(8) AS flags
                ) f
        CROSS JOIN LATERAL
                (
                SELECT  (GET_BYTE(buffer, image_offset + 2)::BIT(8) || GET_BYTE(buffer, image_offset + 1)::BIT(8))::BIT(16)::INT AS left,
                        (GET_BYTE(buffer, image_offset + 4)::BIT(8) || GET_BYTE(buffer, image_offset + 3)::BIT(8))::BIT(16)::INT AS top,
                        (GET_BYTE(buffer, image_offset + 6)::BIT(8) || GET_BYTE(buffer, image_offset + 5)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, image_offset + 8)::BIT(8) || GET_BYTE(buffer, image_offset + 7)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = b'1' AS has_lct,
                        (flags << 1)::BIT(1) = b'1' AS interlace,
                        (flags << 2)::BIT(1) = b'1' AS sort
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CASE WHEN has_lct THEN 2 << (flags << 5)::BIT(3)::INT ELSE 0 END AS lct_size
                ) l
        CROSS JOIN LATERAL
                (
                SELECT  image_offset + 10 + lct_size AS image_data_offset
                ) l2
        ),
        image_data AS
        (
        SELECT  GET_BYTE(buffer, image_data_offset) AS code_size,
                image_data_offset + 1 AS image_first_block_offset
        FROM    image_header
        CROSS JOIN
                bin
        ),
        image_blocks AS
        (
        SELECT  image_first_block_offset AS block_offset,
                block_size,
                SUBSTR(buffer, image_first_block_offset + 2, block_size) AS block_data
        FROM    image_data
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_first_block_offset) AS block_size
                ) l
        UNION ALL
        SELECT  new_offset AS block_offset,
                new_block_size AS block_size,
                SUBSTR(buffer, new_offset + 2, new_block_size) AS block_data
        FROM    image_blocks
        CROSS JOIN LATERAL
                (
                SELECT  block_offset + block_size + 1 AS new_offset
                ) no
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, new_offset) AS new_block_size
                ) nbs
        WHERE   new_block_size > 0
        ),
        lzw_data AS
        (
        SELECT  code_size,
                (1 << code_size) AS clear_code,
                (1 << code_size) + 1 AS eof_code,
                compressed
        FROM    (
                SELECT  STRING_AGG(block_data, '' ORDER BY block_offset) compressed
                FROM    image_blocks
                ) i
        CROSS JOIN
                image_data
        )
SELECT  code_size, ENCODE(compressed, 'HEX')
FROM    lzw_data;

There are always at least two sub-blocks in an image block, the last one having zero length. We now see the compressed data and a cryptic number called code_size. We will need it later to do the decompression.

LZW decompression

Here comes the most interesting part of the post, the actual decompression.

LZW is a dictionary algorithm. It tries to find the repeating patterns of colors and assign them a code in a lookup table. Here's what the lookup table for our sample file looks like (I only list first several entries here):

, etc.

Note how this table starts from 6. There is a reason for that.

The sample file only uses four colors, so the first four entries in the lookup table will correspond to those colors directly. This is common for all implementations of LZW.

Now, the first four entries (0 to 3) correspond to the colors. What do 4 and 5 do then? Those are specific to the flavor of LZW used by GIF.

4 (or, generally speaking, 2^palette_size) means "clear code". It does not correspond to any color or a pattern. It acts like a soft-reset switch: when the algorithm comes across this code, it resets its state and clears the lookup table.

5 (or, generally speaking, 2^palette_size + 1) means "EOF code". It's a signal to stop the conversion.

One of the most amazing properties of LZW is that although we need to look up every pattern in the code table for decompression, the code table itself is not anywhere in the file. Instead, it is reconstructed in real time. The compression algorithm always adds codes to the table successively, and does so for every (yet) unknown pattern it comes across. So does the decompression algorithm. The first entry it adds to the table is always
2^palette_size + 2, the second one is 2^palette_size + 2 and so on. The patterns for those entries are reconstructed from patterns for the previous codes. The lookup table grows with every new code.

Since LZW is a compression algorithm after all, it tries to pack codes to bits in the most efficient manner. Since there are at least
2^palette_size + 2 size codes, every code is encoded with palette_size + 1 bits, at first at least. Since the lookup table grows with every new pattern, at some point codes will stop to fit into this number of bits. When this happens , the algorithm increases the number of bits it packs each code into by one (but no more than 12).

This means that the code stream has variable bit length. For a 4-color image, codes start coming in 3 bits, then the bit length switches to 4, then 5 and so on, up to 12 bits per code if the image is large enough.

This is one of the challenges we have to deal with while implementing the decoder in SQL.

Another challenge is that the bit stream deals with bits, but the file storage format deals with bytes. Within each byte, the bits are stored LSB first. This means that three 3-bit codes would span two bytes. The first two codes would take up the least significant bits of the first byte, and the third code would take up one most significant bit from the first byte and two least significant ones from the second one.

This does not play well with the way PostgreSQL deals with the BYTEA to BIT conversion and BIT arithmetic in general. That's why we have to shuffle the bytes around a little bit and do so for every code we extract from the stream.

To extract a code from the stream, on each step we need to know the offset to the last code (in bits) and current bit length. We then take three next bytes from the stream and concatenate them in reverse. Once we have done that, we can convert them to a BIT(24) value and just use PostgreSQL's native SUBSTRING to extract as many bits as we need to get the next code.

Once we know the next code, we should look at its value and decide what to do with that. If it's a eof_code, we just end the processing. If it's a clear_code, we reset our lookup table and the current bit length to their initial values. Finally, if it's a valid code, we reconstruct the original pattern either from the lookup table (by looking up either current code or the one we retrieved from the previous iteration), and add the new pattern to the lookup table.

To do this we need a shortcut (an analog of a return statement in imperative languages). We would implement it as a series of a sourceless queries (those without a FROM clause), combined with a UNION ALL in an inline view, which is then selected from with LIMIT 1. The first query to return a value will shortcut and the other ones won't be executed.

Finally, we need to store the lookup table between the iterations. We will employ PostgreSQL's HSTORE type for that, as it's more efficient for key lookups than an array. It only can store text keys and text values, so we will have to convert INT keys (codes) and INT[] values (patterns) to text.

06.lzw-stream.sql

WITH    RECURSIVE
        bin AS
        (
        SELECT  DECODE(data, 'HEX') AS buffer
        FROM    input
        ),
        header AS
        (
        SELECT  h.*, io.*
        FROM    bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, 10)::BIT(8) AS flags
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CONVERT_FROM(SUBSTR(buffer, 1, 6), 'LATIN1') AS version,
                        (GET_BYTE(buffer, 7)::BIT(8) || GET_BYTE(buffer, 6)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, 9)::BIT(8) || GET_BYTE(buffer, 8)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = B'1' AS gct,
                        (flags << 1)::BIT(3)::INT + 1 AS depth,
                        (flags << 4) = B'1' AS color_sort,
                        2 << (flags << 5)::BIT(3)::INT AS gct_size,
                        GET_BYTE(buffer, 11)::INT AS bci,
                        GET_BYTE(buffer, 12)::INT AS aspect
                ) h
        CROSS JOIN LATERAL
                (
                SELECT  13 + 3 * gct_size AS blocks_offset
                ) io
        ),
        blocks AS
        (
        SELECT  blocks_offset AS current,
                GET_BYTE(buffer, blocks_offset) AS intro
        FROM    bin
        CROSS JOIN
                header
        UNION ALL
        SELECT  current,
                GET_BYTE(buffer, current) AS intro
        FROM    (
                SELECT  current AS previous,
                        intro AS previous_intro
                FROM    blocks
                ) b
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  previous + GET_BYTE(buffer, previous + 2) + 4 AS current
                ) q
        WHERE   previous_intro = x'21'::INT
        ),
        image_offset AS
        (
        SELECT  current AS image_offset
        FROM    blocks
        WHERE   intro = x'2C'::INT
        ),
        image_header AS
        (
        SELECT  q.*, l.*, l2.*
        FROM    image_offset
        CROSS JOIN
                bin
        CROSS JOIN
                header h
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_offset + 9)::BIT(8) AS flags
                ) f
        CROSS JOIN LATERAL
                (
                SELECT  (GET_BYTE(buffer, image_offset + 2)::BIT(8) || GET_BYTE(buffer, image_offset + 1)::BIT(8))::BIT(16)::INT AS left,
                        (GET_BYTE(buffer, image_offset + 4)::BIT(8) || GET_BYTE(buffer, image_offset + 3)::BIT(8))::BIT(16)::INT AS top,
                        (GET_BYTE(buffer, image_offset + 6)::BIT(8) || GET_BYTE(buffer, image_offset + 5)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, image_offset + 8)::BIT(8) || GET_BYTE(buffer, image_offset + 7)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = b'1' AS has_lct,
                        (flags << 1)::BIT(1) = b'1' AS interlace,
                        (flags << 2)::BIT(1) = b'1' AS sort
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CASE WHEN has_lct THEN 2 << (flags << 5)::BIT(3)::INT ELSE 0 END AS lct_size
                ) l
        CROSS JOIN LATERAL
                (
                SELECT  image_offset + 10 + lct_size AS image_data_offset
                ) l2
        ),
        image_data AS
        (
        SELECT  GET_BYTE(buffer, image_data_offset) AS code_size,
                image_data_offset + 1 AS image_first_block_offset
        FROM    image_header
        CROSS JOIN
                bin
        ),
        image_blocks AS
        (
        SELECT  image_first_block_offset AS block_offset,
                block_size,
                SUBSTR(buffer, image_first_block_offset + 2, block_size) AS block_data
        FROM    image_data
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_first_block_offset) AS block_size
                ) l
        UNION ALL
        SELECT  new_offset AS block_offset,
                new_block_size AS block_size,
                SUBSTR(buffer, new_offset + 2, new_block_size) AS block_data
        FROM    image_blocks
        CROSS JOIN LATERAL
                (
                SELECT  block_offset + block_size + 1 AS new_offset
                ) no
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, new_offset) AS new_block_size
                ) nbs
        WHERE   new_block_size > 0
        ),
        lzw_data AS
        (
        SELECT  code_size,
                (1 << code_size) AS clear_code,
                (1 << code_size) + 1 AS eof_code,
                compressed
        FROM    (
                SELECT  STRING_AGG(block_data, '' ORDER BY block_offset) compressed
                FROM    image_blocks
                ) i
        CROSS JOIN
                image_data
        ),
        lzw_bits AS
        (
        SELECT  current_code_size,
                clear_code AS code,
                ARRAY[]::INT[] AS output_chunk,
                0 AS next_bit_offset,
                NULL::HSTORE AS codes,
                0 AS next_table_key,
                0 AS next_index
        FROM    lzw_data
        CROSS JOIN LATERAL
                (
                SELECT  code_size + 1 AS current_code_size
                ) cc
        UNION ALL
        SELECT  next_code_size,
                code,
                output_chunk,
                bit_offset + current_code_size,
                new_codes AS codes,
                next_table_key,
                next_index
        FROM    (
                SELECT  code AS previous_code, current_code_size, next_bit_offset AS bit_offset, codes, next_table_key AS current_table_key,
                        next_index + COALESCE(ARRAY_UPPER(output_chunk, 1), 0) AS next_index,
                        ld.*
                FROM    lzw_bits
                CROSS JOIN
                        (
                        SELECT  code_size AS initial_code_size,
                                clear_code,
                                eof_code,
                                compressed
                        FROM    lzw_data
                        ) ld
                ) lb
        CROSS JOIN LATERAL
                (
                SELECT  bit_offset / 8 AS byte_offset
                ) bo
        CROSS JOIN LATERAL
                (
                SELECT  (
                        CASE WHEN byte_offset < LENGTH(compressed) - 2 THEN GET_BYTE(compressed, byte_offset + 2) ELSE 0 END::BIT(8) ||
                        CASE WHEN byte_offset < LENGTH(compressed) - 1 THEN GET_BYTE(compressed, byte_offset + 1) ELSE 0 END::BIT(8) ||
                        GET_BYTE(compressed, byte_offset)::BIT(8)
                        )::BIT(24) AS cut
                ) cc
        CROSS JOIN LATERAL
                (
                SELECT  SUBSTRING(cut, 25 - current_code_size - bit_offset % 8, current_code_size)::INT AS code
                ) l
        CROSS JOIN LATERAL
                (
                SELECT  *
                FROM    (
                        SELECT  ARRAY[]::INT[] AS output_chunk,
                                HSTORE(ARRAY[]::TEXT[][]) AS new_codes,
                                eof_code + 1 AS next_table_key,
                                initial_code_size + 1 AS next_code_size
                        WHERE   code = clear_code
                        UNION ALL
                        SELECT  CASE WHEN code < clear_code THEN ARRAY ELSE (codes->(code::TEXT))::INT[] END AS output_chunk,
                                codes AS new_codes,
                                current_table_key AS next_table_key,
                                current_code_size AS next_code_size
                        WHERE   previous_code = clear_code
                        UNION ALL
                        SELECT  output_chunk,
                                CASE current_table_key WHEN 4095 THEN codes ELSE codes || HSTORE(current_table_key::TEXT, next_table_chunk::TEXT) END AS new_codes,
                                next_table_key,
                                CASE next_table_key WHEN (1 << current_code_size) THEN current_code_size + 1 ELSE current_code_size END AS next_code_size
                        FROM    (
                                SELECT  CASE WHEN previous_code < clear_code THEN ARRAY[previous_code] ELSE (codes->(previous_code::TEXT))::INT[] END AS previous_chunk,
                                        LEAST(current_table_key + 1, 4095) AS next_table_key,
                                        code < clear_code OR codes ? (code::TEXT) AS code_in_table
                                ) pc
                        CROSS JOIN LATERAL
                                (
                                SELECT  output_chunk,
                                        previous_chunk || output_chunk[1] AS next_table_chunk
                                FROM    (
                                        SELECT  CASE WHEN code < clear_code THEN ARRAY ELSE (codes->(code::TEXT))::INT[] END AS output_chunk
                                        ) q
                                WHERE   code_in_table
                                UNION ALL
                                SELECT  output_chunk, output_chunk AS next_table_chunk
                                FROM    (
                                        SELECT  previous_chunk || previous_chunk[1] AS output_chunk
                                        ) q
                                WHERE   NOT code_in_table
                                ) q
                        WHERE   code <> eof_code
                        ) q
                LIMIT 1
                ) ns
        WHERE   bit_offset < LENGTH(compressed) * 8 
        )
SELECT  *
FROM    lzw_bits;

Here's what we have here:

• In lb and bo, we select the values from the previous recursion step and give them more meaningful names (so they become "previous" and "current" instead of "current" and "next" they used to be on the previous step). We are also doing some calculations we will need later and give their results aliases.
• In cc, we shuffle the bytes for better representation of BIT so that we could cut our codes from the bit stream more easily.
• In l, we cut the next code from the bit stream and give it a numeric value we will be working with.
• In ns, the actual decompression happens.
  • The first two queries in ns.q do the special processing for the CLEAR code and the one right after it (the first code after CLEAR does not update the lookup table, as it's not a pattern yet).
  • The third query does the table lookup and the table update, as described in the tutorial.
  • Note that there is another UNION ALL within the third query: the table lookup and update rules are different, depending on whether the code is in the lookup table already or not.
  • The third query also handles the EOF code (which just ends the recursion as no results would be returned from either query) and the possible 12-bit overflow.
• Note that we also explicitly select two additional fields in lzw_bits: next_table_key (the next code which we will be putting in the lookup table) and next_index (the running count of decoded color indexes so far). Those are not strictly necessary as they can be reconstructed, respectively, from the lookup table and previous query records (using window functions). However, they improve the query performance greatly, so it makes sense to drag them through the recursion, even though it makes it a little bit more complex.

Mapping color indexes to actual colors

Now that we have a recordset of patterns (arrays), we can make a recordset of individual color indexes by unnesting every array with ordinality and adding the ordinality to the next_index (which is, I will remind, the total length of patterns selected so far). This will give us a rowset of color indexes, each with its ordinal position in the original bitmap:

07.indexes.sql

WITH    RECURSIVE
        bin AS
        (
        SELECT  DECODE(data, 'HEX') AS buffer
        FROM    input
        ),
        header AS
        (
        SELECT  h.*, io.*
        FROM    bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, 10)::BIT(8) AS flags
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CONVERT_FROM(SUBSTR(buffer, 1, 6), 'LATIN1') AS version,
                        (GET_BYTE(buffer, 7)::BIT(8) || GET_BYTE(buffer, 6)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, 9)::BIT(8) || GET_BYTE(buffer, 8)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = B'1' AS gct,
                        (flags << 1)::BIT(3)::INT + 1 AS depth,
                        (flags << 4) = B'1' AS color_sort,
                        2 << (flags << 5)::BIT(3)::INT AS gct_size,
                        GET_BYTE(buffer, 11)::INT AS bci,
                        GET_BYTE(buffer, 12)::INT AS aspect
                ) h
        CROSS JOIN LATERAL
                (
                SELECT  13 + 3 * gct_size AS blocks_offset
                ) io
        ),
        blocks AS
        (
        SELECT  blocks_offset AS current,
                GET_BYTE(buffer, blocks_offset) AS intro
        FROM    bin
        CROSS JOIN
                header
        UNION ALL
        SELECT  current,
                GET_BYTE(buffer, current) AS intro
        FROM    (
                SELECT  current AS previous,
                        intro AS previous_intro
                FROM    blocks
                ) b
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  previous + GET_BYTE(buffer, previous + 2) + 4 AS current
                ) q
        WHERE   previous_intro = x'21'::INT
        ),
        image_offset AS
        (
        SELECT  current AS image_offset
        FROM    blocks
        WHERE   intro = x'2C'::INT
        ),
        image_header AS
        (
        SELECT  q.*, l.*, l2.*
        FROM    image_offset
        CROSS JOIN
                bin
        CROSS JOIN
                header h
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_offset + 9)::BIT(8) AS flags
                ) f
        CROSS JOIN LATERAL
                (
                SELECT  (GET_BYTE(buffer, image_offset + 2)::BIT(8) || GET_BYTE(buffer, image_offset + 1)::BIT(8))::BIT(16)::INT AS left,
                        (GET_BYTE(buffer, image_offset + 4)::BIT(8) || GET_BYTE(buffer, image_offset + 3)::BIT(8))::BIT(16)::INT AS top,
                        (GET_BYTE(buffer, image_offset + 6)::BIT(8) || GET_BYTE(buffer, image_offset + 5)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, image_offset + 8)::BIT(8) || GET_BYTE(buffer, image_offset + 7)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = b'1' AS has_lct,
                        (flags << 1)::BIT(1) = b'1' AS interlace,
                        (flags << 2)::BIT(1) = b'1' AS sort
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CASE WHEN has_lct THEN 2 << (flags << 5)::BIT(3)::INT ELSE 0 END AS lct_size
                ) l
        CROSS JOIN LATERAL
                (
                SELECT  image_offset + 10 + lct_size AS image_data_offset
                ) l2
        ),
        image_data AS
        (
        SELECT  GET_BYTE(buffer, image_data_offset) AS code_size,
                image_data_offset + 1 AS image_first_block_offset
        FROM    image_header
        CROSS JOIN
                bin
        ),
        image_blocks AS
        (
        SELECT  image_first_block_offset AS block_offset,
                block_size,
                SUBSTR(buffer, image_first_block_offset + 2, block_size) AS block_data
        FROM    image_data
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_first_block_offset) AS block_size
                ) l
        UNION ALL
        SELECT  new_offset AS block_offset,
                new_block_size AS block_size,
                SUBSTR(buffer, new_offset + 2, new_block_size) AS block_data
        FROM    image_blocks
        CROSS JOIN LATERAL
                (
                SELECT  block_offset + block_size + 1 AS new_offset
                ) no
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, new_offset) AS new_block_size
                ) nbs
        WHERE   new_block_size > 0
        ),
        lzw_data AS
        (
        SELECT  code_size,
                (1 << code_size) AS clear_code,
                (1 << code_size) + 1 AS eof_code,
                compressed
        FROM    (
                SELECT  STRING_AGG(block_data, '' ORDER BY block_offset) compressed
                FROM    image_blocks
                ) i
        CROSS JOIN
                image_data
        ),
        lzw_bits AS
        (
        SELECT  current_code_size,
                clear_code AS code,
                ARRAY[]::INT[] AS output_chunk,
                0 AS next_bit_offset,
                NULL::HSTORE AS codes,
                0 AS next_table_key,
                0 AS next_index
        FROM    lzw_data
        CROSS JOIN LATERAL
                (
                SELECT  code_size + 1 AS current_code_size
                ) cc
        UNION ALL
        SELECT  next_code_size,
                code,
                output_chunk,
                bit_offset + current_code_size,
                new_codes AS codes,
                next_table_key,
                next_index
        FROM    (
                SELECT  code AS previous_code, current_code_size, next_bit_offset AS bit_offset, codes, next_table_key AS current_table_key,
                        next_index + COALESCE(ARRAY_UPPER(output_chunk, 1), 0) AS next_index,
                        ld.*
                FROM    lzw_bits
                CROSS JOIN
                        (
                        SELECT  code_size AS initial_code_size,
                                clear_code,
                                eof_code,
                                compressed
                        FROM    lzw_data
                        ) ld
                ) lb
        CROSS JOIN LATERAL
                (
                SELECT  bit_offset / 8 AS byte_offset
                ) bo
        CROSS JOIN LATERAL
                (
                SELECT  (
                        CASE WHEN byte_offset < LENGTH(compressed) - 2 THEN GET_BYTE(compressed, byte_offset + 2) ELSE 0 END::BIT(8) ||
                        CASE WHEN byte_offset < LENGTH(compressed) - 1 THEN GET_BYTE(compressed, byte_offset + 1) ELSE 0 END::BIT(8) ||
                        GET_BYTE(compressed, byte_offset)::BIT(8)
                        )::BIT(24) AS cut
                ) cc
        CROSS JOIN LATERAL
                (
                SELECT  SUBSTRING(cut, 25 - current_code_size - bit_offset % 8, current_code_size)::INT AS code
                ) l
        CROSS JOIN LATERAL
                (
                SELECT  *
                FROM    (
                        SELECT  ARRAY[]::INT[] AS output_chunk,
                                HSTORE(ARRAY[]::TEXT[][]) AS new_codes,
                                eof_code + 1 AS next_table_key,
                                initial_code_size + 1 AS next_code_size
                        WHERE   code = clear_code
                        UNION ALL
                        SELECT  CASE WHEN code < clear_code THEN ARRAY ELSE (codes->(code::TEXT))::INT[] END AS output_chunk,
                                codes AS new_codes,
                                current_table_key AS next_table_key,
                                current_code_size AS next_code_size
                        WHERE   previous_code = clear_code
                        UNION ALL
                        SELECT  output_chunk,
                                CASE current_table_key WHEN 4095 THEN codes ELSE codes || HSTORE(current_table_key::TEXT, next_table_chunk::TEXT) END AS new_codes,
                                next_table_key,
                                CASE next_table_key WHEN (1 << current_code_size) THEN current_code_size + 1 ELSE current_code_size END AS next_code_size
                        FROM    (
                                SELECT  CASE WHEN previous_code < clear_code THEN ARRAY[previous_code] ELSE (codes->(previous_code::TEXT))::INT[] END AS previous_chunk,
                                        LEAST(current_table_key + 1, 4095) AS next_table_key,
                                        code < clear_code OR codes ? (code::TEXT) AS code_in_table
                                ) pc
                        CROSS JOIN LATERAL
                                (
                                SELECT  output_chunk,
                                        previous_chunk || output_chunk[1] AS next_table_chunk
                                FROM    (
                                        SELECT  CASE WHEN code < clear_code THEN ARRAY ELSE (codes->(code::TEXT))::INT[] END AS output_chunk
                                        ) q
                                WHERE   code_in_table
                                UNION ALL
                                SELECT  output_chunk, output_chunk AS next_table_chunk
                                FROM    (
                                        SELECT  previous_chunk || previous_chunk[1] AS output_chunk
                                        ) q
                                WHERE   NOT code_in_table
                                        AND code <> eof_code
                                ) q
                        ) q
                LIMIT 1
                ) ns
        WHERE   previous_code IS DISTINCT FROM eof_code
                AND bit_offset < LENGTH(compressed) * 8 
        ),
        indices AS
        (
        SELECT  idx, next_index + position - 1 AS rn
        FROM    lzw_bits
        CROSS JOIN LATERAL
                UNNEST(output_chunk) WITH ORDINALITY q (idx, position)
        )
SELECT  *
FROM    indices;

(results cut for readability).

We have exactly 100 records here, which corresponds to our 10×10 image. Those are not actual colors though yet, just the indexes.

To retrieve the actual colors we have to look them up by index in the Global Color Table (remember that thing)? This is just a region in our original binary stream, and the color indexes are nothing but offsets from the beginning of this region. And while we're at that, we could just as well convert our image to grayscale: there is a formula for that.

08.pixels.sql

WITH    RECURSIVE
        bin AS
        (
        SELECT  DECODE(data, 'HEX') AS buffer
        FROM    input
        ),
        header AS
        (
        SELECT  h.*, io.*
        FROM    bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, 10)::BIT(8) AS flags
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CONVERT_FROM(SUBSTR(buffer, 1, 6), 'LATIN1') AS version,
                        (GET_BYTE(buffer, 7)::BIT(8) || GET_BYTE(buffer, 6)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, 9)::BIT(8) || GET_BYTE(buffer, 8)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = B'1' AS gct,
                        (flags << 1)::BIT(3)::INT + 1 AS depth,
                        (flags << 4) = B'1' AS color_sort,
                        2 << (flags << 5)::BIT(3)::INT AS gct_size,
                        GET_BYTE(buffer, 11)::INT AS bci,
                        GET_BYTE(buffer, 12)::INT AS aspect
                ) h
        CROSS JOIN LATERAL
                (
                SELECT  13 + 3 * gct_size AS blocks_offset
                ) io
        ),
        blocks AS
        (
        SELECT  blocks_offset AS current,
                GET_BYTE(buffer, blocks_offset) AS intro
        FROM    bin
        CROSS JOIN
                header
        UNION ALL
        SELECT  current,
                GET_BYTE(buffer, current) AS intro
        FROM    (
                SELECT  current AS previous,
                        intro AS previous_intro
                FROM    blocks
                ) b
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  previous + GET_BYTE(buffer, previous + 2) + 4 AS current
                ) q
        WHERE   previous_intro = x'21'::INT
        ),
        image_offset AS
        (
        SELECT  current AS image_offset
        FROM    blocks
        WHERE   intro = x'2C'::INT
        ),
        image_header AS
        (
        SELECT  q.*, l.*, l2.*
        FROM    image_offset
        CROSS JOIN
                bin
        CROSS JOIN
                header h
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_offset + 9)::BIT(8) AS flags
                ) f
        CROSS JOIN LATERAL
                (
                SELECT  (GET_BYTE(buffer, image_offset + 2)::BIT(8) || GET_BYTE(buffer, image_offset + 1)::BIT(8))::BIT(16)::INT AS left,
                        (GET_BYTE(buffer, image_offset + 4)::BIT(8) || GET_BYTE(buffer, image_offset + 3)::BIT(8))::BIT(16)::INT AS top,
                        (GET_BYTE(buffer, image_offset + 6)::BIT(8) || GET_BYTE(buffer, image_offset + 5)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, image_offset + 8)::BIT(8) || GET_BYTE(buffer, image_offset + 7)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = b'1' AS has_lct,
                        (flags << 1)::BIT(1) = b'1' AS interlace,
                        (flags << 2)::BIT(1) = b'1' AS sort
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CASE WHEN has_lct THEN 2 << (flags << 5)::BIT(3)::INT ELSE 0 END AS lct_size
                ) l
        CROSS JOIN LATERAL
                (
                SELECT  image_offset + 10 + lct_size AS image_data_offset
                ) l2
        ),
        image_data AS
        (
        SELECT  GET_BYTE(buffer, image_data_offset) AS code_size,
                image_data_offset + 1 AS image_first_block_offset
        FROM    image_header
        CROSS JOIN
                bin
        ),
        image_blocks AS
        (
        SELECT  image_first_block_offset AS block_offset,
                block_size,
                SUBSTR(buffer, image_first_block_offset + 2, block_size) AS block_data
        FROM    image_data
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_first_block_offset) AS block_size
                ) l
        UNION ALL
        SELECT  new_offset AS block_offset,
                new_block_size AS block_size,
                SUBSTR(buffer, new_offset + 2, new_block_size) AS block_data
        FROM    image_blocks
        CROSS JOIN LATERAL
                (
                SELECT  block_offset + block_size + 1 AS new_offset
                ) no
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, new_offset) AS new_block_size
                ) nbs
        WHERE   new_block_size > 0
        ),
        lzw_data AS
        (
        SELECT  code_size,
                (1 << code_size) AS clear_code,
                (1 << code_size) + 1 AS eof_code,
                compressed
        FROM    (
                SELECT  STRING_AGG(block_data, '' ORDER BY block_offset) compressed
                FROM    image_blocks
                ) i
        CROSS JOIN
                image_data
        ),
        lzw_bits AS
        (
        SELECT  current_code_size,
                clear_code AS code,
                ARRAY[]::INT[] AS output_chunk,
                0 AS next_bit_offset,
                NULL::HSTORE AS codes,
                0 AS next_table_key,
                0 AS next_index
        FROM    lzw_data
        CROSS JOIN LATERAL
                (
                SELECT  code_size + 1 AS current_code_size
                ) cc
        UNION ALL
        SELECT  next_code_size,
                code,
                output_chunk,
                bit_offset + current_code_size,
                new_codes AS codes,
                next_table_key,
                next_index
        FROM    (
                SELECT  code AS previous_code, current_code_size, next_bit_offset AS bit_offset, codes, next_table_key AS current_table_key,
                        next_index + COALESCE(ARRAY_UPPER(output_chunk, 1), 0) AS next_index,
                        ld.*
                FROM    lzw_bits
                CROSS JOIN
                        (
                        SELECT  code_size AS initial_code_size,
                                clear_code,
                                eof_code,
                                compressed
                        FROM    lzw_data
                        ) ld
                ) lb
        CROSS JOIN LATERAL
                (
                SELECT  bit_offset / 8 AS byte_offset
                ) bo
        CROSS JOIN LATERAL
                (
                SELECT  (
                        CASE WHEN byte_offset < LENGTH(compressed) - 2 THEN GET_BYTE(compressed, byte_offset + 2) ELSE 0 END::BIT(8) ||
                        CASE WHEN byte_offset < LENGTH(compressed) - 1 THEN GET_BYTE(compressed, byte_offset + 1) ELSE 0 END::BIT(8) ||
                        GET_BYTE(compressed, byte_offset)::BIT(8)
                        )::BIT(24) AS cut
                ) cc
        CROSS JOIN LATERAL
                (
                SELECT  SUBSTRING(cut, 25 - current_code_size - bit_offset % 8, current_code_size)::INT AS code
                ) l
        CROSS JOIN LATERAL
                (
                SELECT  *
                FROM    (
                        SELECT  ARRAY[]::INT[] AS output_chunk,
                                HSTORE(ARRAY[]::TEXT[][]) AS new_codes,
                                eof_code + 1 AS next_table_key,
                                initial_code_size + 1 AS next_code_size
                        WHERE   code = clear_code
                        UNION ALL
                        SELECT  CASE WHEN code < clear_code THEN ARRAY ELSE (codes->(code::TEXT))::INT[] END AS output_chunk,
                                codes AS new_codes,
                                current_table_key AS next_table_key,
                                current_code_size AS next_code_size
                        WHERE   previous_code = clear_code
                        UNION ALL
                        SELECT  output_chunk,
                                CASE current_table_key WHEN 4095 THEN codes ELSE codes || HSTORE(current_table_key::TEXT, next_table_chunk::TEXT) END AS new_codes,
                                next_table_key,
                                CASE next_table_key WHEN (1 << current_code_size) THEN current_code_size + 1 ELSE current_code_size END AS next_code_size
                        FROM    (
                                SELECT  CASE WHEN previous_code < clear_code THEN ARRAY[previous_code] ELSE (codes->(previous_code::TEXT))::INT[] END AS previous_chunk,
                                        LEAST(current_table_key + 1, 4095) AS next_table_key,
                                        code < clear_code OR codes ? (code::TEXT) AS code_in_table
                                ) pc
                        CROSS JOIN LATERAL
                                (
                                SELECT  output_chunk,
                                        previous_chunk || output_chunk[1] AS next_table_chunk
                                FROM    (
                                        SELECT  CASE WHEN code < clear_code THEN ARRAY ELSE (codes->(code::TEXT))::INT[] END AS output_chunk
                                        ) q
                                WHERE   code_in_table
                                UNION ALL
                                SELECT  output_chunk, output_chunk AS next_table_chunk
                                FROM    (
                                        SELECT  previous_chunk || previous_chunk[1] AS output_chunk
                                        ) q
                                WHERE   NOT code_in_table
                                        AND code <> eof_code
                                ) q
                        ) q
                LIMIT 1
                ) ns
        WHERE   previous_code IS DISTINCT FROM eof_code
                AND bit_offset < LENGTH(compressed) * 8 
        ),
        indices AS
        (
        SELECT  idx, next_index + position - 1 AS rn
        FROM    lzw_bits
        CROSS JOIN LATERAL
                UNNEST(output_chunk) WITH ORDINALITY q (idx, position)
        ),
        pixels AS
        (
        SELECT  idx,
                rn % width AS x,
                rn / width AS y,
                luma
        FROM    indices
        CROSS JOIN
                bin
        CROSS JOIN
                header
        CROSS JOIN LATERAL
                (
                SELECT  255 max_value,
                        2.2 gamma,
                        GET_BYTE(buffer, 13 + idx * 3 + 0) r,
                        GET_BYTE(buffer, 13 + idx * 3 + 1) g,
                        GET_BYTE(buffer, 13 + idx * 3 + 2) b,
                        .2126 rw,
                        .7152 gw,
                        .0722 bw
                ) c
        CROSS JOIN LATERAL
                (
                SELECT  ((r::FLOAT / max_value) ^ gamma) * rw +
                        ((g::FLOAT / max_value) ^ gamma) * gw +
                        ((b::FLOAT / max_value) ^ gamma) * bw AS luma
                ) l
        )
SELECT  *
FROM    pixels;

Rendering the result

Finally we need to render our picture on the screen. We do that as usual:

• Cross join two GENERATE_SERIES for width and height
• Left join it with the rowset of pixels on (x, y)
• Look up the ASCII art character depending on the pixel luminance
• Group by y and STRING_AGG the ASCII art characters without the separator, ordering by x

gif.sql

WITH    RECURSIVE
        bin AS
        (
        SELECT  DECODE(data, 'HEX') AS buffer
        FROM    input
        ),
        header AS
        (
        SELECT  h.*, io.*
        FROM    bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, 10)::BIT(8) AS flags
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CONVERT_FROM(SUBSTR(buffer, 1, 6), 'LATIN1') AS version,
                        (GET_BYTE(buffer, 7)::BIT(8) || GET_BYTE(buffer, 6)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, 9)::BIT(8) || GET_BYTE(buffer, 8)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = B'1' AS gct,
                        (flags << 1)::BIT(3)::INT + 1 AS depth,
                        (flags << 4) = B'1' AS color_sort,
                        2 << (flags << 5)::BIT(3)::INT AS gct_size,
                        GET_BYTE(buffer, 11)::INT AS bci,
                        GET_BYTE(buffer, 12)::INT AS aspect
                ) h
        CROSS JOIN LATERAL
                (
                SELECT  13 + 3 * gct_size AS blocks_offset
                ) io
        ),
        blocks AS
        (
        SELECT  blocks_offset AS current,
                GET_BYTE(buffer, blocks_offset) AS intro
        FROM    bin
        CROSS JOIN
                header
        UNION ALL
        SELECT  current,
                GET_BYTE(buffer, current) AS intro
        FROM    (
                SELECT  current AS previous,
                        intro AS previous_intro
                FROM    blocks
                ) b
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  previous + GET_BYTE(buffer, previous + 2) + 4 AS current
                ) q
        WHERE   previous_intro = x'21'::INT
        ),
        image_offset AS
        (
        SELECT  current AS image_offset
        FROM    blocks
        WHERE   intro = x'2C'::INT
        ),
        image_header AS
        (
        SELECT  q.*, l.*, l2.*
        FROM    image_offset
        CROSS JOIN
                bin
        CROSS JOIN
                header h
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_offset + 9)::BIT(8) AS flags
                ) f
        CROSS JOIN LATERAL
                (
                SELECT  (GET_BYTE(buffer, image_offset + 2)::BIT(8) || GET_BYTE(buffer, image_offset + 1)::BIT(8))::BIT(16)::INT AS left,
                        (GET_BYTE(buffer, image_offset + 4)::BIT(8) || GET_BYTE(buffer, image_offset + 3)::BIT(8))::BIT(16)::INT AS top,
                        (GET_BYTE(buffer, image_offset + 6)::BIT(8) || GET_BYTE(buffer, image_offset + 5)::BIT(8))::BIT(16)::INT AS width,
                        (GET_BYTE(buffer, image_offset + 8)::BIT(8) || GET_BYTE(buffer, image_offset + 7)::BIT(8))::BIT(16)::INT AS height,
                        flags::BIT(1) = b'1' AS has_lct,
                        (flags << 1)::BIT(1) = b'1' AS interlace,
                        (flags << 2)::BIT(1) = b'1' AS sort
                ) q
        CROSS JOIN LATERAL
                (
                SELECT  CASE WHEN has_lct THEN 2 << (flags << 5)::BIT(3)::INT ELSE 0 END AS lct_size
                ) l
        CROSS JOIN LATERAL
                (
                SELECT  image_offset + 10 + lct_size AS image_data_offset
                ) l2
        ),
        image_data AS
        (
        SELECT  GET_BYTE(buffer, image_data_offset) AS code_size,
                image_data_offset + 1 AS image_first_block_offset
        FROM    image_header
        CROSS JOIN
                bin
        ),
        image_blocks AS
        (
        SELECT  image_first_block_offset AS block_offset,
                block_size,
                SUBSTR(buffer, image_first_block_offset + 2, block_size) AS block_data
        FROM    image_data
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, image_first_block_offset) AS block_size
                ) l
        UNION ALL
        SELECT  new_offset AS block_offset,
                new_block_size AS block_size,
                SUBSTR(buffer, new_offset + 2, new_block_size) AS block_data
        FROM    image_blocks
        CROSS JOIN LATERAL
                (
                SELECT  block_offset + block_size + 1 AS new_offset
                ) no
        CROSS JOIN
                bin
        CROSS JOIN LATERAL
                (
                SELECT  GET_BYTE(buffer, new_offset) AS new_block_size
                ) nbs
        WHERE   new_block_size > 0
        ),
        lzw_data AS
        (
        SELECT  code_size,
                (1 << code_size) AS clear_code,
                (1 << code_size) + 1 AS eof_code,
                compressed
        FROM    (
                SELECT  STRING_AGG(block_data, '' ORDER BY block_offset) compressed
                FROM    image_blocks
                ) i
        CROSS JOIN
                image_data
        ),
        lzw_bits AS
        (
        SELECT  current_code_size,
                clear_code AS code,
                ARRAY[]::INT[] AS output_chunk,
                0 AS next_bit_offset,
                NULL::HSTORE AS codes,
                0 AS next_table_key,
                0 AS next_index
        FROM    lzw_data
        CROSS JOIN LATERAL
                (
                SELECT  code_size + 1 AS current_code_size
                ) cc
        UNION ALL
        SELECT  next_code_size,
                code,
                output_chunk,
                bit_offset + current_code_size,
                new_codes AS codes,
                next_table_key,
                next_index
        FROM    (
                SELECT  code AS previous_code, current_code_size, next_bit_offset AS bit_offset, codes, next_table_key AS current_table_key,
                        next_index + COALESCE(ARRAY_UPPER(output_chunk, 1), 0) AS next_index,
                        ld.*
                FROM    lzw_bits
                CROSS JOIN
                        (
                        SELECT  code_size AS initial_code_size,
                                clear_code,
                                eof_code,
                                compressed
                        FROM    lzw_data
                        ) ld
                ) lb
        CROSS JOIN LATERAL
                (
                SELECT  bit_offset / 8 AS byte_offset
                ) bo
        CROSS JOIN LATERAL
                (
                SELECT  (
                        CASE WHEN byte_offset < LENGTH(compressed) - 2 THEN GET_BYTE(compressed, byte_offset + 2) ELSE 0 END::BIT(8) ||
                        CASE WHEN byte_offset < LENGTH(compressed) - 1 THEN GET_BYTE(compressed, byte_offset + 1) ELSE 0 END::BIT(8) ||
                        GET_BYTE(compressed, byte_offset)::BIT(8)
                        )::BIT(24) AS cut
                ) cc
        CROSS JOIN LATERAL
                (
                SELECT  SUBSTRING(cut, 25 - current_code_size - bit_offset % 8, current_code_size)::INT AS code
                ) l
        CROSS JOIN LATERAL
                (
                SELECT  *
                FROM    (
                        SELECT  ARRAY[]::INT[] AS output_chunk,
                                HSTORE(ARRAY[]::TEXT[][]) AS new_codes,
                                eof_code + 1 AS next_table_key,
                                initial_code_size + 1 AS next_code_size
                        WHERE   code = clear_code
                        UNION ALL
                        SELECT  CASE WHEN code < clear_code THEN ARRAY ELSE (codes->(code::TEXT))::INT[] END AS output_chunk,
                                codes AS new_codes,
                                current_table_key AS next_table_key,
                                current_code_size AS next_code_size
                        WHERE   previous_code = clear_code
                        UNION ALL
                        SELECT  output_chunk,
                                CASE current_table_key WHEN 4095 THEN codes ELSE codes || HSTORE(current_table_key::TEXT, next_table_chunk::TEXT) END AS new_codes,
                                next_table_key,
                                CASE next_table_key WHEN (1 << current_code_size) THEN current_code_size + 1 ELSE current_code_size END AS next_code_size
                        FROM    (
                                SELECT  CASE WHEN previous_code < clear_code THEN ARRAY[previous_code] ELSE (codes->(previous_code::TEXT))::INT[] END AS previous_chunk,
                                        LEAST(current_table_key + 1, 4095) AS next_table_key,
                                        code < clear_code OR codes ? (code::TEXT) AS code_in_table
                                ) pc
                        CROSS JOIN LATERAL
                                (
                                SELECT  output_chunk,
                                        previous_chunk || output_chunk[1] AS next_table_chunk
                                FROM    (
                                        SELECT  CASE WHEN code < clear_code THEN ARRAY ELSE (codes->(code::TEXT))::INT[] END AS output_chunk
                                        ) q
                                WHERE   code_in_table
                                UNION ALL
                                SELECT  output_chunk, output_chunk AS next_table_chunk
                                FROM    (
                                        SELECT  previous_chunk || previous_chunk[1] AS output_chunk
                                        ) q
                                WHERE   NOT code_in_table
                                        AND code <> eof_code
                                ) q
                        ) q
                LIMIT 1
                ) ns
        WHERE   previous_code IS DISTINCT FROM eof_code
                AND bit_offset < LENGTH(compressed) * 8 
        ),
        indices AS
        (
        SELECT  idx, next_index + position - 1 AS rn
        FROM    lzw_bits
        CROSS JOIN LATERAL
                UNNEST(output_chunk) WITH ORDINALITY q (idx, position)
        ),
        pixels AS
        (
        SELECT  idx,
                rn % width AS x,
                rn / width AS y,
                luma
        FROM    indices
        CROSS JOIN
                bin
        CROSS JOIN
                header
        CROSS JOIN LATERAL
                (
                SELECT  255 max_value,
                        2.2 gamma,
                        GET_BYTE(buffer, 13 + idx * 3 + 0) r,
                        GET_BYTE(buffer, 13 + idx * 3 + 1) g,
                        GET_BYTE(buffer, 13 + idx * 3 + 2) b,
                        .2126 rw,
                        .7152 gw,
                        .0722 bw
                ) c
        CROSS JOIN LATERAL
                (
                SELECT  ((r::FLOAT / max_value) ^ gamma) * rw +
                        ((g::FLOAT / max_value) ^ gamma) * gw +
                        ((b::FLOAT / max_value) ^ gamma) * bw AS luma
                ) l
        ),
        picture AS
        (
        SELECT  STRING_AGG(SUBSTRING(palette, LEAST(FLOOR(COALESCE(luma, 0) * LENGTH(palette))::INT + 1, LENGTH(palette)), 1), '' ORDER BY x)
        FROM    (
                SELECT  *,
                        ' .*:o&8#' AS palette
                FROM    header
                ) h
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, width - 1) x
        CROSS JOIN LATERAL
                GENERATE_SERIES(0, height - 1) y
        LEFT JOIN
                pixels
        USING   (x, y)
        GROUP BY
                y
        ORDER BY
                y
        )
SELECT  *
FROM    picture;

Here we are!

More pictures

So let's do a cat:

And a pig!