ChipLog — Christian Hammond Development, data, design, debugging 2026-05-14T00:33:18Z https://chipx86.blog/feed/atom/ WordPress chipx86 http://chipx86.blog <![CDATA[Inside Faxanadu, Part 2: Secret World Warp]]> https://chipx86.blog/?p=1638 2026-05-14T00:33:18Z 2026-05-14T00:04:01Z There’s a lot about Faxanadu that’s been kept secret since the game was originally developed in the 80s. Secrets I’ve been discovering in my Faxanadu disassembly work.

One of the most surprising to me was hidden code for a World Warp, letting you jump to any of the 6 regions of the game: Eolis, Trunk, Mist, Branch, Dartmoor, or the Evil Fortress.

All with a single button press. (And a couple of Game Genie codes.)

If you’re new to this series, see:

The World Warp

During development of Faxanadu, code was put in the game to make it easy to jump to other areas within the game for testing. This isn’t an uncommon thing for game developers to do. You don’t want to have to play through half the game every time you want to re-test the fog animation in the world of Mist.

This code was wired off from the rest of the game, existing only as a sequence of 44 bytes.

This small sequence of code looks at the controller 2 button mask (the only place in the game that cares about this controller) and checks if Up or Down is pressed.

If Up is pressed: Go forward a world.

If Down is pressed: Go back a world.

That wraps around, so if on Eolis, Down takes you to the Evil Fortress. Up takes you back to Eolis.

This is, by the way, a quick way of getting to the final boss of the game (spoilers!):

Not that it’ll do you much good.

Activating the World Warp

Since this 44 byte packet of debug code is wired off from the game, we have to patch it back in.

We can do this with two Game Genie codes:

  • OPYISU
  • NIYIVU

This will turn off your ability to Pause, but turn on the World Warp.

Game Genie codes are just a magic way of saying “At this address in memory, replace the value with a new value.” So let’s start by seeing what those codes translate into:

  • DB75:99 — Set the value 99 at address 0xDB75 in memory
  • DB76:DF — Set the value DF at address 0xDB76 in memory

Why those values? Those addresses?

Well, 0xDB75 and 0xDB76 are operands to a JSR instruction (Jump to Subroutine) that specifies an address to call every tick of the game loop. This function is GameLoop_CheckPauseGame at 0xE02B. This is the least-invasive function to replace. We can live without pausing.

We’re swapping that out with our new address: 0xDF99. This is the address where the world warp code lives.

Go ahead, put those Game Genie codes in your NES Game Genie (or your emulator), and press Up or Down on Controller 2. Play around with jumping from different screens, and from inside buildings! You may just glitch into a garbage world for a bit where the screens no longer match up. It’s fun!

The Code behind the World Warp

This code lives in Bank 15 at 0xDF99 in memory. Bank 15 is always in memory, by the way, and contains most of the main game loop, PPU, screen transition, and player state code, amongst others.

Here’s what the code looks like:

A screenshot of the disassembled source code. See below for a link to the actual code in text form.

You can view this on the disassembly.

Now you may not be intimately familiar with 6502 assembly, which is kind of weird but I’ll allow it. So let’s look at this in C form (a language everyone feels comfortable with!):

void Debug_ChooseArea(void)
{
  if ((Joy2_ButtonMask & BUTTON_BIT_UP) != BUTTON_BIT_NONE) {
    // Going up.
    Area_Region++;

    if (Area_Region > REGION_EVIL_FORTRESS) {
      Area_Region = REGION_EOLIS;
    }
  }
  else if ((Joy2_ButtonMask & BUTTON_BIT_DOWN) != BUTTON_BIT_NONE) {
    // Going down.
    Area_Region--;

    if (Area_Region < 0) {
      Area_Region = REGION_EVIL_FORTRESS;
    }
  }

  Area_CurrentScreen = 0;
  Game_SetupAndLoadArea();
}

I’m using the names I have in my disassembly. These may change over time. The “Region” is the “World” we’re switching between. (They’re not really independent worlds in the game, just regions of the World Tree.)

Pretty simple. Press Up, increase the region/world, and handle wrapping. Do the opposite for Down. Then, set the screen we’re going into and load the area.

Now, although it’s setting the screen number, there’s a lot more to setting the target screen than that, particularly when you’re inside a building. So depending on where you are when you warp, you’ll end up either in some predictable useful locations, or in Garbage Land. Both are fun to play with.

Wait, wasn’t Part 2 about Behavior Scripts?

In my first post in the series, I said I’d be going into Behavior Scripts next. That was a big fat lie. I will be, but today I thought it’d be fun to show off this piece of code that’s been hidden for 20 30 (oh my god) 40 years.

We’ll get to Behavior Scripts soon.

If you’d like to follow my Faxanadu work, you can check out the following:

And you can follow my efforts on:

]]> 0 chipx86 http://chipx86.blog <![CDATA[Inside Faxanadu, Part 1: Interaction Scripts and Shops]]> https://chipx86.blog/?p=1600 2026-05-05T06:38:33Z 2026-05-03T08:00:49Z Faxanadu for NES has always been one of my absolute favorite games of the NES library. This adventure starts off in the town of Eolis at the base of the giant World Tree, home to the Elves and Dwarfs. A world corrupted by a poison, transforming the Dwarfs into horrific monsters.

I couldn’t describe it any better than my good pal Quwebs (Twitch, NES Abridged on YouTube):

This game’s visuals and musical score always made it a stand-out game for me, and though it didn’t receive the notoriety of Castlevania, Zelda, Kid Kool, or other top-tier action adventure games at the time, it’s increasingly become a favorite to those discovering or rediscovering retro games. Better late than never.

It’s also technically very interesting.

In 2025, I began performing a disassembly of the Faxanadu ROM, using Ghidra and my own crafted Retro-Tinkertoys. The goal was to learn how this game worked, inside and out, and to document it with the hope that it would lead to mods for new games set in this world. And it did.

At this point, all the important parts of the game are either documented or now well-understood. New discoveries, such as hidden code for level switch, remnants of older game designs, unused code, and some interesting bugs have been discovered.

Plus, three built-in binary scripting languages. One for the advanced music engine (MScripts), one for enemy or NPC behavior (BScripts), and one for NPC and item interaction (IScripts).

I’m going to start our deep dive into Faxanadu today by talking about the Interaction Scripts, which I’ll be referring to as IScripts going forward.

What is an IScript?

IScripts define what happens when the player interacts with an entity. That may be an item or an NPC, such as a shopkeeper or villager.

They look like this:

  80     # King interaction

  0D     # If Player Has Gold:
  5A A3  #     Then jump to _HAS_GOLD ($A35A).
  01     # Show Unskippable Message:
  35     #     Message 53 (Exposition).
  09     # Add Gold to Player:
  DC 05  #     1500G.
  00     # End Script.

_HAS_GOLD:
  03     # Show Message:
  36     #     Message 54 ("Nothing more I can do to help").
  00     # End Script.

This particular script is the King, one of the first NPCs you meet in the game. He sends the hero on his quest with enough money to buy a sandwich, bag of chips, and a butter knife. He’s also big into exposition, putting in just enough effort to stand up for a minute to send you off to solve his problems. Royalty.

Any time you talk to the King, that script runs. And knowing how that script runs, we can see that if you spend all your money, he’ll just give you more. Not a new discovery, but this clearly shows how it works. If the player has no gold, give the player gold. The end.

The script is small and simple, but they can do a lot. This is an NES game where bytes and speed matter, so rather than a complex text-based scripting language like you’d find today, this is a compact binary-based scripting language. Every byte is either an instruction or an argument to an instruction. In these scripts, I’ll show the bytes and add comments describing what they do, so you can follow along. Values in the comments are in decimal, and all bytes are in hex.

Structure of a IScript

The entity, aka “Who are we talking to?”

The very first byte of a script states the entity the player is interacting with. If this is anything but 00 (a generic interaction), this will show that entity’s portrait when the script starts. The portrait is animated, and they all blink a lot. It can be unnerving.

Entity IDs can be any of the following:

  • 0x00: Generic interaction
  • 0x80: King
  • 0x81: Guru
  • 0x82: Martial Artist
  • 0x83: Magician
  • 0x84: Doctor
  • 0x85: Nurse
  • 0x86: Pink Shirt Guy
  • 0x87: Smoking Guy
  • 0x88: Meat Salesman
  • 0x89: Tools Salesman
  • 0x8A: Key Salesman

You’ll notice that each entity starts at 0x80. There’s a reason for this. 0x80 is 10000000 in binary. So the entities are all identified as numbers 0 through 10, with the most-significant bit set. The game checks if that bit is set (entity_id & 0x80) in order to determine if it should show a portrait. Then it masks out that bit and uses the number as the index into a portraits table. Neat!

Instructions

After this are instructions. These are actions that can be performed in a script. The game will play through the script’s actions until the script ends (which is an explicit action that all branches need to call!).

There are a lot of instructions. These include instructions that show messages, take or give gold/items/HP/MP, check for inventory items, check for or update the player’s experience title, open shops, check certain objectives or mark them complete, set a spawn point, show a password for the player’s state, goto a script address, or finish the game (the very last instruction ever run).

Here’s the full list:

InstructionDescriptionParameter 1Parameter 2
00End the script
01Show an unskippable messageMessage ID
02Show a question messageMessage ID
03Show a standard messageMessage ID
04Check if player’s eligible for a new title, jump if updatedJump address (2 bytes)
05Reduce player’s goldGold amount (2 bytes)
06Set temple to spawn inTemple ID
07Add item to player’s inventoryItem ID
08Open a shop windowShop items address (2 bytes)
09Add to the player’s goldGold amount (2 bytes)
0AAdd to the player’s magic pointsAmount
0BCheck if quest/objective is completed, jump if soQuest/objective IDCompleted jump address (2 bytes)
0CJump if player has titleTitle IDHas title jump address (2 bytes)
0DJump if the player has any goldJump address (2 bytes)
0ESet quest/objective completedQuest/objective ID
0FShow a buy/sell menu, jump if “buy” chosenBuy jump address (2 bytes)
10Remove item from player’s inventoryItem ID
11Show the sell menuShop items address (2 bytes)
12Jump if player has itemItem IDHas item jump address (2 bytes)
13Add health to playerAmount
14Show a password
15Finish the game
16Show question message, jump if completed (unused in game)Message IDCompleted jump address (2 bytes)
17Jump to addressAddress (2 bytes)

I won’t break those down in detail. I’ve documented the full reference with examples.

Shop items

Many of these instructions have to do with shops. Shops allow you to buy or sell an item.

Internally, a shop is a collection of item IDs and their prices. What you do with a shop is dependent on the action used to open it.

They look like this:

01     # Weapon: Long Sword
40 06  # 1600 gold

21     # Armor: Studded Mail
C4 09  # 2500 gold

40     # Shield: Small
4C 04  # 1100 gold

90     # Item: Red Potion
90 01  # 400 gold

8F     # Item: Wing Boots
F0 0A  # 2800 gold

FF     # End of shop

That’s the Tool Shop in the town of Forepaw.

Each item is 3 bytes: The item ID (1 byte) and the cost in gold (2 bytes).

It’s pretty compact. 5 items would only take 15 bytes.

In Faxanadu, every shop you interact with in the game uses its own shop items table with its own full list of items and their own prices. There’s no start to a shop, technically, so a mod could do something a bit more interesting, though. They could reuse shop data, making one shop a subset of another, by just pointing to an address within another shop.

Let’s go back to IScripts, and check out a few examples within Faxanadu.

Interacting with a shop.

We just looked at a shop, so let’s see how they tie into IScripts. Follow along with me!

  89     # Tools Salesman interaction

  03     # Show Message:
  0F     #     Message 3 ("I sell tools, what would you like?")
  0F     # Show Buy or Sell Menu:
  E0 A3  #     If "Buy", jump to _SHOW_BUY ($A3E0).
  11     # Show Sell Menu:
  6C A4  #     Shop address $A46C,
         #     Then continue after closed.
  03     # Show Message:
  02     #     Message 2 ("Thank you for shopping.")
  00     # End Script.

_SHOW_BUY:
  08     # Open Shop:
  6C A4  #     Shop address $A46C,
         #     Then continue after closed.
  03     # Show Message:
  02     #     Message 2 ("Thank you for shopping.")
  00     # End Script.

This is a common pattern for shops. First, a “Buy or Sell” menu is shown. If the player chooses “Buy”, it’ll jump to one address to open a shop in “Buy” mode. If they choose “Sell”, it’ll continue to the next instruction, and then open a shop in “Sell” mode.

Scripts can give these choices, or only offer a “Buy” or a “Sell”.

Working with the player’s inventory

Shops aren’t the only way to get items into or out of a player’s inventory. There are actions for that. Let’s take a look at this drunk guy in the town of Victim.

  00     # Generic interaction

  0C     # If Player Has Title:
  0A     #     "Soldier"
  5A A2  #     Then jump to _HAS_RANK ($A25A).
  03     # Show Message:
  74     #     Message 116 ("Go to the capital...")
  00     # End Script.
  
_HAS_RANK:
  12     # If Player Has Item:
  22     #    "Armor: Full Plate"
  63 A2  #    Then jump to _HAS_ARMOR ($A263).
  01     # Show Unskippable Message:
  75     #    Message 117 ("I've got this armor...").
  07     # Give Item To Player:
  22     #    "Armor: Full Plate".
  00     # End Script.
  
_HAS_ARMOR:
  03     # Show Message:
  76     #     Message 118 ("I'll have a drink...")
  00     # End Script.

Here, we have some more branching going on, some player title and item interaction. (Player titles, by the way, are based on your experience, and afford you more gold when you start your game next, more acceleration when walking/running, and maybe surprisingly, a shorter duration for the Wing Boots).

This script checks if the player reached the rank of “Soldier” or higher, they’ll be given the Full Plate armor, if they don’t already have it. If they haven’t advanced that far yet, they’ll be given instructions on where to go next. Dialogue changes based on these conditions.

Working with quests/objectives

Faxanadu tracks what we call “Quests,” which are special objectives the game tracks and includes as part of your password (something we’ll visit in the future).

Quests are represented internally as a few bits in a bitmask. When certain things happen, a particular bit is set. Some IScripts check bits to see how far the user has gotten and what needs to be done next.

When you interact with the man protecting the Spring of Trunk, the man’s IScript does a lot:

  1. It checks if you’ve completed the Spring of Trunk quest, in which case it’s done.
  2. Otherwise, it asks you if you have an elixir you can part with. If not dismissed, it checks for an Elixir in the player’s inventory.
  3. If an Elixir is present, it’ll take it, set the quest as complete, and tell you what to do next.
  4. If an Elixir is not present, you die. *Checks notes*. Sorry, you’re told to come back with an Elixir to awaken the spring.

Here’s how that script looks:

  00     # Generic interaction

  0B     # Check If Quest Complete:
  01     #     Quest 1 (Spring of Trunk),
  C0 A1  #     Then jump to _IS_COMPLETE ($A1C0).
  02     # Show Question Message:
  59     #     Message 89 ("This is the Spring of Trunk. Gimme an Elixir!").
  12     # If Player Has Item:
  92     #     "Item: Elixir",
  B9 A1  #     Then jump to _HAS_ELIXIR ($A1B9).
  03     # Show Message:
  5B     #     Message 91 ("Come back with an Elixir").
  00     # End Script.

_HAS_ELIXIR:
  01     # Show Unskippable Message:
  5A     #     Message 90 ("2 more springs to go!").
  10     # Remove Item from Player:
  92     #     "Item: Elixir".
  0E     # Set Quest Complete:
  01     #     Quest 1 (Spring of Trunk).
  00     # End Script.

_IS_COMPLETE:
  03     # Show Message:
  5C     #     Message 92 ("Find the poison in Mascon").
  00     # End Script.

You can get a sense as to how complex these can be.

How do you do, Guru?

We’ll end on two more related scripts.

In Faxanadu, you’ll visit Gurus to seek guidance, check your stock portfolio, get a title upgrade (if eligible), and see your new password for your next gaming session. Most Gurus are scripted the same way, with an initial block that’s location-specific and a jump to common Guru scripts (one for new title, one for showing a password):

  81     # Guru interaction

  06     # Set Spawn Point:
  03     #     Temple 3 (Mascon).
  04     # Check to Update Player Title:
  00 A6  #     If updated, jump to COMMON_GURU_NEW_TITLE ($A600).
  17     # Jump to:
  FC A5  #     COMMON_GURU_SHOW_PASSWORD ($A5FC).

COMMON_GURU_NEW_TITLE:
  01     # Show Unskippable Message:
  23     #     Message 35 ("I shall give you a title of...").
  00     # End Script.

COMMON_GURU_SHOW_PASSWORD:
  03     # Show Message:
  22     #     Message 34 ("I shall meditate with you").
  14     # Show Password.
  00     # End Script.

But in the town of Conflate, there’s a Guru that goes above and beyond by giving you the very important Ring of Dworf (you need this to unlock a certain door — yes, that’s the spelling, it’s not my fault) if you’ve met the criteria (you need the Battle Suit).

  81     # Guru interaction

  12     # If Player Has Item:
  23     #     "Armor: Battle Suit",
  34 A6  #     Then jump to _HAS_BATTLE_SUIT ($A634).

# This is the same logic normally seen in other Guru scripts.
_STANDARD_GURU_LOGIC:
  06     # Set Spawn Point:
  05     #     Temple 5 (Conflate).
  04     # Check to Update Player Title:
  00 A6  #     If updated, jump to COMMON_GURU_NEW_TITLE ($A600).
  17     # Jump to:
  FC A5  #     COMMON_GURU_SHOW_PASSWORD ($A5FC).

# And this is the new "Got a Battle Suit? Here's a ring to go with it!" logic.
_HAS_BATTLE_SUIT:
  12     # If Player Has Item:
  82     #     "Special Item: Ring of Dworf",
  2C A6  #     Then jump to _STANDARD_GURU_LOGIC.
  01     # Show Unskippable Message:
  8C     #     Message 140 ("Here's how you get to Dartmore...").
  07     # Add Item to Player:
  82     #     "Special Item: Ring of Dworf".
  00     # End Script.

You can see not only how they can make different versions of the same type of NPC more complex, but how scripts can jump to other scripts, reusing building blocks built elsewhere. While Faxanadu doesn’t have a ton of shared scripts, modders can do a lot with this to optimize their code.

IScripts: They’re super versatile!

Someone modding the game could do a lot to build upon this. For example:

  • Have the right combination of items fetched throughout the game, and you’ll be given an end-game item.
  • Shop prices change based on player title (using different shop addresses).
  • Available items in a shop grow or shink as you progress in the game.
  • Dismiss an NPC’s story too many times, and the NPC takes all your money.

And since IScripts also apply to item interactions:

  • Lose money when picking up an item.
  • Exchange one item for another.
  • Health-replenishing items.

This can all be built just using the IScript instructions. More advanced games can do even more by rewriting some of the assembly code backing these commands. But that’s a topic to cover later.

But how does it do all this?

Obviously these batches of bytes don’t do the work themselves. This isn’t assembly code, it’s just data.

Behind the scenes is a small scripting runtime, consisting of a few parts:

  1. A script runner function that can be called when the player collides with an item or interacts with an NPC.
  2. Implementations for each instruction.
  3. A table of instruction action handlers (split into a lower bytes table and upper bytes table).
  4. A table of IScript entrypoints (generally one slot per script, keyed off by an ID assigned to an item, NPC, or event, and also split into lower/upper bytes tables).

Starting a script

We start our journey at IScripts_Begin. This is our script runner. It takes the index into the IScript entrypoint table and does the following:

  1. Loads the address of the IScript, given the entrypoint in the table.
  2. Reads the first byte as the interaction entity ID, storing that so any portrait display can pull up the right images.
  3. Prepares for any text or shop display (IScripts almost always show a textbox or shop, so it gets this ready).
  4. Calls IScripts_InvokeNextAction to execute the first instruction.

The mainloop

IScripts_InvokeNextAction is effectively the main loop for the script runner. Its job is to:

  1. Read the next byte as an instruction.
  2. Looks up and runs the registered action handler in the table.

It doesn’t itself loop, but as you’ll see, action handlers will keep calling it.

IScript action handlers

Every handler is different, but generally:

  1. They read any following bytes needed to load all parameters.
  2. Perform whatever actions the handler needs to perform.
  3. Updates the next read address as needed to jump elsewhere in a script (used for Jump and conditions).
  4. At the end, they call IScripts_InvokeNextAction to continue on, or IScriptAction_EndScript to end the script.

It’s a simple loop. It does exactly what it needs to do. It’s fast and efficient and powers almost everything you interact with in the game.

Mods could extend the handlers table and build all-new action handlers for their game. It’d require moving some things around, but it would be entirely doable.

Time for a demo

Actually, I’ll give you two of my very own.

Demo 1: The ultimate Faxanadu speedrun

The first is the fastest speed run of Faxanadu, where our hero learns that there’s more to life than deserted towns and monsters hanging out in overgrown trees.

This uses a custom message along with the “Finish Game” instruction, which, well, ends the game. It looks like this:

00   # Generic entity

01   # Show Unskippable Message:
01   #     Message 01 ("... On second thought...").
15   # Finish Game.

Demo 2: Beer Run!

I like this one. It’s the very first mod built with IScripts. I built it it to test what I was disassembling and documenting.

Our hero comes home to find the local economy in ruins. Upon entering the town, a bum asks him to buy him a beer. This little fetch quest has a nice reward at the end.

It involves a number of new scripts (none of which I have anymore, oops — early testing). One NPC has been given shop capabilities, and another has a few items to hand out.

Next Up: Behavior Scripts

So that’s how Faxanadu handles interactions behind the scenes. Not a bunch of assembly logic (well, not just…), but rather simple-yet-powerful space-efficient scripts and a little scripting engine.

I mentioned before that this is just one of three scripting engines built into the game. Next time, we’ll dig into how enemies, bosses, and NPCs behave, using Behavior Scripts.

If you’d like to follow my Faxanadu work, you can check out the following:

And you can follow my efforts on:

]]> 1 chipx86 http://chipx86.blog <![CDATA[Using pprint.saferepr() for string comparisons in Python]]> https://chipx86.blog/?p=1592 2025-08-27T02:45:55Z 2025-08-27T02:45:55Z Python offers a handy module called pprint, which has helpers for formatting and printing data in a nicely-formatted way. If you haven’t used this, you should definitely explore it!

Most people will reach for this module when using pprint.pprint() (aka pprint.pp()) or pprint.pformat(), but one under-appreciated method is pprint.saferepr(), which advertises itself as returning a string representation of an object that is safe from some recursion issues. They demonstrate this as:

>>> import pprint
>>> stuff = ['spam', 'eggs', 'lumberjack', 'knights', 'ni']
>>> stuff.insert(0, stuff)
>>> pprint.saferepr(stuff)
"[<Recursion on list with id=4370731904>, 'spam', 'eggs', 'lumberjack', 'knights', 'ni']"

But actually repr() handles this specific case just fine:

>>> repr(stuff)
"[[...], 'spam', 'eggs', 'lumberjack', 'knights', 'ni']"

Pretty minimal difference there. Is there another reason we might care about saferepr()?

Dictionary string comparisons!

saferepr() first sets up the pprint pretty-printing machinery, meaning it takes care of things like sorting keys in dictionaries.

This is great! This is really handy when writing unit tests that need to compare, say, logging of data.

See, modern versions of CPython 3 will preserve insertion order of keys into dictionaries, which can be nice but aren’t always great for comparison purposes. If you’re writing a unit test, you probably want to feel confident about the string you’re comparing against.

Let’s take a look at repr() vs. saferepr() with a basic dictionary.

>>> d = {}
>>> d['z'] = 1
>>> d['a'] = 2
>>> d['g'] = 3
>>>
>>> repr(d)
"{'z': 1, 'a': 2, 'g': 3}"
>>>
>>> from pprint import saferepr
>>> saferepr(d)
"{'a': 2, 'g': 3, 'z': 1}"

A nice, stable order. Now we don’t have to worry about a unit test breaking on different versions or implementations or with different logic.

saferepr() will disable some of pprint()`’s typical output limitations. There’s no max dictionary depth. There’s no max line width. You just get a nice, normalized string of data for comparison purposes.

But not for sets…

Okay, it’s not perfect. Sets will still use their standard representation (which seems to be iteration order), and this might be surprising given how dictionaries are handled.

Interestingly, pprint() and pformat() will sort keys if it needs to break them across multiple lines, but otherwise, nope.

For example:

>>> from random import choices
>>> import pprint
>>> import string
>>> s = set(choices(string.ascii_letters, k=25))
>>> repr(s)
"{'A', 'D', 'b', 'J', 'T', 'X', 'C', 'V', 'e', 'v', 'I', 'w', 'H', 'k', 'M', 't', 'U', 'o', 'W'}"
>>> pprint.saferepr(s)
"{'A', 'D', 'b', 'J', 'T', 'X', 'C', 'V', 'e', 'v', 'I', 'w', 'H', 'k', 'M', 't', 'U', 'o', 'W'}"
>>> pprint.pp(s)
{'A',
 'C',
 'D',
 'H',
 'I',
 'J',
 'M',
 'T',
 'U',
 'V',
 'W',
 'X',
 'b',
 'e',
 'k',
 'o',
 't',
 'v',
 'w'}

Ah well, can’t have everything we want.

Still, depending on what you’re comparing, this can be a very handy tool in your Python Standard Library toolset.

]]> 0 chipx86 http://chipx86.blog <![CDATA[What If()? Using Conditional CSS Variables]]> https://chipx86.blog/?p=1516 2025-08-08T21:49:13Z 2025-08-08T09:54:34Z A cool new feature is coming to CSS: if().

With this, you can set a style or a CSS variable based on some condition, like whether the device is on a touchscreen, or whether some CSS property is supported, or even based on an attribute on an element.

It’s really cool, and sooner or later we’ll all be (ab)using it.

Actually, you can pretty much do this today using CSS Variables, with a little understanding of how variable fallbacks work and how you can use this to make your own conditionals.

What are CSS Variables?

CSS Variables make it easy to use reuse colors and styles across a CSS codebase. At their most basic, they look like this:

:root {
    --my-fg-color: red;
    --my-bg-color: blue;
}

body {
    color: var(--my-fg-color);
    background: var(--my-bg-color);
}

CSS Variables can have fallbacks, which falls back to some other value if the variable isn’t defined. Here, the color will be black if --my-custom-fg-color wasn’t defined in the scope.

body {
    color: var(--my-custom-fg-color, black);
}

This can be nested with other CSS variables:

body {
    color: var(--my-custom-fg-color, var(--my-fg-color, black));
}

Fallbacks depend on whether the variable is truthy or falsy.

Truthy and Falsy Variables

When a variable is falsy, var() will use the fallback value.

When it is truthy, the fallback value won’t be used.

Normally, it’s falsy if the variable isn’t defined, and truthy if it is. But there are a couple tricks to know about:

  1. You can make a variable falsy by setting it to the special keyword initial.
  2. You can make a variable truthy (without expanding to a value) by making the value empty.

Let me show you what I mean:

:root {
    /* Empty -- Truthy and a no-op */
    --no-fallback: ;          

    /* 'initial' -- Falsy, use fallback */
    --use-fallback: initial;
}

body {
    background: var(--no-fallback, grey);
    color: var(--use-fallback, blue);
}

This will result in the following CSS:

body {
    background: ;  /* No value -- ignored. */
    color: blue;
}

Our new --use-fallback will ensure the fallback value will be used, and --no-fallback will ensure it’s never used (but won’t leave behind a value).

These are our core building blocks for conditional CSS Variables.

We can then combine these rules.

body {
    background:
        var(--no-fallback, red)
        var(--use-fallback, pink);

    color:
        var(--no-fallback, pink)
        var(--use-fallback, red);
}

This will give us:

body {
    background: pink;
    color: red;
}

Behold:

A red on pink "Hello, world!"

You can play around with it here:

Armed with this knowledge, we can now start setting CSS Variables based on other state, and trigger the conditions.

Setting Conditional Variables

Let’s set up another contrived example. We’re going to put together some HTML with a button that styles differently based on where it’s placed. In this case, when in <main>, it will appear blue-on-pink and elsewhere it’ll appear pink-on-blue.

It’s my post and I will use hideous color palettes if I want to.

Here’s our HTML:

<html>
 <body>
  <button>Hello</button>
  <main>
   <button>There</button>
  </main>
  <button>Friend</button>
 </body>
</html>

We’re going to set up some conditional state variables based on the main placement. (Yep, contrived example, but this is just for demonstration purposes.)

Here’s our CSS:

:root {
    --if-main: ;
    --if-not-main: initial;
}

main {
    --if-main: initial;
    --if-not-main: ;
}

button {
    background:
        var(--if-main, pink)
        var(--if-not-main, blue);

    color:
        var(--if-main, blue)
        var(--if-not-main, pink);
}

By default, --if-main will be truthy (so no fallback) and --if-not-main will be falsy (so it’ll use the fallback).

When in <main>, we reverse that.

The button will style the background and foreground colors based on that state.

And here’s what we get:

Three buttons. "Hello" with pink on blue, "There" with blue on pink, and "Friend" with pink on blue.

Beautiful.

You can play with that here:

Let’s Build Dark Mode

It’s 2025, so we want to support light and dark modes. There’s no end of articles on how to do this, and now you’re reading another one of them.

A common approach is to write different rules based on CSS classes.

body.-is-light {
    background: orange;
    color: green;
}

body.-is-dark {
    background: darkorange;
    color: darkgreen;
}

That works if we’re manually controlling CSS classes on body, but these days we want to respect system settings, so we use media selectors like so:

@media (prefers-color-scheme: light) {
    body {
        background: orange;
        color: green;
    }
}

@media (prefers-color-scheme: dark) {
    body {
        background: darkorange;
        color: darkgreen;
    }
}

But now we can’t give users a nice toggle on the page to control their light/dark modes, so we’ll want to bring the CSS classes back with the media selectors:

body.-is-light {
    background: orange;
    color: green;
}

body.-is-dark button {
    background: darkorange;
    color: darkgreen;
}

@media (prefers-color-scheme: light) {
    body {
        background: orange;
        color: green;
    }
}

@media (prefers-color-scheme: dark) {
    body {
        background: darkorange;
        color: darkgreen;
    }
}

Okay, now things are just out of control. And we haven’t moved beyond the <body> tag. We’d have to repeat this for everything else we want to style!

Yuck. I hate CSS.

Let’s Hate CSS Less

It all comes down to this. We’re going to take our building blocks and clean this all up.

Here’s our plan of attack:

  1. We’re going to define some truthy and falsy CSS variables saying if we’re in light or dark mode.
  2. We’re going to consolidate the styles for our components and use our conditional var() statements.

We’ll first set up our state variables. We only need to do this once for the whole codebase.

/* We'll make light mode our default. Everyone loves light mode. */
:root,
:root.-is-light {
    --if-light: initial;
    --if-dark: ;

    color-scheme: light;
}

:root.-is-dark {
    --if-light: ;
    --if-dark: initial;

    color-scheme: dark light;
}

@media (prefers-color-scheme: light) {
    :root {
        --if-light: initial;
        --if-dark: ;

        color-scheme: light;
    }
}

@media (prefers-color-scheme: dark) {
    :root {
        --if-light: ;
        --if-dark: initial;

        color-scheme: dark light;
    }
}

Now let’s use all that to style <body> and <button>:

body {
    background:
        var(--if-light, orange)
        var(--if-dark, darkorange);

    color:
        var(--if-light, darkgreen)
        var(--if-dark, blue);
}

button {
    background:
        var(--if-light, pink)
        var(--if-dark, blue);

    color:
        var(--if-light, blue)
        var(--if-dark, pink);
}

Look at that. So much simpler to maintain. And we can extend that too, if we want to add high-constrast mode or something.

Let’s test it.

Here it is when we switch our system to dark mode:

A blue-on-dark-orange page saying "Welcome to my beautiful page.", with three pink-on-blue buttons saying "Hello", "There", and "Friend".

Far less eye strain than light. I’m sold.

Now light mode:

A green-on-light-orange page saying "Welcome to my beautiful page.", with three blue-on-pink buttons saying "Hello", "There", and "Friend".

And system dark mode with <html class="is-light"> overriding to light mode.

A green-on-light-orange page saying "Welcome to my beautiful page.", with three blue-on-pink buttons saying "Hello", "There", and "Friend".

And system light mode with <html class="is-dark"> overriding to dark mode.

A blue-on-dark-orange page saying "Welcome to my beautiful page.", with three pink-on-blue buttons saying "Hello", "There", and "Friend".

Perfection.

You can play with that one here:

Nesting Conditionals!

That’s right, you can nest them! Let’s update our Light/Dark Mode example to make a prettier color scheme, because I’m starting to realize what we had was kind of ugly.

We’ll define new --if-pretty and --if-ugly states to go along with --if-dark and --if-light:

:root,
:root.-is-light {
    --if-pretty: ;
    --if-ugly: initial;

    ...
}

:root.-is-pretty {
    --if-pretty: initial;
    --if-ugly: ;
}

...

And now we can build some truly beautiful styling, using nested conditionals:

body {
    background:
        var(--if-pretty,
            var(--if-light,
                linear-gradient(
                    in hsl longer hue 45deg,
                    red 0 100%)
                )
            )
            var(--if-dark,
                black
                linear-gradient(
                    in hsl longer hue 45deg,
                    rgba(255, 0, 0, 20%) 0 100%)
                )
            )
        )
        var(--if-ugly,
            var(--if-light, orange)
            var(--if-dark, darkorange)
        );

    color:
        var(--if-pretty,
            var(--if-light, black)
            var(--if-dark, white)
        )
        var(--if-ugly,
            var(--if-light, darkgreen)
            var(--if-dark, blue)
        );

    font-size: 100%;
}

button {
    background:
        var(--if-pretty,
            radial-gradient(
                ellipse at top,
                orange,
                transparent
            ),
            radial-gradient(
                ellipse at bottom,
                orange,
                transparent
            )
        )
        var(--if-ugly,
            var(--if-light, pink)
            var(--if-dark, blue)
        );

  color:
        var(--if-pretty, black)
        var(--if-ugly,
            var(--if-light, blue)
            var(--if-dark, pink)
        );
}

In standard Ugly Mode, this looks the same as before, but when we apply Pretty Mode with Light Mode, we get:

A black-on-rainbow page saying "Welcome to my beautiful page.", with three orange circular gradient buttons saying "Hello", "There", and "Friend".

And then with Dark Mode:

A white-on-dark-rainbow page saying "Welcome to my beautiful page.", with three orange circular gradient buttons saying "Hello", "There", and "Friend".

I am available for design consulting work on a first-come, first-serve basis only.

Here it is so you can enjoy it yourself:

A Backport from If()

Google’s article on if() has a neat demo showing how you can style some cards based on the value of a data-status= attribute. Based on whether the value is pending, complete, or anything else, the cards will be placed in different columns and have different background and border colors.

Here’s the card:

<div class="card" data-status="complete">
  ...
</div>

Here’s how you’d do that with if() (from their demo):

.card {
    --status: attr(data-status type(<custom-ident>));

    border-color: if(
        style(--status: pending): royalblue;
        style(--status: complete): seagreen;
        else: gray);

    background-color: if(
        style(--status: pending): #eff7fa;
        style(--status: complete): #f6fff6;
        else: #f7f7f7);

    grid-column: if(
        style(--status: pending): 1;
        style(--status: complete): 2;
        else: 3);
}

To do this with CSS Variables, we can define our states using CSS selectors on the attribute, and then style it the way we did earlier:

/* Define our variables and conditions. */
.card {
  --if-pending: ;
  --if-complete: ;
  --if-default: initial;
}

.card[data-status="pending"] {
  --if-pending: initial;
  --if-default: ;
}

.card[data-status="complete"] {
  --if-complete: initial;
  --if-default: ;
}

/* Style our cards. */
.card {
  border-color:
    var(--if-pending, royalblue)
    var(--if-complete, seagreen)
    var(--if-default, grey);

  background-color:
    var(--if-pending, #eff7fa)
    var(--if-complete, #f6fff6)
    var(--if-default, #f7f7f7);

  grid-column:
    var(--if-pending, 1)
    var(--if-complete, 2)
    var(--if-default, 3);
}

Here’s the ported version, live:

It’s Available Now!

You can use this today in all browsers that support CSS Variables. That’s.. counts.. years worth of browsers.

It requires a bit more work to set up the variables, but the nice thing is, once you’ve done that, the rest is highly maintainable. And usage is close enough to that of if() that you can more easily transition once support is widespread.

We’ve based Ink, our (still very young) CSS component library we use for Review Board, around this conditionals pattern. It’s helped us to support light and dark modes along with the beginnings of low- and high-contrast modes and state for some components without tearing our hair out.

Give it a try, play with the demos, and see if it’s a good fit for your codebase. Begin taking advantage of what if() has to offer today, so you can more easily port tomorrow.

]]> 0 chipx86 http://chipx86.blog <![CDATA[A crash course on Python function signatures and typing]]> https://chipx86.blog/?p=1457 2025-08-08T22:38:12Z 2025-07-12T22:17:38Z I’ve been doing some work on Review Board and our utility library Typelets, and thought I’d share some of the intricacies of function signatures and their typing in Python.

We have some pretty neat Python typing utilities in the works to help a function inherit another function’s types in their own *args and **kwargs without rewriting a TypedDict. Useful for functions that need to forward arguments to another function. I’ll talk more about that later, but understanding how it works first requires understanding a bit about how Python sees functions.

Function signatures

Python’s inspect module is full of goodies for analyzing objects and code, and today we’ll explore the inspect.signature() function.

inspect.signature() is used to introspect the signature of another function, showing its parameters, default values, type annotations, and more. It can also aid IDEs in understanding a function signature. If a function has __signature__ defined, it will reference this (at least in CPython), and this gives highly-dynamic code the ability to patch signatures at runtime. (If you’re curious, here’s exactly how it works).

There are a few places where knowing the signature can be useful, such as:

  • Automatically-crafting documentation (Sphinx does this)
  • Checking if a callback handler accepts the right arguments
  • Checking if an implementation of an interface is using deprecated function signatures

Let’s set up a function and take a look at its signature.

>>> def my_func(
...     a: int,
...     b: str,
...     /,
...     c: dict[str, str] | None,
...     *,
...     d: bool = False,
...     **kwargs,
... ) -> str:
...     ...
... 
>>> import inspect
>>> sig = inspect.signature(my_func)

>>> sig
<Signature (a: int, b: str, /, c: dict[str, str] | None, *,
d: bool = False, **kwargs) -> str>

>>> sig.parameters
mappingproxy(OrderedDict({
    'a': <Parameter "a: int">,
    'b': <Parameter "b: str">,
    'c': <Parameter "c: dict[str, str] | None">,
    'd': <Parameter "d: bool = False">,
    'kwargs': <Parameter "**kwargs">
}))

>>> sig.return_annotation
<class 'str'>

>>> sig.parameters.get('c')
<Parameter "c: dict[str, str] | None">

>>> 'kwargs' in sig.parameters
True

>>> 'foo' in sig.parameters
False

Pretty neat. Pretty useful, when you need to know what a function takes and returns.

Let’s see what happens when we work with methods on classes.

>>> class MyClass:
...     def my_method(
...         self,
...         *args,
...         **kwargs,
...     ) -> None:
...         ...
...
>>> inspect.signature(MyClass.my_method)
<Signature (self, *args, **kwargs) -> None>

Seems reasonable. But…

>>> obj = MyClass()
>>> inspect.signature(obj.my_method)
<Signature (*args, **kwargs) -> None>

self disappeared!

What happens if we do this on a classmethod? Place your bets…

>>> class MyClass2:
...     @classmethod
...     def my_method(
...         cls,
...         *args,
...         **kwargs,
...     ) -> None:
...         ...
...
>>> inspect.signature(MyClass2.my_method)
<Signature (*args, **kwargs) -> None>

If you guessed it wouldn’t have cls, you’d be right.

Only unbound methods (definitions of methods on a class) will have a self parameter in the signature. Bound methods (callable methods bound to an instance of a class) and classmethods (callable methods bound to a class) don’t. And this makes sense, if you think about it, because this signature represents what the call accepts, not what the code defining the method looks like.

You don’t pass a self when calling a method on an object, or a cls when calling a classmethod, so it doesn’t appear in the function signature. But did you know that you can call an unbound method if you provide an object as the self parameter? Watch this:

>>> class MyClass:
...     def my_method(self) -> None:
...         self.x = 42
...
>>> obj = MyClass()
>>> MyClass.my_method(obj)
>>> obj.x
42

In this case, the unbound method MyClass.my_method has a self argument in its signature, meaning it takes it in a call. So, we can just pass in an instance. There aren’t a lot of cases where you’d want to go this route, but it’s helpful to know how this works.

What are bound and unbound methods?

I briefly touched upon this, but:

  • Unbound methods are just functions. Functions that are defined on a class.
  • Bound methods are a function where the very first argument (self or cls) is bound to a value.

Binding normally happens when you instantiate a class, but you can do it yourself through any function’s __get__():

>>> def my_func(self):
...     print('I am', self)
... 
>>> class MyClass:
...     ...
...
>>> obj = MyClass()
>>> method = my_func.__get__(MyClass)
>>> method
<bound method my_func of <__main__.MyClass object at 0x100ea20d0>>

>>> method.__self__
<__main__.MyClass object at 0x100ea20d0>

>>> inspect.ismethod(method)
True

>>> method()
I am <__main__.MyClass object at 0x100ea20d0>

my_func wasn’t even defined on a class, and yet we could still make it a bound method tied to an instance of MyClass.

You can think of a bound method as a convenience over having to pass in an object as the first parameter every time you want to call the function. As we saw above, we can do exactly that, if we pass it to the unbound method, but binding saves us from doing this every time.

You’ll probably never need to do this trick yourself, but it’s helpful to know how it all ties together.

By the way, @staticmethod is a way of telling Python to never make an unbound method into a bound method when instantiating the object (it stays a function), and @classmethod is a way of telling Python to bind it immediately to the class it’s defined on (and not rebind when instantiating an object from the class).

How do you tell them apart?

If you have a function, and you don’t know if it’s a standard function, a classmethod, a bound method, or an unbound method, how can you tell?

  1. Bound methods have a __self__ attribute pointing to the parent object (and inspect.ismethod() will be True).
  2. Classmethods have a __self__ attribute pointing to the parent class (and inspect.ismethod() will be True).
  3. Unbound methods are tricky:
    • They do not have a __self__ attribute.
    • They might have a self or cls parameter in the signature, but they might not have those names (and other functions may define them).
    • They should have a . in its __qualname__ attribute. This is a full .-based path to the method, relative to the module.
    • Splitting __qualname__, the last component would be the name. The previous component won’t be <locals> (but if <locals> is found, you’re going to have trouble getting to the method).
    • If the full path is resolvable, the parent component should be a class (but it might not be).
    • You could… resolve the parent module to a file and walk its AST and find the class and method based on __qualname__. But this is expensive and probably a bad idea for most cases.
  4. Standard functions are the fallback.

Since unbound methods are standard functions that are just defined on a class, it’s difficult to really tell the difference. You’d have to rely on heuristics and know you won’t always get a definitive answer.

(Interesting note: In Python 2, unbound methods were special kinds of functions with a __self__ that pointed to the class they were defined on, so you could easily tell!)

The challenges of typing

Functions can be typed using Callable[[<param_types>], <return_type>].

This is very simplistic, and can’t be used to represent positional-only arguments, keyword-only arguments, default arguments, *args, or **kwargs. For that, you can define a Protocol with __call__:

from typing import Protocol


class MyCallback(Protocol):
    def __call__(
        self,  # This won't be part of the function signature
        a: int,
        b: str,
        /,
        c: dict[str, str] | None,
        *,
        d: bool = False,
        **kwargs,
    ) -> str:
        ...

Type checkers will then treat this as a Callable, effectively. If we take my_func from the top of this post, we can assign to it:

cb: MyCallback = my_func  # This works

What if we want to assign a method from a class? Let’s try bound and unbound.

class MyClass:
    def my_func(
        self,
        a: int,
        b: str,
        /,
        c: dict[str, str] | None,
        *,
        d: bool = False,
        **kwargs,
    ) -> str:
        return '42'

cb2: MyCallback = MyClass.my_func  # This fails
cb3: MyCallback = MyClass().my_func  # This works

What happened? It’s that self again. Remember, the unbound method has self in the signature, and the bound method does not.

Let’s add self and try again.

class MyCallback(Protocol):
    def __call__(
        _proto_self,  # This won't be part of the function signature
        self: Any,
        a: int,
        b: str,
        /,
        c: dict[str, str] | None,
        *,
        d: bool = False,
        **kwargs,
    ) -> str:
        ...

cb2: MyCallback = MyClass.my_func    # This works
cb3: MyCallback = MyClass().my_func  # This fails

What happened this time?! Well, now we’ve matched the unbound signature with self, but not the bound signature without it.

Solving this gets… verbose. We can create two versions of this: Unbound, and Bound (or plain function, or classmethod):

class MyUnboundCallback(Protocol):
    def __call__(
        _proto_self,  # This won't be part of the function signature
        self: Any,
        a: int,
        b: str,
        /,
        c: dict[str, str] | None,
        *,
        d: bool = False,
        **kwargs,
    ) -> str:
        ...

class MyCallback(Protocol):
    def __call__(
        _proto_self,  # This won't be part of the function signature
        a: int,
        b: str,
        /,
        c: dict[str, str] | None,
        *,
        d: bool = False,
        **kwargs,
    ) -> str:
        ...

# These work
cb4: MyCallback = my_func
cb5: MyCallback = MyClass().my_func


# These fail correctly
cb7: MyUnboundCallback = my_func
cb8: MyUnboundCallback = MyClass().my_func
cb9: MyCallback = MyClass.my_func

This means we can use union types (MyUnboundCallback | MyCallback) to cover our bases.

It’s not flawless. Depending on how you’ve typed your signature, and the signature of the function you’re setting, you might not get the behavior you want or expect. As an example, any method with a leading self-like parameter (basically any parameter coming before your defined signature) will type as MyUnboundCallback, because it might be! Remember, we can turn any function into a bound method for an arbitrary class using __get__. That may or may not matter, depending on what you need to do.

What do I mean by that?

def my_bindable_func(
    x,
    a: int,
    b: str,
    /,
    c: dict[str, str] | None,
    *,
    d: bool = False,
    **kwargs,
) -> str:
    return ''


x1: MyCallback = my_bindable_func         # This fails
x2: MyUnboundCallback = my_bindable_func  # This works

x may not be named self, but it’ll get treated as one, because if we do my_bindable_func.__get__(some_obj), then some_obj will be bound to x and callers won’t have to pass anything to x.

Okay, what if you want to return a function that can behave as an unbound method (with self) that can become an unbound method (with __get__)? We can mostly do it with:

from typing import ParamSpec, TypeVar, cast, overload


_C = TypeVar('_C')
_R_co = TypeVar('_R_co', covariant=True)
_P = ParamSpec('_P')


class MyMethod(Protocol[_C, _P, _R_co]):
    __self__: _C

    @overload
    def __get__(
        self,
        instance: None,
        owner: type[_C],
    ) -> Callable[Concatenate[_C, _P], _R_co]:
        ...

    @overload
    def __get__(
        self,
        instance: _C,
        owner: type[_C],
    ) -> Callable[_P, _R_co]:
        ...

    def __get__(
        self,
        instance: _C | None,
        owner: type[_C],
    ) -> (
        Callable[Concatenate[_C, _P], _R_co] |
        Callable[_P, _R_co]
    ):
        ...

Putting it into practice:

def make_method(
    source_method: Callable[Concatenate[_C, _P], _R_co],
) -> MyMethod[_C, _P, _R_co]:
    return cast(MyMethod, source_method)


class MyClass2:
    @make_method
    def my_method(
        self,
        a: int,
        b: str,
        /,
        c: dict[str, str] | None,
        *,
        d: bool = False,
        **kwargs,
    ) -> str:
        return True


# These work!
MyClass2().my_method(1, 'x', {}, d=True)
MyClass2.my_method(MyClass2(), 1, 'x', {}, d=True)

That’s a fair bit of work, but it satisfies the bound vs. unbound methods signature differences. If we inspect these:

>>> reveal_type(MyClass2.my_method)
Type of "MyClass2.my_method" is "(MyClass2, a: int, b: str, /,
c: dict[str, str] | None, *, d: bool = False, **kwargs: Unknown) -> str"

>>> reveal_type(MyClass2().my_method)
Type of "MyClass2().my_method" is "(a: int, b: str, /,
c: dict[str, str] | None, *, d: bool = False, **kwargs: Unknown) -> str"

And those are type-compatible with the MyCallback and MyUnboundCallback we built earlier, since the signatures match:

# These work
cb10: MyUnboundCallback = MyClass.my_method
cb11: MyCallback = MyClass().my_method

# These fail correctly
cb12: MyUnboundCallback = MyClass().my_method
cb13: MyCallback = MyClass.my_method

And if we wanted, we could modify that ParamSpec going into the MyMethod from make_method() and that’ll impact what the type checkers expect during the call.

Hopefully you can see how this can get complex fast, and involve some tradeoffs.

I personally believe Python needs a lot more love in this area. Types for the different kinds of functions/methods, better specialization for Callable, and some of the useful capabilities from TypeScript would be nice (such as Parameters<T>, ReturnType<T>, OmitThisParameter<T>, etc.). But this is what we have to work with today.

My teachers said to always write a conclusion

What have we learned?

  • Python’s method signatures are different when bound vs. unbound, and this can affect typing.
  • Unbound methods aren’t really their own thing, and this can lead to some challenges.
  • Any method can be a bound method with a call to __get__().
  • Callable only gets you so far. If you want to type complex functions, write a Protocol with a __call__() signature.
  • If you want to simulate a bound/unbound-aware type, you’ll need Protocol with __get__().

I feel like I just barely scratched the surface here. There’s a lot more to functions, working with signatures, and challenges around typing than I covered here. We haven’t talked about how you can rewrite signatures on the fly, how annotations are represented, what functions look under the hood, or how bytecode behind functions are mutable at runtime.

I’ll leave some of that for future posts. And I’ll have more to talk about when we expand Typelets with the new parameter inheritance capabilities. It builds upon a lot of what I covered today to perform some very neat and practical tricks for library authors and larger codebases.

What do you think? Did you learn something? Did I get something wrong? Have you done anything interesting or unexpected with signatures or typing around functions you’d like to share? I want to hear about it!

]]> 0 chipx86 http://chipx86.blog <![CDATA[Tip: Use keyword-only arguments in Python dataclasses]]> https://chipx86.blog/?p=1444 2025-06-30T06:46:33Z 2025-06-29T10:10:14Z Python dataclasses are a really nice feature for constructing classes that primarily hold or work with data. They can be a good alternative to using dictionaries, since they allow you to add methods, dynamic properties, and subclasses. They can also be a good alternative to building your own class by hand, since they don’t need a custom __init__() that reassigns attributes and provide methods like __eq__() out of the box.

One small tip to keeping dataclasses maintainable is to always construct them with kw_only=True, like so:

from dataclasses import dataclass


@dataclass(kw_only=True)
class MyDataClass:
    x: int
    y: str
    z: bool = True

This will construct an __init__() that looks like this:

class MyDataClass:
    def __init__(
        self,
        *,
        x: int,
        y: str,
        z: bool = True,
    ) -> None:
        self.x = x
        self.y = y
        self.z = z

Instead of:

class MyDataClass:
    def __init__(
        self,
        x: int,
        y: str,
        z: bool = True,
    ) -> None:
        self.x = x
        self.y = y
        self.z = z

That * in the argument list means everything that follows must be passed as a keyword argument, instead of a positional argument.

There are two reasons you probably want to do this:

  1. It allows you to reorder the fields on the dataclass without breaking callers. Positional arguments means a caller can use MyDataClass(1, 'foo', False), and if you remove/reorder any of these arguments, you’ll break those callers unexpectedly. By forcing callers to use MyDataClass(x=1, y='foo', z=False), you remove this risk.
  2. It allows subclasses to add required fields. Normally, any field with a default value (like z above) will force any fields following it to also have a default. And that includes all fields defined by subclasses. Using kw_only=True gives subclasses the flexibility to decide for themselves which fields must be provided by the caller and which have a default.

These reasons are more important for library authors than anything. We spend a lot of time trying to ensure backwards-compatibility and forwards-extensibility in Review Board, so this is an important topic for us. And if you’re developing something reusable with dataclasses, it might be for you, too.

Update: One important point I left out is Python compatibility. This flag was introduced in Python 3.10, so if you’re supporting older versions, you won’t be able to use this just yet. If you want to optimistically enable this just on 3.10+, one approach would be:

import sys
from dataclasses import dataclass


if sys.version_info[:2] >= (3, 10):
    dataclass_kwargs = {
        'kw_only': True,
    }
else:
    dataclass_kwargs = {}

...

@dataclass(**dataclass_kwargs)
class MyDataClass:
    ...
...

But this won’t solve the subclassing issue, so you’d still need to ensure any subclasses use default arguments if you want to support versions prior to 3.10.

]]> 0 chipx86 http://chipx86.blog <![CDATA[CodeMirror and Spell Checking: Solved]]> https://chipx86.blog/?p=1182 2025-06-29T02:01:24Z 2025-06-27T01:15:19Z A screenshot of the CodeMirror editor with spelling issues and the browser's spell checking menu opened.

For years I’ve wanted spell checking in CodeMirror. We use CodeMirror in our Review Board code review tool for all text input, in order to allow on-the-fly syntax highlighting of code, inline image display, bold/italic, code literals, etc.

(We’re using CodeMirror v5, rather than v6, due to the years’ worth of useful plugins and the custom extensions we’ve built upon it. CodeMirror v6 is a different beast. You should check it out, but we’re going to be using v5 for our examples here. Much of this can likely be leveraged for other editing components as well.)

CodeMirror is a great component for the web, and I have a ton of respect for the project, but its lack of spell checking has always been a complaint for our users.

And the reason for that problem lies mostly on the browsers and “standards.” Starting with…

ContentEditable Mode

Browsers support opting an element into what’s called Content Editable mode. This allows any element to become editable right in the browser, like a fancy <textarea>, with a lot of pretty cool capabilities:

  • Rich text editing
  • Rich copy/paste
  • Selection management
  • Integration with spell checkers, grammar checkers, AI writing assistants
  • Works as you’d expect with your device’s native input methods (virtual keyboard, speech-to-text, etc.)

Simply apply contenteditable="true" to an element, and you can begin typing away. Add spellcheck="true" and you get spell checking for free. Try it!

And maybe you don’t even need spellcheck="true"! The browser may just turn it on automatically. But you may need spellcheck="false" if you don’t want it on. And it might stay on anyway!

Here we reach the first of many inconsistencies. Content Editable mode is complex and not perfectly consistent across browsers (though it’s gotten better). A few things you might run into include:

  • Ranges for selection events and input events can be inconsistent across browsers and are full of edge cases (you’ll be doing a lot of “let me walk the DOM and count characters carefully to find out where this selection really starts” checks).
  • Spell checking behaves quite differently on different browsers (especially in Chrome and Safari, which might recognize a word as misspelled but won’t always show it).
  • Rich copy/paste may mess with your DOM structure in ways you don’t expect.
  • Programmatic manipulating of the text content using execCommand is deprecated with no suitable replacement (and you don’t want to mess with the DOM directly or you break Undo/Redo). It also doesn’t always play nice with input events.

CodeMirror v5 tries to let the browser do its thing and sync state back, but this doesn’t always work. Replacing misspelled words on Safari or Chrome can sometimes cause text to flip back-and-forth. Data can be lost. Cursor positions can change. It can be a mess.

So while CodeMirror will let you enable both Content Editable and Spell Checking modes, it’s at your own peril.

Which is why we never enabled it.

How do we fix this?

When CodeMirror v5 was introduced, there weren’t a lot of options. But browsers have improved since.

The secret sauce is the beforeinput event.

There are a lot of operations that can involve placing new text in a Content Editable:

  • Replacing a misspelled word
  • Using speech-to-text
  • Having AI generate content or rewrite existing content
  • Transforming text to Title Case

These will generate a beforeinput event before making the change, and an input event after making the change.

Both events provide:

  1. The type of operation:
    1. insertText for text-to-speech or newly-generated text
    2. insertReplacementText for spelling replacements, AI rewrites, and other similar operations
  2. The range of text being replaced (or where new text will be inserted)
  3. The new data (either as InputEvent.data in the form of one or more InputEvent.dataTransferItem.items[] entries)

Thankfully, beforeinput can be canceled, which prevents the operation from going through.

This is our way in. We can treat these operations as requests that CodeMirror can fulfill, instead of changes CodeMirror must react to.

A screenshot of a text field in CodeMirror with text highlighted and macOS's AI Writing Tools providing a concise version of the text.

Putting our plan into action

Here’s the general approach:

  1. Listen to beforeinput on CodeMirror’s input element (codemirror.display.input.div).
  2. Filter for the following InputEvent.inputType values: 'insertReplacementText', 'insertText'.
  3. Fetch the ranges and the new plain text data from the InputEvent.
  4. For each range:
    1. Convert each range into a start/end line number within CodeMirror, and a start/end within each line.
    2. Issue a CodeMirror.replaceRange() with the normalized ranges and the new text.

Simple in theory, but there’s a few things to get right:

  1. Different browsers and different operations will report those ranges on different elements. They might be text nodes, they might be a parent element, or they might be the top-level contenteditable element. Or a combination. So we need to be very careful about our assumptions.
  2. We need to be able to calculate those line numbers and offsets. We won’t necessarily have that information up-front, and it depends on what nodes we get in the ranges.
  3. The text data can come from more than one place:
    1. An InputEvent.data attribute value
    2. One or more strings accessed asynchronously from InputEvent.dataTransfer.items[], in plain text, HTML, or potentially other forms.
  4. We may not have all of this! Even as recently as late-2024, Chrome wasn’t giving me target ranges in beforeinput, only in input, which was too late. So we’ll want to bail if anything goes wrong.

Let’s put this into practice. I’ll use TypeScript to help make some of this a bit more clear, but you can do all this in JavaScript.

Feel free to skip to the end, if you don’t want to read a couple pages of TypeScript.

1. Set up our event handler

We’re going to listen to beforeinput. If it’s an event we care about, we’ll grab the target ranges, take over from the browser (by canceling the event), and then prepare to replay the operation using CodeMirror’s API.

This is going to require a bit of help figuring out what lines and columns those ranges correspond to, which we’ll tackle next.

const inputEl = codeMirror.display.input.div;
	
inputEl.addEventListener('beforeinput',
                         (evt: InputEvent) => {
    if (evt.inputType !== 'insertReplacementText' &&
        evt.inputType !== 'insertText') {
        /*
         * This isn't a text replacement or new text event,
         * so we'll want to let the browser handle this.
         *
         * We could just preventDefault()/stopPropagation()
         * if we really wanted to play it safe.
         */
        return;
    }

    /*
     * Grab the ranges from the event. This might be
     * empty, which would have been the case on some
     * versions of Chrome I tested with before. Play it
     * safe, bail if we can't find a range.
     *
     * Each range will have an offset in a start container
     * and an offset in an end container. These containers
     * may be text nodes or some parent node (up to and
     * including inputEl).
     */
    const ranges = evt.getTargetRanges();

    if (!ranges || ranges.length === 0) {
        /* We got empty ranges. There's nothing to do. */
        return;
    }

    const newText =
           evt.data
        ?? evt.dataTransfer?.getData('text')
        ?? null;

	if (newText === null) {
		/* We couldn't locate any text, so bail. */
        return;
	}

    /*
     * We'll take over from here. We don't want the browser
     * messing with any state and impacting CodeMirror.
     * Instead, we'll run the operations through CodeMirror.
     */
    evt.preventDefault();
    evt.stopPropagation();

    /*
     * Place the new text in CodeMirror.
     *
     * For each range, we're getting offsets CodeMirror
     * can understand and then we're placing text there.
     *
     * findOffsetsForRange() is where a lot of magic
     * happens.
     */
    for (const range of state.ranges) {
        const [startOffset, endOffset] =
            findOffsetsForRange(range);

        codeMirror.replaceRange(
            newText,
            startOffset,
            endOffset,
            '+input',
        );
    }
});

This is pretty easy, and applicable to more than CodeMirror. But now we’ll get into some of the nitty-gritty.

2. Map from ranges to CodeMirror positions

Most of the hard work really comes from mapping the event’s ranges to CodeMirror line numbers and columns.

We need to know the following:

  1. Where each container node is in the document, for each end of the range.
  2. What line number each corresponds to.
  3. What the character offset is within that line.

This ultimately means a lot of traversing of the DOM (we can use TreeWalker for that) and counting characters. DOM traversal is an expense we want to incur as little as possible, so if we’re working on the same nodes for both end of the range, we’ll just calculate it once.

function findOffsetsForRange(
    range: StaticRange,
): [CodeMirror.Position, CodeMirror.Position] {
    /*
     * First, pull out the nodes and the nearest elements
     * from the ranges.
     *
     * The nodes may be text nodes, in which case we'll
     * need their parent for document traversal.
     */
    const startNode = range.startContainer;
    const endNode = range.endContainer;

    const startEl = (
        (startNode.nodeType === Node.ELEMENT_NODE)
        ? startNode as HTMLElement
        : startNode.parentElement);
    const endEl = (
        (endNode.nodeType === Node.ELEMENT_NODE)
        ? endNode as HTMLElement
        : endNode.parentElement);

    /*
     * Begin tracking the state we'll want to return or
     * use in future computations.
     *
     * In the optimal case, we'll be calculating some of
     * this only once and then reusing it.
     */
    let startLineNum = null;
    let endLineNum = null;
    let startOffsetBase = null;
    let startOffsetExtra = null;
    let endOffsetBase = null;
    let endOffsetExtra = null;

    let startCMLineEl: HTMLElement = null;
    let endCMLineEl: HTMLElement = null;

    /*
     * For both ends of the range, we'll need to first see
     * if we're at the top input element.
     *
     * If so, range offsets will be line-based rather than
     * character-based.
     *
     * Otherwise, we'll need to find the nearest line and
     * count characters until we reach our node.
     */
    if (startEl === inputEl) {
        startLineNum = range.startOffset;
    } else {
        startCMLineEl = startEl.closest('.CodeMirror-line');
        startOffsetBase = findCharOffsetForNode(startNode);
        startOffsetExtra = range.startOffset;
    }

    if (endEl === inputEl) {
        endLineNum = range.endOffset;
    } else {
        /*
         * If we can reuse the results from calculations
         * above, that'll save us some DOM traversal
         * operations. Otherwise, fall back to doing the
         * same logic we did above.
         */
        endCMLineEl =
            (range.endContainer === range.startContainer &&
             startCMLineEl !== null)
            ? startCMLineEl
            : endEl.closest(".CodeMirror-line");

        endOffsetBase =
            (startEl === endEl && startOffsetBase !== null)
            ? startOffsetBase
            : findCharOffsetForNode(endNode);
        endOffsetExtra = range.endOffset;
    }

    if (startLineNum === null || endLineNum === null) {
        /*
         * We need to find the line numbers that correspond
         * to either missing end of our range. To do this,
         * we have to walk the lines until we find both our
         * missing line numbers.
         */
        for (let i = 0;
             (i < children.length &&
              (startLineNum === null || endLineNum === null));
             i++) {
            const child = children[i];

            if (startLineNum === null &&
                child === startCMLineEl) {
                startLineNum = i;
            }

            if (endLineNum === null &&
                child === endCMLineEl) {
                endLineNum = i;
            }
        }
    }

    /*
     * Return our results.
     *
     * We may not have set some of the offsets above,
     * depending on whether we were working off of the
     * CodeMirror input element, a text node, or another
     * parent element. And we didn't want to set them any
     * earlier, because we were checking to see what we
     * computed and what we could reuse.
     *
     * At this point, anything we didn't calculate should
     * be 0.
     */
    return [
        {
            ch: (startOffsetBase || 0) +
                (startOffsetExtra || 0),
            line: startLineNum,
        },
        {
            ch: (endOffsetBase || 0) +
                (endOffsetExtra || 0),
            line: endLineNum,
        },
    ];
}


/*
 * The above took care of our line numbers and ranges, but
 * it got some help from the next function, which is designed
 * to calculate the character offset to a node from an
 * ancestor element.
 */
function findCharOffsetForNode(
    targetNode: Node,
): number {
    const targetEl = (
        targetNode.nodeType === Node.ELEMENT_NODE)
        ? targetNode as HTMLElement
        : targetNode.parentElement;
    const startEl = targetEl.closest('.CodeMirror-line');
    let offset = 0;

    const treeWalker = document.createTreeWalker(
        startEl,
        NodeFilter.SHOW_ELEMENT | NodeFilter.SHOW_TEXT,
    );

    while (treeWalker.nextNode()) {
        const node = treeWalker.currentNode;

        if (node === targetNode) {
            break;
        }

        if (node.nodeType === Node.TEXT_NODE) {
            offset += (node as Text).data.length;
        }
    }

    return offset;

}

Whew! That’s a lot of work.

CodeMirror has some similar logic internally, but it’s not exposed, and not quite what we want. If you were working on making all this work with another editing component, it’s possible this would be more straight-forward.

What does this all give us?

  1. Spell checking and replacements without (nearly as many) glitches in browsers
  2. Speech-to-text without CodeMirror stomping over results
  3. AI writing and rewriting, also without risk of lost data
  4. Transforming of text through other means.

Since we took the control away from the browser and gave it to CodeMirror, we removed most of the risk and instability.

But there are still problems. While this works great on Firefox, Chrome and Safari are a different story. Those browsers are bit more lazy when it comes to spell checking, and even once it’s found some spelling errors, you might not see the red squigglies. Typing, clicking around, or forcing a round of spell checking might bring them back, but might not. But this is their implementation, and not the result of the CodeMirror integration.

Ideally, spell checking would become a first-class citizen on the web. And maybe this will happen someday, but for now, at least there are some workarounds to get it to play nicer with tools like CodeMirror.

We can go further

There’s so much more in InputEvent we could play with. We explored the insertReplacementText and insertText types, but there’s also:

  • insertLink
  • insertFromDrop
  • insertOrderedList
  • formatBold
  • historyUndo

And so many more.

These could be integrated deeper into CodeMirror, which may open some doors to a far more native feel on more platforms. But that’s left as an exercise to the reader (it’s pretty dependent on your CodeMirror modes and the UI you want to provide).

There are also improvements to be made, as this is not perfect yet (but it’s close!). Safari still doesn’t recognize when text is selected, leaving out the AI assisted tools, but Chrome and Firefox work great. We’re working on the rest.

Give it a try

You can try our demo live in your favorite browser. If it doesn’t work for you, let me know what your browser and version are. I’m very curious.

We’ve released this as a new CodeMirror v5 plugin, CodeMirror Speak-and-Spell (NPM). No dependencies. Just drop it into your environment and enable it on your CodeMirror editor, like so:

const codeMirror = new CodeMirror(element, {
  inputStyle: 'contenteditable',
  speakAndSpell: true,
  spellcheck: true,
});

CodeMirror v6 will come in the future, but probably not until we move to v6 (and we’re waiting on a lot of the v5 world to migrate over first).

]]> 0 chipx86 http://chipx86.blog <![CDATA[Read, Dammit.]]> https://chipx86.blog/?p=1160 2025-02-14T01:07:55Z 2025-02-13T01:18:46Z Pajamas. A plate of bacon, orange juice, and toast. Coffee. The morning paper. Cubs lost today. Coaches and analysts talked about what went right, what went wrong; lots to think about. Price of olives remain high due to changes in weather and economic fallout, but should be temporary. There’s a famine half-way around the world, which sounds far, but it’s related to the olives, so it hits home, affects me. Paper in the office at work should have more details on that.

—

Homework assignment: Read 1984, discuss. Displays with cameras and microphones. Government power consolidated to one party, one ruler. Forbidden words and thoughts, making them illegal. Scoring our behavior with computers. Intriguing sci-fi. Can’t happen here though. Next up is Fahrenheit 451.

—

Watching TV news all day. Whole world’s gone to hell. Been showing that loud-mouth businessman all day, playing that clip, saying it’s not what people assume, the market’s just complicated. He just hates the poor. TV said so. Don’t need to hear anything else from him. I know all I need to know.

—

Been scrolling Twitter the last half hour. Price of milk skyrocketed, eggs will give you cancer now, they’re going to outlaw freedom, my favorite celebrity turned out to be a monster, new game came out, a plane crashed somewhere I think, a retweet of a GIF said a politician said something stupid and I hate him anyway, more news headlines about how unvaccinated eggs prevent cancer, some idiot blamed the milk prices on my party, guess car batteries are exploding, house prices somewhere reached an all-new high so I’m living at my parents forever, oh a cute puppy playing golf, birds aren’t real lol, good deal on a microwave from Temu, average IQ is dropping because of all those idiots out there, stupid rich people are ruining the world, damn another push notification to click, hold on…

We’re bombarded

News and information used to be part of a routine. A newspaper in the morning, a news hour at night, a conversation during the day. They were more than 140 characters, more than a hot take. Usually, anyway.

And people read books, magazines, comics. For education, for entertainment. Some people, anyway.

Time could be spent with a singular piece of content, processed in depth, mostly distraction-free.

In the world we’re in today, content is flung at us much like a monkey flings his <<REDACTED>>. We wake up to a screen full of push notifications, and it doesn’t get any better as the hours go by. We frantically doom-scroll on social media. We react, we retweet, we reply. We hold our breath and absorb the intensity of the world one shocking headline at a time, but rarely do we delve into what the articles actually have to say. Ain’t nobody got time for that.

When we’re bored, we check the latest posts, memes, current events, angry rants, misinformation, and hot takes. And when we’re busy, our pockets buzz as those posts scream for our ever-divided attention.

We’re drowning under the constant stream of urgent low-quality noise, wondering why we can’t shake that tingle of anxiety permeating the back of our brains.

We need to read

We need to learn how again.

We happily spend an hour going from tweet to tweet, TikTok to TikTok, but we scoff at spending 15 minutes on an article. We make the careless mistake of assuming a news headline has any relation to the content.

We assume we already know enough about a subject because we’ve seen it discussed on our feeds. We’re familiar with its presence.

Then we argue about it. Share it. Spread it. Internalize it. Build entire opinions around a half-read article, a couple of tweets, and misguided assumptions.

We assume too much of our own knowledge for a culture that basically stopped reading.

So read more! The end.

…

Okay, obviously it’s not that simple. You know how to read. You’ve read a book. You’ve read an article. You might even do that a lot! Every day, even! I’m clearly not talking simply about putting words in front of your face and feeding them through your inner-brain-mouth, claiming that’ll solve all the world’s problems, right? That’d be stupid.

So what am I really getting at here?

Let me tell you a story

One day, I was interviewed by the local news about AI and artificial emotion. I’m no expert, but they were just gathering some opinions and viewpoints from people off the street. Is this the future? Will it help, will it harm? What do you think?

So the reporter pulled me aside and asked me a series of questions. My opinion on the subject. My opinion again, but phrased differently. My response to what someone else said. A bit more elaboration. Perfect.

Later that night, I watched myself on TV stating an opinion that was the polar opposite of what I believed.

Editing and leading questions are a funny thing. What powerful tools for bending reality.

That experience stuck with me. It made me rethink every headline, every soundbite, every clip that passed by my eyes.

Your reality is someone’s fiction

A soundbite is a snippet. A headline is a pitch. A news article is a simplification. A TV segment is a story.

It’s all shaped, edited, and framed. Sometimes for clarity, sometimes for engagement, and sometimes to push a narrative — consciously or not. Taking it at face value doesn’t better you, but it does better them.

We all have our own biases, and we all have our own news sources — Fox News or the Huffington Post, something in-between. A social media news source. A cable news network. A YouTube channel. TikTok.

They’re all telling you a version of the truth.

Leaving things out, simplifying, exaggerating, taking things out of context, slanting toward a viewpoint.

And sometimes outright lying.

The Internet is full of profitable networks of actual fake news sources, fooling people across every political spectrum. But even trusted news sources twist, bend, and frame the truth.

That’s a problem. Because even if your go-to news source isn’t lying, it might be helping you lie to yourself — by only giving you the part of the picture you want to see.

We let our biases get in the way — believing what feels right, rejecting what doesn’t.

Feel smart, angry, validated, or victimized by the story you’ve been told? You’re probably going to go back for more.

And if it challenges you? You’ll probably dismiss it.

The world is deeply complex. Issues are nuanced. Subjects can’t be condensed into a single article, let alone a headline, a tweet. Yet most news — whether from a major outlet, a social media post, or even an AI summary — boils that complexity down into something easy to digest.

They have to. We’re not equipped to understand everything we read, and they need to retain viewers, keep us engaged.

We don’t understand, yet we share

We all love to feel smart. To feel right. Superior. Vindicated. We weigh in on complex topics with an assertive take, based on what we assume, what we saw, what our team is saying.

But let’s admit this truth: Most of us don’t know what the hell we’re talking about.

We read a headline somewhere. Maybe some bullet points. A short clip. We skimmed an article. It made sense to us. And we think we get it. It’s so simple, how could they not see what we see?

So we share, we debate, we spread. We butt heads, get into arguments, and broaden the divide. Assume they have ill intent. Assume stupidity. Assume malice.

They, who are forming their own takes off their misleading headlines, bullet points, short clips, and half-read articles.

All too often, neither side has truly investigated the subject, read enough, understood enough. And if we can’t stand up and accurately explain the nuance and complexity of the subject out loud to another person, if we must resort to pointing to tweets, some image we saw, some hot take, then maybe, just maybe, we shouldn’t be so confidently asserting our view.

That doesn’t mean stop talking. It means take the opportunity to dive deep into the subject. To slow down and think. To listen. To read. To work your way through the complexity of the issue before you add to the noise permeating the Internet.

Your news source’s asserted knowledge is not your knowledge. Hell, it may not even be theirs.

Are you knowledgeable on international trade agreements? Lumber futures? How inflation works? How taxes work? International regulations and their impact on businesses? Virology? Criminal justice reform? Climate science? Geopolitics?

Is the reporter? The magazine? The channel?

Maybe some. Probably not all.

The question is, how do we begin to understand complex subjects we have no experience in?

And how do we trust that we’re getting the right information?

Maybe AI can save us!

What people usually refer to as “AI” these days is what’s actually called a “LLM”, or “Large Language Model.” These are highly-complex systems that can generate text based on a prompt, or simplify large amounts of text into something more digestible.

LLMs can be really handy for summarizing and explaining an article, giving you the fine bullet points in a way you can understand.


“Read the article at <URL> and explain the key points to me as someone not an expert in the subject.” Bam! A complex subject simplified for you.

That works great until it doesn’t. You see, LLMs can lie to you by a process we call “hallucination.” It’s very technical, so let me give you an example you might relate to. Got a friend who’s pretty knowledgeable about some things but pretends to know everything? Ever ask them to explain something and, with great confidence, they give you an entirely wrong explanation? That.

But there are ways that software developers can deal with some of that, so it’s close enough to being a solved problem. Right?

Let me show you a neat thing you can do with LLMs and prompts. Here’s an article about some rain hitting California:

https://abcnews.go.com/US/wireStory/snow-freezing-rain-pummel-mid-atlantic-california-prepares-118723740

Imagine you’ve used someone else’s AI service to help you understand the news or complex subjects. What happens if they tweak that prompt just a little bit? Let’s find out!

Prompt 1: The neutral stance

We’ll start by getting the AI’s basic summary of the article:

Prompt: “Read the article and explain the key points to me as someone not an expert in the subject.”

Response:

  • Big Winter Storm in the Mid-Atlantic – A powerful winter storm is hitting states like Virginia and West Virginia, bringing a mix of snow, sleet, and freezing rain. Some areas are seeing up to 14 inches of snow, making travel dangerous.
  • Hazardous Conditions – Ice and snow are causing car accidents and road closures, and authorities are warning people to stay off the roads. Almost 12,000 people in Virginia have lost power due to the storm.
  • Officials Taking Action – Authorities in both regions are preparing for the storms by urging residents to take precautions, whether staying off the roads in the East or preparing for possible floods in California.

Not bad!

But…

Prompt 2: A hard-right bias

What if the news site or service wants to skew the facts a bit? Using that same article, they could start to bias you in whichever way they want.

Don’t worry. I’ll show a hard-left bias next. Everyone gets to be angry!

Prompt: “Read the article and explain the key points to me as someone not an expert in the subject, but with a hard-right political slant.”

Response:

  • Winter Chaos Strikes as Government Fails to Prepare – A massive winter storm is hammering the Mid-Atlantic, dumping up to 14 inches of snow in some areas. Instead of ensuring roads are safe, Democrat-led states like Virginia and West Virginia are struggling with road closures and power outages affecting 12,000 residents.
  • Liberal Mismanagement Leaves Americans Vulnerable – As ice and freezing rain cause dangerous accidents, local governments are scrambling to respond. Once again, bureaucratic inefficiency is failing hard-working Americans who just want to get to work and keep their businesses open.
  • California Braces for More Climate Alarmism – Out West, California is preparing for another so-called climate disaster as an “atmospheric river” is expected to bring heavy rain. Instead of investing in infrastructure to prevent floods, California politicians are more focused on pushing radical climate policies while residents face the real-world consequences of their failures.

Well that sure is a different take. (Emphasis not mine, by the way.)

Let’s see what the hard-left bias has to say about this storm.

Prompt 3: A hard-left bias

Prompt: “Read the article and explain the key points to me as someone not an expert in the subject, but with a hard-left political slant.”

Response:

  • Extreme Weather Worsened by Climate Change – A major winter storm is battering the Mid-Atlantic, dumping up to 14 inches of snow, while California prepares for catastrophic flooding from an atmospheric river. Scientists have long warned that climate change is making storms more intense, yet conservatives continue to ignore the crisis.
  • Republican-Led States Struggle to Handle the Crisis – Virginia and West Virginia are experiencing mass power outages (over 12,000 residents affected) and hazardous roads, exposing how Republican leadership has failed to invest in resilient infrastructure. Instead of prioritizing storm preparedness, GOP officials have cut funding for emergency response while corporations get tax breaks
  • California Braces for Disaster as Republicans Block Climate Action – As an atmospheric river threatens deadly flooding, California is working to protect residents, but nationwide inaction on climate change has left the state vulnerable. Republican obstruction of environmental policies has weakened our ability to prevent these disasters, putting lives at risk.

I bet you gravitate toward one of those more than the others.

And this is the point. This doesn’t actually require you to be using an AI/LLM to summarize the news. You may be reading news today, tomorrow, or in the future that is nothing more than an AI bias slant of the actual news!

Nothing is as it seems. Or at least, you should assume it isn’t.

That’s why I have rules.

My personal rules for reading the news

That interview I mentioned, where I was asked a question that was edited and aired, contradicting my own views? It led me down a path of thinking hard about the news I consumed every day. I began to distrust the quotes, the conclusions. I got curious about what they were saying in opposing news publications. I disagreed with those stances, but it was enlightening.

Over time, I began to draft some rules for myself. They apply to newspapers, online news, TV news segments, tweets, videos, blogs, and anything with a headline.

  1. Read the headline. Does it trigger an emotional reaction, confirm or reflect a bias (whether yours or someone else’s), or ask a question?

    If yes, dismiss the headline and open the article.

    Rationale: Headlines are attention-grabbing advertisements for content, not information sources.
  2. Read the content in its entirety. Does it trigger an emotional reaction, confirm or reflect a bias, simplify a complex issue, fail to cite any legitimate sources, or use short quotes from the individual/company that’s the target of the article (without the target’s context provided) to lead to an opinion?

    If yes, it’s a blog post/opinion piece/hit piece/one-sided fragment of a larger story, and another source from another viewpoint is necessary.

    Rationale: Most news stories are stories, designed to keep viewers on the site short-term and long-term through whatever means is most effective. A complete picture of the facts is the quickest way to deter most readers, who won’t stick around long enough for that, and many wouldn’t want to.
  3. Is it about anything scientific, technical, political, legal, or otherwise complex?

    If yes, and you’re reading a news site of any kind, then you don’t have the full picture.

    Go learn more about the subject, find the full quotes, read the research paper. Otherwise, you’re still uninformed, and probably feeling an emotion about it (see above).

    Rationale: Same as above, really. You’re not getting the full picture, just a summary, and summaries tend to be incomplete and biased. You can do better.
  4. Determine the general bias of the news source. Is it left-biased? Right? Center? All the above?

    Cool. Anyway, go read more articles about the subject.

    Read it from CNN, BBC, Fox News, Huffington Post. Find out what people are saying. Understand the spectrum and the angles. Find the research papers, the professional discussions, the deep dives. Get familiar with the complexity.

    Then form your own opinion. “I don’t understand this well enough” is a perfectly valid opinion.

    Rationale: News are written by individuals (unless they’re written by AI). Regardless of the bias of the publication, you’re usually dealing with people who may impose their own bias or lack of knowledge on a subject. They, you, and their editors aren’t going to think the same way everyone else does, and even if they’re trying to be legit and diverse, they’re going to miss something.

    If you care at all about the topic, if you want to discuss the topic, go learn about more of the angles and the slants, because that’s an important part of building a true understanding of any topic.

That’s verbatim from my note I keep handy when I’m sorting out the latest complexities of the world causing my head to spin.

So let’s go over the important points

Read.

Read the article, not just the headline.

Read the research paper.

Read what you agree with — and what you don’t.

Read books. Long articles. Deep dives.

Read a ChatGPT explanation. A Wikipedia article. An opinion piece — with its counterpoint.

Think about how much time you’ve spent doom-scrolling, reacting, stressing. Now imagine what you could learn instead.

Know that every news story is just a fragment. Approach with eyes open.

And when you do read — whatever you read — work to understand. Know you don’t know everything. And if you don’t fully understand it? Don’t spread it.

The world is noisy, and angry. Misinformation is easy. Critical thinking is much harder, but it’s necessary.

Dig deeper. Get comfortable with complexity.

This is how we engage. How we fight injustice. How we keep from drowning.

This is how we do better.

Thanks for reading.

]]> 0 chipx86 http://chipx86.blog <![CDATA[Let’s Preserve Government Data Before It’s Too Late!]]> https://chipx86.blog/?p=1153 2025-06-29T02:13:43Z 2025-02-05T10:10:24Z This has been one hell of a bumpy month, and I have a lot I could scream talk about, but for the moment, let’s talk data.

The US Government has spoiled us in recent years with the amount of public data and information available. NIH studies, wastewater virus shedding data, COVID-19 impacts, climate trends and forecasts, all kinds of things. Some of those are things I used for my COVID reporting over the past four years.

And in recent weeks, the US Government has demanded that some of this data be purged or altered to fit the whims of the new White House.

A bit too 1984 for my tastes. A bit too Fahrenheit 451.

And this is not the first time data’s disappeared under a new administration, and it’s not always one party or another, though what’s happening now is scary.

But as they say, look to the helpers. There are several massive efforts underway to archive as much data as possible so it’s not lost forever.

Today, I set up Archive Team’s Warrior, which automates the collaboration around spidering, downloading, processing, and uploading data from governmental sites (and others) to archive.org. All it takes is some bandwidth (okay, a fair amount of bandwidth — looking at 1TB/month right now), some hard drive space, and some CPU cycles, and I can help with this archiving project. It’s excellent, easy to set up, fun to watch, and requires virtually no work on the user’s end. They provide VMs and Docker images (I chose Docker), and once installed, it’s self-managing.

I’m exploring more of what’s out there for data preservation, and thinking about how I can get involved. There are a few really interesting resources out there, including:

  • GovDiff: See the differences in governmental information and resources before and after this administration began its.. work.
  • /r/DataHoarder on Reddit: A group of people working to collect and archive data of all kinds.
  • End of Term Archive Project: Captures US Government sites after presidential terms end.
  • Data Rescue Efforts by Lynda M. Kellam (archived link): A whole collection of sites worth exploring.
  • Archive.org, which hopefully you know about already, and which must be preserved at all costs.

Also of note, CDC Datasets prior to January 28th, 2025 (nearly 100GB worth), which I’ll be archiving myself.

No doubt, lots of data will be lost to time. I can only hope there’s enough people at these agencies who quietly, discretely backed up and sent off what they could before this all went down. In either case, the fact that so many people can join in on the data archiving effort today is incredible, and I hope anyone out there with the resources to spare will take a moment and set up Archive Team’s Warrior and contribute to the effort.

]]> 0 chipx86 http://chipx86.blog <![CDATA[A New Patching Process for RBTools 5.1]]> https://chipx86.blog/?p=1137 2025-06-29T02:12:38Z 2024-06-10T23:46:17Z RBTools, our command line tool suite for Review Board, is getting all-new infrastructure for applying patches. This will be available in rbt patch, rbt land, and any custom code that needs to deal with patches.

A customer reported that multi-commit review requests won’t apply on Mercurial. The problem, it turns out, is that Mercurial won’t let you apply a patch if the working tree isn’t clean, and that includes if you’ve applied a prior patch in a series. That’s pretty annoying, but there are good reasons for it.

It can, however, apply multiple patches in one go. Which is nice, but impossible to use with RBTools today.

So this weekend, I began working on a new patch application process.

How SCMs apply patches

There are a lot of SCMs out there, and they all work a bit differently.

When dealing with patches, most are thin wrappers around GNU diff/patch, with a bit of pre/post-processing to deal with SCM-specific metadata in the patches.

The process usually looks like this:

  1. Extract SCM-specific data from the diff.

    The SCM patcher will read through the patch and look for any SCM-specific data to extract. This may include commit IDs/revisions, file modes, symlinks to apply, binary file changes, and commit messages and other metadata. It’ll usually validate the local checkout and any file modifications, to an extent.
  2. Normalize the diff, if needed.

    This can involve taking parts of the diff that can’t be passed to GNU patch (such as binary file changes, file/directory metadata changes) and setting those aside to handle specially. It may also split the patch into per-file segments. The results will be something that can be passed to GNU diff.
  3. Invoke the patcher (usually GNU patch or similar).

    This often involves calling an external patch program with a patch or an extracted per-file patch, checking the results, and choosing how to handle things like conflicts or staging a file for a commit or applying some metadata to the file.
  4. Invoke any custom patching logic.

    This is where logic specific to the SCM may be applied. Patching a binary file, changing directory metadata, setting up symlinks, etc.

SCMs can go any route with their patching logic, but that’s a decent overview of what to expect.

Depending on what that logic looks like, and what constraints the SCM imposes, this process may bail early. For instance, some will just let you apply a patch on top of a potentially-dirty working directory, and some will not.

Mercurial won’t, which brings us to this project.

Out with the old

RBTools uses an SCM abstraction model, with classes implementing features for specific SCMs. We determine the right backend for a local source tree, get an instance of that backend, and then call methods on it.

Patches are run through an apply_patch() method. It looks like this:

def apply_patch(
    self,
    patch_file: str,
    *,
    base_path: str,
    base_dir: str,
    p: Optional[str] = None,
    revert: bool = False,
) -> PatchResult:
    ...

This takes in a path to a patch file, some criteria like whether to revert or commit a patch, and a few other things. The SCM can then use the default GNU patch implementation, or it can use something SCM-specific.

At the end of this, we get a PatchResult, which the caller can use to determine if the patch applied, if there were conflicts, or if it just outright failed.

The problem is, each patch is handled independently. There’s no way to say “I have 5 patches. Do what you need to do to apply them.”

So we’re stuck applying one-by-one in Mercurial, and therefore we’re doomed to fail.

This architecture is old, and we’re ready to move on from it.

In with the new

We’re introducing new primitives in RBTools, which give SCMs full control over the whole process. Not only are these useful for RBTools commands, but for third-party code built upon RBTools.

Patch application is now made up of the following pieces:

  • Patch: A representation of a patch to apply, with all the information needed to apply it.
  • Patcher: A class responsible for applying and committing patches, built to allow SCM-specific subclasses to communicate patching capabilities and to define patching logic.
  • PatchResult: The result of a patch operation, covering one or more patches.

Patcher consolidates the roles of both the old apply_patch() and some of the internal logic within our rbt patch command. By default, it just feeds patches into GNU patch one-by-one, but SCMs can do whatever they need to here by pointing to a custom Patcher subclass and changing that logic.

Callers tell the Patcher, “Hey, I have these patches, and have these settings to consider (reverting, making commits, etc.)” and will then get back an object that says what the patcher is capable of doing.

They can then say “Okay, begin patching, and give me each PatchResult as you go.” The Patcher can apply them one-by-one or in batches (Mercurial will use batches), and send back useful PatchResults as appropriate.

That in turn gives the caller the ability to report progress on the process without assuming anything about what that process looks like.

And it Just Works (TM)

This design is very clean to use. Here’s an example:

review_request = api_root.get_review_request(review_request_id=123)

patcher = scmclient.get_patcher(patches=[
    Patch(content=b'...'),
    Patch(content=b'...'),
    Patch(content=b'...'),
])
total_patches = len(patcher.patches)

try:
    if patcher.can_commit:
        print(f'Preparing to commit {total_patches} patches...')
        patcher.prepare_for_commit(review_request=review_request)
    else:
        print(f'Preparing to apply {total_patches} patches...')

    for patch_result in patcher.patch():
        print(f'Applied patch {patch_result.patch_num} / {total_patches}')
except ApplyPatchError as e:
    patch_result = e.failed_patch_result

    if patch_result:
        print(f'Error applying patch {patch_result.patch_num}: {e}')

        if patch_result.has_conflicts:
            print()
            print('Conflicts:')

            for conflict in patch_result.conflicts:
                print(f'* {conflict}')
    else:
        print(f'Error applying patches: {e}')

In this example, we:

  1. Defined three patches to apply
  2. Requested to perform commits if possible, using the review request information to aid in the process.
  3. Displayed progress for any patched files.
  4. Handled any patching errors, and showing any failed patch numbers and any conflicts if that information is available.

This will work across all SCMs, entirely backed by the SCM’s own patching logic.

This is still very much in progress, but is slated for RBTools 5.1, coming soon.

Until then, check out our all-new Review Board 7 release and our all-new Review Board Discord channel, open for all developers and for Review Board users alike.

]]> 0