This is naturally a more technical piece than most that I write, given the nature of what I am describing. I’ll do my best to smooth out those rough edges, but know that I am aware this isn’t for my usual audience.

Lazy, lazy, lazy. And not the right intention. Fortunately, I caught onto what I was doing when only a thousand words or so had been set down. Pauciloquy² is called for and there is still (barely) enough to recover. And so here we are, about to talk about ark‘s store, or, where’d all those files go? Where to begin…?

The Dreaded Org Chart

I’ve been unsatisfied with the hierarchical structure of file systems since I ran that very first catalog command on an Apple ][ e. I’ve lived with them ever since, an accumulating succession of decades that have done little to quench the burning dissatisfaction with the way files are stored on computers. It sometimes seems like a large part of my avocation in technology has been a desperate search for a way out of the rigid hierarchy³. “Tear down the wall!”

The search took me to Evernote with its notebooks and tags, and then to Obsidian with its org chart and tags. But Obsidian introduced, to me at least, the notion of a graph: that is links between files that form an edge between nodes. It is a powerful idea, more powerful than I realized. And so when I approached this hobby project and was considering the design, I had two strong ideas in mind:

1. Get me off this org chart!
2. How might I take advantage of graphs?

Simple Requirements

In my limited imagination, there are two poles on the file storage spectrum⁴: a simple listing of files and a graph of files where every file points to every other file. ark is designed to be as close to the simple listing of files as possible. My requirements were, therefore, simple:

1. There should only ever be one of each item. Preventing duplicates makes things easier to find. I can’t tell you how many times I have found three different copies of a Word document or photo on my computer.
2. Items in the archive must be described separately from the files themselves. File systems provide the bare minimum capacity for describing a file. An archive is more than a file system so I need a way of describing those files to make finding them as easy as possible.
3. The archive only stores finished products. Working documents, working files don’t get into the archive until they are finished.

With these requirements in hand, I set about meeting each one. To ensure that each item in the archive is unique, it gets a unique file name based on its digital DNA. The unique set of bytes that make up a file can be “hashed” into a number that is unique for that set of bytes. ark uses sha256 for its hashing mechanism. That number becomes not only the name of the file, but its identifier in ark‘s database. What it means in practice is that if I bring an exact copy of a file into ark that already exists, it doesn’t get added a second time, it is simply ignored in favor of the copy that is already in the system.

To describe the files in the system, ark uses a SQLite database. This allows ark to have full-text search and semantic search capabilities without running a database server. The SQLite database is just another file on my computer. True, SQLite is not designed to be a multi-user database, but ark is not designed to be a multi-user application, so we’re all good here. All of the meta-data needed to describe a file is stored in the database. That meta-data breaks down into five categories:

1. Classification (doc type, series, sub-series, format, sensitivity, priority, etc.)
2. Provenance (date authored, record origin, original source path, physical location, etc.)
3. Identification (doc ID, sha256, title, store path)
4. Quality (OCR status, OCR quality)
5. Content (full-text search content, LLM summary, embeddings)

Finally, ark uses a “copy-in” strategy for files. That means that the source file is left untouched, wherever it comes from, and a copy of the file is brought into the archive. Because of this, and the other requirements I listed above, I can store all of the files in a flat structure within ark‘s store. After all, I never need to know the actual file name. I just need to be able to describe what I am looking for and the database takes care of the rest.

Connections

In your standard OS, files sit there on a file system completely unconnected. But in a personal archive, people are first-class citizens. So in addition to the database storing information about files, it stores information about the people in those files. And since it is the connections that make an archive like this come alive, ark supports three kinds of connections:

1. Document-to-document: one email is linked to another as a “reply-to”. A receipt is linked to an invoice. Documents can link to one another.
2. Document-to-person: Who wrote it, received it, who’s mentioned, who’s the subject. The same person may show up in lots of roles across documents. This is the single most-queried table in the archive. Anything person-aware reads from here.
3. Person-to-person: Friends, family, colleagues, including date ranges. A handyman who works on your house may retire, and someone else takes over. The connections capture it all.

Here’s where things get fun! In an org chart, the answer to “show me everything about my son from the last two years” depends on remembering which folder you filed each thing in. In a graph, it doesn’t. I asked Claude Desktop⁵ that exact question and got back this:

Theater & Performance — The dominant thread. He’d been active in community theater for a while: directing plays, performing in musicals (the creepy Wolf in Into the Woods), and preparing for the role of Edwin Drood at his high school. He’s a baritone/tenor in the choral group and was being recruited for an a cappella group. By May 2026, a choral concert I attended. Then the big news: he and a friend auditioned for Guys and Dolls — a six-hour audition — and he was offered the lead role of Nathan Detroit. Around 200 lines.

Prom — May 19, 2026. He went with a friend. Dinner first, then photos at the Jefferson Memorial via Uber Black, then the prom itself. I left the car at the venue so he could drive home. He got home around 10pm, changed, headed to a post-prom party, was home just after 1am. I noted he “seemed very grown up about the whole thing.”

Memorial Day 2026 — At our friends’ house. He “christened the pool as he does every year.”

College Prep — From a family Zoom in late May: he’s working on the Common Application and asking for recommendation letters. Summer plans still up in the air — SAT prep, camp counselor work, or a possible Europe trip.

Lawn — An October 2025 iMessage asks whether he can cut the lawn that weekend.

Rutgers campus tour — Mentioned in passing in the Zoom notes: I drove by my childhood New Jersey house “after a Rutgers campus tour with him.”

Cotillion — Per a January 2026 letter, he was working as an instructor — teaching students to dance and how to conduct themselves in society.

Driving — Also from that January letter, the dry observation that his having a license “does wonders for the nervous system.”

One question; every connected thing — pulled together by edges the archive had been quietly maintaining while I wasn’t looking. Letters, iMessages, Zoom call notes, photos, calendar entries, diaries, social media — formats I don’t normally think of as connected, returning one coherent picture of my kid over a two-year span⁶. The org chart could have stored these. Only the graph could have answered them. (More on how that LLM call works under the hood — bundles, edges, redaction — next post.)

Two Ways to Organize

In looking at how archivists tend to organize archives, a 4-tiered, um, hierarchy (sorry!) emerged as a trend:

1. Series (biographical, correspondence, writings, research, professional, financial, legal, medical, etc.).
2. Sub-series (fixed categories that fall underneath each of the series).
3. File. A collection of items in a series/sub-series in a physical archive.
4. Item. The thing itself.

In ark we have hard mappings to three of the four: series, sub-series, and the item itself.

Series and sub-series are categories that form a controlled vocabulary. But I find it useful to have user-curated groupings as well. While ark can use tags, I created something called a “collection” which is a curated grouping named after the reason that the items are grouped together. For example #2026-tax-documents, or #2019-house-purchase, or #vacation-in-the-golden-age-notes. Documents can, of course, have a series and sub-series, be tagged, and be members of one or more collections. The nice thing about collections is that they can be used as input for other ark commands. For instance:

ark bundle '#vacation-in-the-golden-age-notes' | ark task summarize

which will create a bundle of all of the items in the #vacation-in-the-golden-age-notes collection and then use an LLM to summarize the entire bundle.

Bottom line: a collection is a list of items in the archive with context.

Some Things Aren’t Documents

Most things in ark are documents: emails, PDFs, photos, diary entries, Office documents, text files. But several things in the archive aren’t documents; instead, they have their own dedicated database tables. These include reading events (to manage my reading list), health data from Apple Health and FitBit, location data pulled off photos and extracted from other sources like diary entries, and more.

I’ll write about each of these later on in this series. For the store, it is useful to know that these live alongside the document model and follow the same rules: they are addressable with a unique sha256 identifier, auditable, and integrated with ark‘s core command set (although they sometimes have commands of their own).

Two Things That Touch Everything

Two things in ark‘s data model don’t sit in any one layer; they sit over all of them:

1. Sensitivity: This is set on every item that comes into the archive. The archive treats sensitivity as a query filter, not a display hint. Items that are sensitive are automatically routed through different code paths. For example, if I use Claude Desktop to ask a question and the result includes sensitive data, the code path prevents “restricted” data from leaving the local machine so that it never gets to Claude Desktop. If the data is marked “sensitive”, any sensitive information is stripped and replaced with “[REDACTED]” before being sent off the local machine. This is true everywhere data might be exfiltrated off the local machine.
2. Annotations: I have a lot to say about things (pauciloquy goes only so far). I engineered the annotations layer to sit atop everything in the archive. This way, I can add notes and comments to a document, a book record, a watch event, an Apple Health record, a person — anything in the archive can be annotated. Those annotations are searchable, and they are surfaced most commonly when looking at a document in the archive. This allows me to add context without touching the original item.

Conclusion

This design keeps ark entirely self-contained on my local machine. The sensitivity layer ensures that documents that shouldn’t leave the machine don’t. SQLite handles full-text and semantic search. And so far, this scales well. As of this writing, my store is 125 GB not counting the SQLite database which adds another 9 GB:

=== ark store stats ===
  Strategy:       copy
  Store path:     /Users/[username]/.local/share/ark/store
  Store size:     125.0 GB  (390,749 files on disk)
  In store (DB):  365,774 document(s)
  Index-only:     330,680 document(s)  (no managed copy)
  Total docs:     696,454
  Type                    Total   In store   Index-only
  ────────────────────  ───────  ─────────  ───────────
  email                 281,740    272,068        9,672
  browser_visit         106,616          0      106,616
  image                  85,415     77,182        8,233
  imessage               41,267          0       41,267
  tweet                  27,599          0       27,599
  watch_event            26,149          0       26,149
  calendar_event         21,564          0       21,564
  facebook-post          19,234          0       19,234
  cli_command            15,189          0       15,189
  note                   12,914          7       12,907
  health_day              9,411          0        9,411
  pdf                     8,400      8,400            0
  blog_comment            8,040          0        8,040
  blog_post               7,477          0        7,477
  purchase                5,813          0        5,813
  music_play              5,194          0        5,194
  attachment              4,302      4,302            0
  git_commit              2,261          0        2,261
  office                  2,078      2,078            0
  reading_finished        1,547          0        1,547
  book                    1,257          0        1,257
  text                    1,114      1,114            0
  diary_entry               804        612          192
  reminder                  373          0          373
  action_item               209          0          209
  playlist                  124          0          124
  video                      96          0           96
  review                     83          0           83
  weather_snapshot           52          0           52
  blog_page                  38          0           38
  subscription               27          0           27
  message                    22          0           22
  reading_started            19          0           19
  code_file                  11         11            0
  outbox_draft                8          0            8
  conversation                3          0            3
  day_summary                 3          0            3
  timeline_event              1          0            1

You’ll notice the “Index-only” column has some big numbers. Some doc types — browser history, iMessages, tweets — live as index entries pointing at source databases or cloud accounts. The original bytes aren’t worth duplicating, so ark keeps the metadata and content for search but doesn’t manage a separate copy.

That’s nearly 700,000 items in ark. Most of these items were ingested automatically into ark from a variety of sources. I’ll talk about “ingestion at scale” next time.

1. I suppose I could do it over, but a long-standing tenet of this blog is that I make mistakes in public, and learn from them. It does no one any good to erase those mistakes and pretend they don’t exist. ︎
2. Today’s A.Word.A.Day word. This is the first mailing list I ever subscribed to. According to `ark` my first message from the list came on February 2, 1997. I think I’ve subscribed longer, but I may have been deleting that email. (I did delete email early on, not thinking about future value.) I still enjoy reading it each day it arrives in my inbox. I wonder how many people out there have subscribed to, and actively read, a mailing list for just about 30 years now? ︎
3. That word is just too hard to type. I’m going to stick with “org chart” going forward if you don’t mind. ︎
4. I’m speaking here of a file as an atomic unit of measure. I know that files are made up of bytes, but for the purposes of a workable archive, the file is the base unit. ︎
5. Recall from the previous post how `ark` commands are exposed to LLMs via an MCP server. This is an example of that in action. Here, Claude Desktop is taking my natural language question and turning it into a series of `ark` queries (this is the agentic model) to get back the best possible answer. ︎
6. Keep in mind I’ve only been using `ark` for a few months so many of these connections were “discovered” by the ingestion process (more on this next time). Over time, this should become a richer source of linkages. ︎

“Pivot! Pivot! PIVOT!” — Ross Geller

I had planned to write about ark‘s store and how it manages the items in the archive today. But it occurred to me that in writing about ark last week, I didn’t really show any of the system. Instead, I told a lot about the system. And as they say in the writing world: show, don’t tell. So this week I’ve opted to show what ark looks like so you get a tour before seeing the plumbing. Part 1 made the case for the archivist’s lens. Part 2 shows what it looks like. I’ve cast this as seven “scenes” through a day — not necessarily a literal day, but seven realistic moments, stitched into one. The post looks long only because of the archive output for the examples I’ve provided.

Scene 1 – Morning: Catching Up

ark has a wide range of items throughout my life. I need an easy way to see what’s there for a given day, and this is where the ark day command comes in handy. Here is what it looks like for the day before I began writing this post:

ark day yesterday
# Saturday, May 30, 2026  ·  yesterday
📍 Arlington, VA  ·  🌤 Partly cloudy  73°/63°F
💪 7,405 steps  6h 37m sleep  72 bpm  HRV 38 ms
   SpO2 95.4%  VO2max 35.3
## Calendar (2 events)
[1]  dare to dream  *Family*
[2]  zach soccer  *Family*
## Diary (1 entry)
[3] Evening Notes
    A beautiful day today. From the time I woke up and headed out for my walk, through dinner on the deck with Kelly and the…
## Reading (3)
[4] *The Complete Essays* by Michel de Montaigne  (started today)
[5] *In Memory Yet Green* by Isaac Asimov  (finished today ✓)  ✎4
[6] *Proto* by Laura Spinney  (started 2 days ago)
## Reminders (1 open)
[7] Check into my flight  *My Tasks*  ⚠ overdue 1 day
## Email (37 messages — none from known contacts, 30 filtered)
… 37 other messages  (use --all-emails to see all)
## People
[8] Kelly R.  (in 3 documents today)
[9] Eric S.  (in 2 documents today)
[10] Lisa K.  (in 1 document today)
[11] Zachary R.  (in 1 document today)
## Messages (7 messages)
[12] Kelly R. —   (+2 messages)
[13] Eric S. — Yeah but mine's not on your...  (+1 message, group)
[14] Zachary R. — https://www.youtube.com/watch?v=0h2jZtuRuic
[15] Lisa K. — Maybe he’s role playing  (group)

A few things to note here. First, ark is pulling information from the archive from a variety of sources. The header information contains the date (and how long it has been from that date to the time the command was run). It has my location, which ark captures either explicitly through artifacts in the archive, or through an inferred default location over time. The health information (steps, sleep, etc) comes from Apple Health.

It has seven sections that follow that try to provide a high level picture of the day: what was on the calendar and diary entries lead the way. The numbers ([1]) are sigils that can be used with other ark commands in place of longer document IDs. So I can type ark show @3 to see the full diary entry.

Reading is important to me so I can see what I was reading on that day, including what I started reading and what I finished. Any reminders from Apple Reminders show up for that day. Also email, but note that the email is filtered in this view to show only mail from known contacts, and despite getting 37 messages, none of them were from my known contacts that I’d want appearing here.

There is a list of people I interacted with on that day, or who appear in documents that were ingested on that day, followed by a list of text messages (or message groups) that took place on that day.

The command provides an easy way to get a view of any day in the archive. I could also have ark summarize the day for me via ark day 2026-05-30 --summarize. Like most ark commands, there are plenty of options for filtering and controlling the output. Here is what ark day --help looks like:

usage: ark day [-h] [--on-this-day] [--deep] [--all-emails] [--all-calendars] [--verbose] [--full]
               [--format {table,json}] [--bundle] [--summarize] [--regenerate]
               [DATE]
Compact daily digest — every item gets a [N] sigil for follow-on commands.
positional arguments:
  DATE                  Date to view (YYYY-MM-DD, 'today', 'yesterday'). Default: today
options:
  -h, --help            show this help message and exit
  --on-this-day         Include documents from the same date in other years
  --deep                Include low-priority items (music plays, browser visits)
  --all-emails          Show all emails, ignoring contact and blocklist filters
  --all-calendars       Show events from all calendars, ignoring the day.calendar.include filter
  --verbose             Full narrative output (old default behavior)
  --full                Show full document text instead of excerpts (--verbose mode)
  --format {table,json}
                        Output format: table (default) or json
  --bundle              Output as ark bundle format (pipe to ark task)
  --summarize           Display or generate a narrative summary of the day via LLM
  --regenerate          Force regeneration of the day summary (implies --summarize)
Subcommands:
  ark day rank [DATE] <-2..+2> [--note NOTE]   # set the Jim-Collins-style day-rank
  ark day rank [DATE] --clear                  # remove a rank
  ark day rank [DATE]                          # show the rank for a date
  ark day ranks [--since 30d|--year YYYY]      # chart + summary of recorded ranks
See 'ark day rank --help' or 'ark day ranks --help' for details.

Scene 2 – Triage: What Wants Attention

With a critical mass in the archive, and new items always coming in, it was a logical next step to make use of all of that information. After reviewing the day, I might make use of ark‘s “stewardship” layer. ark reviews items in the archive, as well as items coming into the archive, and identifies things that require my attention or action. From the CLI I can get a digest of this as follows:

ark prepare digest
# Stewardship digest — 2026-05-31
_act today: 0 · aging: 2 · outbox: 0 · stalled: 0 · review: 1_
**Yesterday (2026-05-30) unranked.**  Rate it: `ark day rank yesterday <-2..+2>` (or skip — auto-clears once ranked).
## Aging into Q1
- **[orange] Pay Arlington County utilities bill due 06/16/26** — Arlington County Utilities Services Office
  age *10d* · Q2 · *reminder*  (`ark prepare show 722866`)
- **[orange] Complete patient experience survey for my doctor's office** — (medical provider)
  age *8d* · Q4 · *reminder*  (`ark prepare show 724994`)
## Review (low-confidence classifications)
- **Prepare for upcoming trip to Boston per American Airlines reminder** — American Airlines
  conf *0.70* · *reminder*  (`ark prepare show 727451`)

or in TUI:

Here, I can see at a glance things that require my attention and take action on them¹. ark uses the Eisenhower matrix to put actions into quadrants around (1) Do First, (2) Schedule, (3) Delegate, and (4) Delete. That is what those Q1, Q2, etc., represent.

ark knows how to dedupe these items as they come into the archive so that if I get 3 reminder emails about the same thing, the thing only shows up once in the digest (even though all 3 emails are stored in the archive).

Working through the day, ark prepare digest is a nice follow-up to looking at the day or the previous day to get the context of what happened and what needs to happen.

Scene 3 – Looking Back: Finding What’s In the Archive

Let’s face it, a big reason for having an archive is so that you can search it. ark has sophisticated search capabilities that allow for a rich search experience. First, ark can do full-text searches of anything in the archive, whether text, PDFs, Office documents, email, text message, annotations, even photos. That’s because when something enters the archive it it goes through a process that “enriches” what comes in making anything searchable.

Let’s start with a regular full-text search. Here, I am going to search for items in the archive with the words “tax 2025”. The other parameters in the command tell it to limit the results to 10 items, to leave out snippets of the items, and to show how long it took the search to run:

ark search "tax 2025" --limit 10 --no-snippet --timing
Searching: 'tax 2025'
[1]   557281  imessage  2025-12-19  iMessage to High School Gang — 2025-12-19: He can now claim his tax-free…
[2]   151377  pdf       2025-08-09  Holiday Inn Express Neptune Receipt — 2025-08-07
[3]   152738  pdf       2025-01-07  Holiday Inn Express Florence Hotel Receipt — 2025-01-06
[4]   413583  pdf       2026-03-01  2025 St. Ann Faith Formation Donation Tax Statement
[5]   149508  pdf       2026-01-02  Holiday Inn Express and Suites Boynton Beach West — 2025-12-31
[6]   151497  pdf       2025-08-02  Holiday Inn Express and Suites Dickson City-Scranton Receipt — 2025-08-01
[7]   165270  email     2026-03-01  2025 Tax documents have been uploaded!
[8]   165269  email     2026-03-01  Re: 2025 Tax documents have been uploaded!
[9]   413659  pdf       2026-03-25  2025 Tax Summary
[10]  146864  email     2026-03-19  Prepare for tax time - access your 2025 prescription spending report
# 10 results
# query: 295ms

Not a bad mix of items, but probably, not quite was I was looking for, either. The search ran in about quarter of a second, but the results were meh. I was really looking for tax-related documents, not documents with the words “tax” and “2025” in them.

This is where a semantic search comes in. Semantic searches allow searching for meaning.

ark search "documents I'll need at year's end" --semantic --from 2026-01-01 --timing --limit 10
Semantic search: "documents I'll need at year's end"
[1]   148828  email           2026-01-23  Your 2025 tax documents are ready
[2]   413602  pdf             2026-03-01  2025 Tax Organizer - Completed
[3]   165270  email           2026-03-01  2025 Tax documents have been uploaded!
[4]   708408  pdf             2026-05-12  Brokerage Tax Package 10 — 2024 Tax Year
[5]   145936  email           2026-01-08  Important information about the year-end tax statement for your ho…
[6]   165269  email           2026-03-01  Re: 2025 Tax documents have been uploaded!
[7]   165261  email           2026-03-26  Re: 2025 Tax documents have been uploaded!
[8]   614083  calendar_event  2026-02-11  discuss CDMS remaining use cases
[9]   413598  pdf             2026-02-04  Form 1099-G — 2024 Tax Year
[10]  413659  pdf             2026-03-25  2025 Tax Summary
# 10 results
# query: 1052ms

In this case, I searched for “documents I’ll need at the end of the year” added a filter from the beginning of the year, and added the --semantic parameter to tell ark do perform a semantic search. The results are much closer to what I was looking for. They are tax documents or tax-adjacent documents when compared with the simple full-text search. The trade-off is (a) more time during ingestion to give each item embeddings² that can be used by semantic search, and (b) a slightly longer time to execute the query (because it uses AI) — just over 1 second compared to 0.250 seconds. But even against an archive with 700,000 items, it is a negligible change in speed and the value of the quality of the search far outweighs it.

Scene 4: A Conversation with My Archive via Claude Desktop (MCP)

With more than 7,300 posts here written over 20+ years, I can’t always remember if I’ve written about something before. For instance, as I write this, I am about to head up to Boston for a conference. In reviewing the profiles of the various attendees, I couldn’t help being bemused by the wild garden of jargon on display. Immediately I thought: great topic for the blog. But I’ve written a lot over the years and it was entirely possible I’d written about this before. Since all of my blog posts are captured in ark, I decided to check there first.

One of the great things about the design of the archive is that many of the commands in the system are exposed as tools to LLMs via an MCP server. That means I don’t always have to sit at the command line. I can go into Claude Desktop, for instance, and ask it directly: “I want to write a blog post about how business jargon annoys me. What have I written about jargon in the past?”

If you can’t read the screenshot, here is what Claude said in answer to my question:

Searched memory

The conversation search didn’t turn up anything specifically about jargon. Let me try the ark search directly, since that’s where your blog posts live.

Used ark integration, loaded tools

There’s a very promising hit: “Meaningless Statements, Vague Gibberish, and Useless Qualifications” (2021) and “What not to do in a resume” (2006). Also that “AI jargon” email from 2025. Let me pull the relevant ones.

Used ark integration

You’ve got two solid posts in the archive that directly touch on jargon. Here’s what I found:

“Meaningless Statements, Vague Gibberish, and Useless Qualifications” (March 10, 2021) — Your most directly relevant piece. It was prompted by a spam email pitching IT services that was so thick with buzzwords it meant nothing. You used the Foundation passage where Holk’s symbolic logic analysis of a politician’s speech finds, after eliminating all the meaningless statements and vague gibberish, that nothing is left. You particularly skewered “talk synergies,” looked it up in Merriam-Webster, and imagined an email parser that auto-replies with Strunk & White’s Rule 17: Omit needless words. The post got three comments.

“What not to do in a resume” (November 30, 2006) — This is a broader piece about resume writing, but item #4 is explicitly about jargon: “Don’t overuse jargon.” You wrote that you grow annoyed when bullet points read like a foreign language, and gave the example “Implemented best-of-breed, open-standards ITIL-based WEP-encryption application” as something that sounds ridiculous even if true.

So you’ve touched on this twice — once obliquely (resumes) and once head-on (the spam email). The 2021 post has the richest material, with the Asimov hook being particularly strong. For a new blog post, you might want to build on that Foundation framing more broadly, or shift the lens from spam email specifically to workplace communication generally. The 2021 post was reactive; a new one could be more systematic — here’s what jargon actually costs us, here are the worst offenders, here’s what plain language looks like instead.

So right away, I know that I’ve already written about jargon on the blog at least twice before, the last time in 2021 — so maybe it is time for something new.

I wanted to point out the line where Claude said “Used ark integration, loaded tools.” When Claude couldn’t find anything in our conversations about jargon, it immediately knew to use the ark tool provided via the MCP integration to check the archive, and it used the correct search terms to find previous blog posts about jargon. All I had to do was ask my question.

This is easier to use than the search commands on the CLI, but they also serve different purposes. The CLI tools can be used in conjunction with other tools and commands through pipes, redirects, and scripts. Using Claude lets me interact with the archive as part of other work I am doing. It also allows me to follow up on things. For instance, I might wonder how my writing style changes in the 15 years that separates the two posts.

It is cut-off in the screenshot, but here is the short version at the end of Claude’s response:

Short version: 2006 Jamie is a practical manager sharing hard-won tips. 2021 Jamie is a writer using an annoyance as a lens to say something about language, meaning, and the value of clarity — and reaching for Asimov to make the point stick.

Keep in mind, I could do this with any of the 700,000 documents in the archive. That said, there are some limits with the MCP, all intentionally imposed.

1. All documents in the archive get a sensitivity level. Any documents with a “Restricted” sensitivity are never returned to the MCP. These documents never leave the local machine. If I want LLM summaries of these documents, I can get them from the CLI, which will fall back to a local LLM that runs on my laptop.
2. Documents marked “Sensitive” are swept for sensitive information and that information is redacted before returning it to the MCP. For instance, if an SSN, account number, phone number, etc., appears in a document, those all get replaced with “[REDACTED]” before the document is returned to the MCP.
3. All documents are swept for a known list of sensitive information (SSNs, etc.) and are redacted before leaving the local machine, regardless of sensitivity.

Still, I can converse with about 90% of the documents that are in the archive, and that can be fun. For instance, I asked Claude what music I was listening to on past May 31sts³, and it came back with this:

The best part about the MCP integration, for me, is that I use Claude as a kind of personal assistant and with access to my archive, Claude is much more effective in doing things for me because it has a rich context to draw from.

Scene 5 – Adding My Voice: The Annotation System

It is one thing to have artifacts in an archive. But real value is added to the archive when you can layer in context to those artifacts. This is where the annotation system comes in. When items are pulled into the archive, they become read-only and cannot be changed⁴. The annotation system provides a layer atop the item itself to provide context to that item. When showing that item in ark, any annotations that are associated with the item are displayed as part of the output. Annotations are available in searches, and there are multiple ways of making annotations.

1. Directly through the CLI via the ark annotate command. This command uses the default editor set in your shell⁵.
2. Redirected into the ark annotate command. For instance, if I made a bunch of notes in a files called notes.txt, I could do this as the command line: cat notes.txt | ark annotate document 1234 to add the annotation to document 1234.
3. Via the MCP. Yes, I can ask Claude to annotate for me.

I have a book club meeting this evening. Our book was Wild Dark Shore by Charlotte McConaghy. ark captures my books and reading so I can search for the book in ark as follows:

ark search "wild dark shore" --type book
Searching: 'wild dark shore'
[1]  725665  book    Wild Dark Shore  ✎2
   ↳ Wild Dark Shore Charlotte McConaghy A family on a remote island. A mysterious woman washed ashore. A rising storm …
# 1 result

That [1] on the line with the matching record is a “sigil”. It is a shortcut reference to the item that I can use in future commands (until another search is run) instead of having to provide the full id (725665 in this case) each time I want to refer to it. When a book is added, ark searches sources on the Internet to pull in metadata about the book. When I look at the record via ark show here is what I see:

ark show @1
id:                725665
title:             Wild Dark Shore
doc_type:          book
record_class:      document
date_authored:
date_precision:    undated
date_ingested:     2026-05-24T14:29:23.876608+00:00
metadata_updated:  2026-05-24T14:31:12.699497+00:00
series:
sub_series:
sensitivity:       normal
priority:          normal
starred:           no
tags:              bookclub
slug:
physical_location:
source:
source_path:       ark:/books/work
store_path:
sha256:            7b3a2e46382a6c857413db215adb76e6192fe47d9f0c2658dd36b7b5406d2366
doc_status:        active
record_origin:     unknown
origin:
last_accessed:     2026-05-31 14:24:32
access_count:      8
author:            Charlotte McConaghy
published:         2025
genre:             Mystery
isbn:              9789025477356
Wild Dark Shore Charlotte McConaghy A family on a remote island. A mysterious woman washed ashore. A rising storm on the horizon.
Dominic Salt and his three children are caretakers of Shearwater, a tiny island not far from Antarctica. Home to the world’s largest seed bank, Shearwater was once full of researchers, but with sea levels rising, the Salts are now its final inhabitants. Until, during the worst storm the island has ever seen, a woman mysteriously washes ashore.
Isolation has taken its toll on the Salts, but as they nurse the woman, Rowan, back to strength, it begins to feel like she might just be what they need. Rowan, long accustomed to protecting herself, starts imagining a future where she could belong to someone again.
But Rowan isn’t telling the whole truth about why she set out for Shearwater. And when she discovers sabotaged radios and a freshly dug grave, she realizes Dominic is keeping his own secrets. As the storms on Shearwater gather force, they all must decide if they can trust each other enough to protect the precious seeds in their care before it’s too late―and if they can finally put the tragedies of the past behind them to create something new, together.
A novel of breathtaking twists, dizzying beauty, and ferocious love, Wild Dark Shore is about the impossible choices we make to protect the people we love, even as the world around us disappears. Mystery Thriller Literary Fiction
--- annotations ---
[23] 2026-05-31 11:21
  Initial thoughts after reading:
- characters: Rowan, Dom, Raff Orly, Fen
- a kind of Robinson Crusoe or Swiss Family Robinson about the story, only darker
- did a very good job for the first two-thirds at playing out the mystery
- last third of the book took too many zigs and zags and was on the verge of becoming parody
- loved the setting of the book
- wondered why there would only be one place in the world where the seeds would be kept for rebuilding civilization; one would think there would be multiple places; also why one an island as wracked and ravaged as this one.
- odd to have one chapter from Alex POV -- it throws off the rhythm of the story; as if author could not come up with any other way of conveying that info
[24] 2026-05-31 11:30
  **Book Club Discussion Notes — May 31, 2026**
*Initial reactions:* Strong first two-thirds that sustains mystery well; final third overloaded with reversals, verging on parody. The island setting is the book's most consistently successful element, grounded in McConaghy's research on the real Macquarie Island.
...

After all of that metadata for the book record, note the section called --- annotations ---. This is where I have gone and added annotations to this particular book record. I’ve added two, as you can see, but I can add as many as I want. These are not part of the book record, but rather are linked to the book record. The ark show command is pulling all of this together when it shows the book to me.

Adding the annotation is as simple as running the command. Or, asking Claude to do it for you.

Here, not only was Claude easily able to find what I was looking for, but once found, it summarized it as requested, and then added the annotation that I asked for. For me, this turned annotation from a chore into something easy. I can sit with my book and as I read, tell Claude to add various annotations to the book for the thoughts or passages I want to comment on.

Scene 6 – ark quality: The Archive That Knows Where it is Weak

Archivists work to ensure the quality of the archives they manage. That not only means preserving the archive but enriching it with as much information as they can about the contents of the archive. Each item in ark has a certain set of metadata. It has the date of the item, the date it came into the archive, the source it came from, its location in the real world (if it is not digital in the first place). Each item has a title, a series and sub-series, a format, a sensitivity level. Items can have OCR data, embeddings, generated descriptions, and more. With all of that possible data surrounding an item in the archive, some way is needed to assess the quality of items in the archive.

Enter the ark quality command.

This command provides a way of looking at the archive as a whole seeing the overall quality of items. It also allows for looking at a specific item and seeing the quality of that item. Quality is a score between 0-100 that is a measure of how complete the picture of the item (or archive) is. An item with no metadata at all rates a 0. An item with every possible element filled in rates 100. Here is a quality summary of the entire archive.

ark quality summary
=== ark quality — Archive Completeness ===
  Total documents:  681,576
  Median score:     76/100
  Score distribution:
       0-25                                   11,116  (1.6%)
      26-50  ███                              85,298  (12.5%)
      51-75  ███████                         178,002  (26.1%)
     76-100  █████████████████               407,160  (59.7%)
  Median score by type:
    attachment        ████                   24/100
    subscription      ███████                35/100
    reading_started   ████████               40/100
    calendar_event    █████████              47/100
    cli_command       █████████              47/100
    reminder          █████████              47/100
    note              ██████████             53/100
    video             ██████████             53/100
    reading_finished  ████████████           60/100
    code_file         █████████████          65/100
    conversation      █████████████          65/100
    imessage          █████████████          65/100
    outbox_draft      █████████████          65/100
    text              █████████████          65/100
    image             ██████████████         70/100
    browser_visit     ██████████████         73/100
    timeline_event    ██████████████         73/100
    watch_event       ██████████████         73/100
    blog_comment      ███████████████        76/100
    blog_page         ███████████████        76/100
    book              ███████████████        76/100
    day_summary       ███████████████        76/100
    email             ███████████████        76/100
    facebook-post     ███████████████        76/100
    message           ███████████████        76/100
    office            ███████████████        76/100
    pdf               ███████████████        76/100
    purchase          ███████████████        76/100
    review            ███████████████        76/100
    tweet             ███████████████        76/100
    git_commit        █████████████████      87/100
    music_play        █████████████████      87/100
    weather_snapshot  █████████████████      87/100
    action_item       █████████████████      88/100
    blog_post         █████████████████      88/100
    health_day        █████████████████      88/100
    playlist          █████████████████      88/100
    diary_entry       ████████████████████  100/100

Here you can see that about 60% of the items in the archive fall into the top quadrant of quality. The highest quality class of items are diary entries. The lowest quality are attachments. This makes sense because attachments are pulled in from email messages and outside of getting descriptions from vision models and embeddings on those descriptions, they have almost no other metadata.

Anything that comes into the archive has a title. Often times it is simply the name of the file that was ingested and we all know that filenames can be ambiguous or provide little useful information about what the file contains. ark can assess the quality of titles as well, for instance:

ark quality titles --limit 20
=== ark quality titles — 20 document(s) with bad titles ===
       ID  Reason              Type            Date        Title
  ───────  ──────────────────  ──────────────  ──────────  ──────────────────────────────────────────────────
     3792  filename w/ ext     attachment      —           Suggestions_for_OS_8_image.doc
     4048  filename w/ ext     attachment      —           Nuv.doc
     4245  filename w/ ext     attachment      —           database.doc
     4326  filename w/ ext     attachment      —           OS_8_seminar_outline.doc
     5032  filename w/ ext     attachment      —           Proposal.doc
     5342  filename w/ ext     attachment      —           g970527.doc
     5343  filename w/ ext     attachment      —           g970515.doc
     9101  filename w/ ext     attachment      —           updaterr.doc
    11223  filename w/ ext     attachment      —           REV980617ACOSTEDBOM1.XLS
    11608  filename w/ ext     attachment      —           adopters2.xls
    11937  filename w/ ext     attachment      —           Development.xls
    12092  filename w/ ext     attachment      —           kpatel.doc
    12859  filename w/ ext     attachment      —           BDY.RTF
    14769  filename w/ ext     attachment      —           98issues.doc
    15137  filename w/ ext     attachment      —           michael.doc
    15326  filename w/ ext     attachment      —           Invoice1.xls
    16099  filename w/ ext     attachment      —           Dun12.doc
    16200  filename w/ ext     attachment      —           Lanrover.doc
    16533  filename w/ ext     attachment      —           %adopters.xls
    16534  filename w/ ext     attachment      —           adopters.xls
  20 document(s) shown.
  Use 'ark enrich titles' to fix with LLM-proposed replacements.

Note that in addition to listing items in the archive with poor quality titles, the command also tells you how you can raise the quality of those titles using ark enrich titles. Automated enrichment is a key element of ark. It uses LLMs to read the document and give a descriptive title to the document based on the context. Each document class has its own template of title format. So for instance a letter might have the format “Sender-to-recipient (date)” and the LLM will produce titles in that format so that they are consistent across that class of items, raising the quality and consistency of the archive.

ark also uses a controlled vocabulary for its metadata like series and sub-series. When something is ingested it may get a series that falls outside that controlled vocabulary. ark quality vocab can help to identify such items:

ark quality vocab
=== ark quality vocab — Controlled Vocabulary Check ===
  Vocabulary: 15 series configured
  Violations: 4 unknown series value(s), 10 unknown sub-series value(s)
  Unknown series (4):
    Series                      Docs
    ────────────────────────  ──────
    health                      9392
    shopping                    5813
    diary                        624
    environmental                 43
  Unknown sub-series (10):
    Series              Sub-series              Docs
    ──────────────────  ────────────────────  ──────
    media               watch                  26149
    shopping            digital                 2642 (unknown series)
    shopping            audible                 2049 (unknown series)
    shopping            retail                  1122 (unknown series)
    media               reviews                   83
    correspondence      amazon                    22
    legal               identity                  11
    legal               power-of-attorney          1
    legal               will                       1
    professional        systems administration       1
  Sample documents with unknown series (up to 10):
         ID  Series              Sub-series        Title
    ───────  ──────────────────  ────────────────  ────────────────────────────────────────
     728106  environmental       —                 Weather: Arlington, VA on May 30, 2026
     728102  health              —                 Fitbit: May 30, 2026
     727759  environmental       —                 Weather: Arlington, VA on May 29, 2026
     727758  health              —                 Health: May 30, 2026
     727751  health              —                 Fitbit: May 29, 2026
     727457  environmental       —                 Weather: Arlington, VA on May 28, 2026
     727456  health              —                 Health: May 29, 2026
     727444  health              —                 Fitbit: May 28, 2026
     727087  environmental       —                 Weather: Arlington, VA on May 27, 2026
     727086  health              —                 Health: May 28, 2026
  Fix: ark set document <id> --series <valid-series>
  Or:  ark normalize-series  (if renaming a whole class)

Once identified, ark provides commands for automatically fixing the items, either manually, or across the entire series.

All of this helps to raise the quality of the archive, making searching more accurate.

Scene 7 – Closing the Day

We’ll conclude the tour with one way of looking at the work I got done yesterday. This is a good example because it shows how ark was really designed with core Unix principles, each command doing something the output of which could be used as input for other commands.

Let’s begin by looking at what the archive ingested from yesterday. The easy way to do this is as following (I’m limiting this to just 10 entries for space):

ark recent --days 1 --limit 10
[1]   728102  health_day    2026-05-31  2026-05-30  Fitbit: May 30, 2026
[2]   728101  diary_entry   2026-05-31  2026-05-30  Evening Notes
[3]   728100  git_commit    2026-05-31  2026-05-30  feat(cli): auto-follow supersedes chain to head on reads…
[4]   728099  git_commit    2026-05-31  2026-05-30  feat(search): identify multi-version docs via --has-vers…
[5]   728098  git_commit    2026-05-31  2026-05-30  feat(vim): :BlogShot — SnagIt → Vim → WordPress screensh…
[6]   728097  git_commit    2026-05-31  2026-05-30  fix(reading): ark reading finish <id|@N> stored raw CLI …
[7]   728096  git_commit    2026-05-31  2026-05-30  fix(blog): standalone images publish as wp:image blocks;…
[8]   728095  outbox_draft  2026-05-31  2026-05-31  Attempts  ✎1
[9]   728094  playlist      2026-05-31  2026-05-31  Recently Added (v5)
[10]  728089  music_play    2026-05-31  2026-05-28  On Your Shore — Enya
# 10 results

But if we wanted to know the total number of things ingested from yesterday we can do this:

ark recent --days 1 --limit 200 | wc -l
# 81 results
      82

Next, we can use the ark bundle command to generate a “bundle” of one or more documents from ark. This one bundle would contain the contents of all of the documents we put into it. So for instance, if we wanted to generate a bundle with the full contents of everything that came into ark yesterday we could do this:

ark recent --days 1 --limit 200 | ark bundle

which would produce output containing the contents of all 81 items ingested yesterday — a total of 799 lines so I won’t show them all here.

Finally, we can redirect that bundle to an ark task. A task is a prompt for an LLM to do something with the data. We’ll use my “shipped-yesterday” task, which summarizes everything I “shipped” (code commits, posts written, emails sent, etc.) Putting it all together, it looks like this:

ark recent --days 1 --limit 200 | ark bundle | ark task shipped-yesterday
# 81 results
ark: warning: Redacted 1 sensitive pattern(s) before sending.
Running task: 'shipped-yesterday' (standard)
## May 30, 2026 — Daily Output Report
### Shipped
Five commits to the **ark** repo:
- feat(cli): auto-follow supersedes chain to head on reads (#94 Phase 1); bump to 1.68.0 (728100)
- feat(search): identify multi-version docs via --has-versions + (vN of M) annotation (#95); bump to 1.69.0 (728099)
- feat(vim): :BlogShot — SnagIt → Vim → WordPress screenshot workflow; bump to 1.69.1 (728098)
- fix(reading): ark reading finish stored raw CLI arg as event title; bump to 1.69.2 (728097)
- fix(blog): standalone images publish as wp:image blocks; caption via title (#96); bump to 1.69.3 (728096)
### Written
**Diary** — "Evening Notes" (May 30) covers a pleasant day at home, mentions writing an internal company blog post on AI use, notes a traffic spike (2,300+ views) on his public blog, and reflects on starting Montaigne's *Essays* and a desire to re-read *The Stand*.
**Self-email** — "Tomorrow's Plan — Sunday, May 31, 2026" sent from j**.**@icloud.com to himself; a detailed day-plan covering the DCA→BOS flight, the dare-to-dream volunteer event, open action items, and upcoming occasions.
### In Flight
**"Attempts"** (outbox draft, WordPress destination) — a personal essay connecting the etymology of "essay" to the author's own writing history, from junior high through college, prompted by a morning walk and an archive search. The draft is complete in the outbox but not yet published.

The first thing to note is the warning that ark gives on that second line after the command:

ark: warning: Redacted 1 sensitive pattern(s) before sending

This is an example of what I was talking about earlier. Sensitive information never leaves the local machine; it is either blocked, or, as in this case, redacted before sending it to an LLM.

The rest summarizes the work I did yesterday, based on how I specified the way I wanted it summarized in the “shipped-yesterday” task.

Seven scenes, but really seven angles on one habit. Most days I use four or five of these without thinking about it: the archive surfaces the day, I notice what wants attention, I look for something I half-remember, I add a note to something I just read, I check what shipped. None of those scenes are unique to ark taken in isolation — search engines search, task managers triage, note apps annotate. What’s unusual is that they sit on top of the same archive, share the same discipline, and the parts compound. That’s the payoff Part 1 promised the archivist’s lens would deliver.

The next post takes the floor out from under all of this and shows what holds it up — ark‘s data model. A single SQLite database, a clean graph connecting documents to people to one another, and a schema disciplined enough that it should still answer the same questions thirty years from now. Plumbing always sounds duller than the demo, but it’s where this stops being a pile of clever commands and starts being an archive built to last.

1. Eventually, `ark` will be able to take some of these actions for me! ︎
2. An embedding is a way of converting text (or other data) into a list of numbers — a vector — that captures its meaning. The key property is that things with similar meanings end up with similar numbers, so you can do math on meaning. When you ask an LLM a question, your question is turned into an embedding and that math is applied to see how “similar” it is to other embeddings. ︎
3. Yes, `ark` has a history of everything I’ve listened to (via Apple Music) and watched (via YouTube, Amazon, Netflix and a variety of sources). More on this in the next post in the series. ︎
4. They can be versioned and superseded, but that is a topic for a future post. ︎
5. MacVim in my case. ︎

And then this morning, on my early walk, I was listening to SiriusXM 80s on 8 top 40 countdown for this week in 1987. The music immediately had me back in 9th grade, my final year at Porter Junior High School, and less than a month from graduating. I don’t remember much about the writing I did in Junior High. I know that in 7th grade, we had to research and write books (short books) and then made bindings for them. My book was on Egypt, a subject I never would have chosen, except that I was trying to impress a girl, who liked Egypt. Indeed, I was much more into computers and technology than writing at that point in my life, even receiving recognition for a rudimentary flight simulator I created for my programming class. I received similar recognition for math. But writing? Not until high school!

And yet… Hearing those songs this morning had me wondering about that final year at Porter. I returned from my walk and sat down at this very laptop and at the command-line, I asked [ark](https://jamierubin.net/ark/) what I had in my archive from 9th grade. And the very first match surprised me: an outstanding merit award for the young authors project!

Now, truth be told, I don’t remember the young authors project, and I would never refer to myself as a young author (or an old one for that matter). A writer, yes; an author? Not me. And I suspect that whatever it was that I was writing², it was not essays. But it was a bridge of some kind that got me to high school, where the program I was in tested its students through a single method: the essay. It was there that I made countless “attempts” at writing essays to make an argument, to convince, or cajole, or to see just how many pages I could whip out in cursive in a 2-hour period without my hand falling off.

It is because of those essays that the essays I wrote in college were both more enjoyable and easier to do. I’d already had my 10,000 hours of practice³. And it was in those college essays — where I Puckishly avoided the traditional stilted style and wrote mostly colloquially — that I developed the style that would become what I think is the voice of these attempts.

Incidentally, this week’s countdown wasn’t a bad one. But given that I am now out of junior high school some 39 years, I can’t for the life of me recall what the #1 song was.

1. This never really changed. In college I was something of an anomaly among my friends in putting off writing an essay until the last possible minute — often in the wee hours on the morning the paper was due. And yet, it flowed naturally somehow, without much effort, and I nearly always received an A. ︎
2. It is not preserved in my archive. ︎
3. It seemed like 10,000 hours. ︎

You know that ol’ Peanuts cartoon? The one where Lucy holds the football for Charlie Brown so that he can kick it? C.B. is reluctant because he’s afraid Lucy will pull the ball away at the last second and he’ll fall on his can. But somehow, Lucy always convinces him that this time she won’t pull the ball away, and C.B. goes to kick the ball and Lucy pulls the ball away, there is C.B. on his can.

That is me, chasing windmills.

Windmills, in this instance, are interminably long books, that despite their interminable length, have an allure, a siren song, a mystique, and a cultish air of mystery that makes me return to them again and again, only to have Lucy pull the football from me, and whoops! Here I am on my can once again.

I am foolishly seduced by these types of books, unable to resist their charms. And yet, unable to commit fully to the relationship. I am thinking about this now because, well, not to beat around the bush, but I’ve been ensorcelled by one of these windmills. Sitting in a waiting room today, I found myself reading¹, for no good reason other than, well, wily charms, Boswell’s Life of Johnson. I could say it was because, again and again, I’ve heard that it is the ultimate biography, and as one who reads a lot of biographies² who could resist that tag line? But the truth is, I am weak in the knees for these windmills. The Oxford paperback edition on my shelf is 1,402 pages, not counting the index. And on some pages, footnotes consume almost the entire page! I feel faint.

I’ll tilt at that windmill for a while. And then I’ll start to think to myself: what am I missing by continuing this fruitless quest? There’s that Laura Spinney book, Proto on early language that I’ve wanted to read. Jim Rasenberger just came out with Perfect Coincidence about the friendship of Jefferson and Adams that I’ve had my eye on. Or there’s The Death and Life of Zebulon Finch, Volume One by Daniel Kraus, which while long, is not in the category of a windmill.

I’ve learned to indulge these peccadilloes knowing that the affairs will burn out, the attempts will fail. But the allure of those windmills remain.

Other hills I’ve tried to climb, other windmills I’ve tilted at? Well, there’s Infinite Jest by David Foster Wallace; there’s Marlborough: His Life and Times by Winston Churchill; there’s The Structure of Evolutionary Theory by Stephen Jay Gould.

It is the height of irony that one of these windmills of mine would be Don Quixote by Miguel de Cervantes.

But here I am, enchanted at the moment by The Life of Johnson, unable, at the moment, to her aside for something more frivolous, with fewer moving blades, and a less imposing demeanor. Or until Lucy pulls away the football and I am back on my can.

1. I usually carry a book with me wherever I am, but for Puckish reasons, I find it delightful to be the only person in a waiting room / line / theater lobby who is reading a paper book rather than looking at a phone. ︎
2. A couple of hundred at least, if anyone’s wondering. ︎

A couple of months ago, I was searching for a trust document. I’ve got two computers and two external drives. I tried multiple combinations of searches across all devices. I used Spotlight on both machines, and then switched to Unix-based search commands. Despite knowing the document existed somewhere, my search abilities couldn’t surface it.

I. A Moment of Friction

I have thought about personal archives now and then — the kind you read about in biographies of notable people. I recall reading about how Boston University asked Isaac Asimov to collect his papers. I began to wonder if it was possible to create a similar archive for all of my papers. I took a small step in this direction in late 2024 with my Personal Archive System. This was an experiment to see what was in the realm of the possible, it was web-based, but there wasn’t much thought about the long-term design and architecture — the very kinds of things I do in my day job.

At the same time, I had been using Claude Code at work to help me build a command-line system that made it easy to interact with Jira and tie Jira into an LLM for easy summarizing. I was impressed by how well Claude Code worked and how it felt almost like a real collaborator. In that project, I acted as an architect, and Claude Code did all of the grunt work for me.

At the end of March, I decided to sit down and, with Claude Code as a kind of partner¹, talk through how to best design a personal archive system that takes its core design principles from real archiving principles, while meeting my requirements for what I wanted in an archive system.

II. An Archivist’s Lens

Longtime readers know that I have gone through at least two major iterations of personal archiving of sorts. The first, in the early 2010s was the time I spent using Evernote to go paperless. That experiment lasted several years. Ultimately, however, it wasn’t a good fit for what I was trying to do. Part of the problem was that, at the time, I’m not sure I knew what I was trying to do. I knew that I wanted to be able to find things quickly. In order to do that, Evernote required some amount of metadata infrastructure (notebooks, tags) and for me, maintaining that became a roadblock.

The second wave, in the 2020s has been my use of Obsidian to go practically paperless. One thing that attracted me to Obsidian was its simplicity. At its core, it was plain text, the most basic, most portable form of data there is. Another thing that attracted me was that it was entirely local. No need to store data in the cloud. Everything was on my local machine. Ultimately, however, what I discovered was that both Evernote and Obsidian were working tools. That is, places to do work, take notes, etc., rather than a stable archive of work already done. In other words, these tools are optimized for now. An archive is optimized for posterity.

As I worked on the design of the system, one question that guided every decision was how would a real archive handle this, and where does the personal context require an adaptation? One obvious difference: in a public archive, the archivist and the subject are different people; here they are the same person. Other than that, the principles that guide a public archive could apply here. These principles include things like:

• Provenance
• Finding aids
• Controlled vocabulary
• Sensitivity
• Accession

For instance, every item in the archive comes from somewhere in either the digital or physical world. This is its provenance. An archivist typically organizes items in an archive in a hierarchy that starts with series at the top. A series might have sub-series. Items in a sub-series might go in one or more files. Within the files are the items themselves. There is always, therefore, a clear path to an item in the archive.

III. Non-Negotiables

I came into the design discussion with several non-negotiable design decisions:

1. Local data storage: The archive would be designed to be stored locally on a file system.
2. A clear data egress boundary: A personal archive by its very nature will contain sensitive documents in a life: medical, financial, legal, etc. The sensitivity of these documents is decided at ingest and enforced at egress as a hard rule. As we will see, the archive makes use of LLMs for a variety of tasks. But most LLMs reside on the internet and that means sending data to them. ark was designed to block sensitive data from egress to these sources².
3. Hands-off automation: When bringing in hundreds of thousands of documents, manual classification, linking, etc. is out of the question. There has to be mechanisms for automating this process.
4. Durability over cleverness: Every architectural decision, from the local file storage, to the database, Unix composability, agnostic LLM layer, etc., all favor longevity. Services come and go. Formats change. APIs get deprecated. The goal isn’t to have the cleverest tool; it is to have a tool that still works fifty or a hundred years from now.

IV. Pipeline as an Operational Spine

There are seven stages that form the operational backbone of ark.

1. Ingest: Includes parsers for email, PDFs, images, Office documents, Markdown, as well as other sources of data like calendars, text messages, read data, music, health data, and more. When ingested, this data goes through type detection, text extraction, OCR, deduping. Think of this as the narrow waist where everything entering the archive gets normalized into the same shape.
2. Enrich: Two flavors of enrichment: (1) automated, which includes things like classification, embeddings, mood scoring, place and person extraction; (2) human enrichment, which includes things like annotations and people curation.
3. Store: Content-addressed file storage plus SQLite database holding records, the full text index, embeddings, annotations, and the relationship graph. Ingested data enters the store and becomes a permanent immutable document³.
4. Search: Or more formally, retrieval, includes full-text, semantic, hybrid, and person-aware searches, and a scoring system that ranks results by signal rather than by raw text match.
5. Surface: What the archive can show you outside of searches: timelines, “on this day”, action-item digests, etc. These are distinct from retrieval in that you don’t explicitly ask for them. The archive offers them.
6. Synthesis: Creating “bundles”, LLM tasks, deep searches, MCP. This is where the archive stops being a corpus and starts answering questions.
7. Stewardship: Proactively looking for and creating actions based on what is coming into the archive; automatically drafting replies, etc., for incoming activities. This is the bridge to automation that may allow the archive to act on my behalf as a kind of personal assistant.

V. What’s Coming in the Rest of the Series

The remainder of this series will describe each of these layers from inside the design:

• Part 2 zooms into the data model and graph that serves as the connective tissues between documents, people, and annotations.
• Part 3 covers the ingestion process — how data comes into the archive, in bulk, and through a daily ingestion process.
• Part 4 covers retrieval.
• Parts 5-6 cover the enrichment layers.
• Part 7 gives a tour of the way books and reading are handled by ark. Both are a significant part of my daily life, and warranted their own functions in the system.
• Part 8 covers synthesis: bundles, LLM tasks, deep searches, and the “recommend” surface.
• Part 9 covers stewardship — the layer that turns ingested material into prioritized work.
• Part 10 covers publishing back out via the outbox.
• Part 11 covers the operational tools and functions of living with the archive day-to-day: sync, location, timeline, day rank, etc.
• Part 12 covers the four interfaces: CLI, TUI, MCP, and Vim.
• Part 13 is all about the long game: production, backup, export, and durability.
• Part 14 closes with a portrait of what the archive actually holds.

VI. The Long Game

ark looks the way it does because archivists solved these problems first, and because the design choices that matter most at a five-year horizon are different from those that matter most at a fifty-year horizon. Keep that in mind as you read the rest of the series.

VII. Coda

It took about a week of work to get the core system up and running, and to get 80% of the documents I had into the archive. As I write this, there are 680,497 items in the archive. There were around 400,000 at the end of that first week. At that point, I used the tool to search for the trust document I was searching for. By then, the enrichment layer was in place, and the basic search functionality automatically enriched the search as well. I ran a simple search command in ark:

ark search "living trust" --type pdf

which after about 0.30 seconds, returns a single match: the exact document I was looking for in the first place.

Two months of nights and weekends in — about 1,250 commits, 116,000 lines of Python, 4,800 tests — ark holds 680,497 documents across fifty tables: emails going back to 1994, every diary entry I’ve digitized, photos with their GPS, scanned tax records, blog posts, calendar events, books I’ve finished, music I’ve listened to. The 9 GB SQLite database and 125 GB content store live on this laptop and back themselves up overnight. Everything I’ve described above is what makes that pile navigable.

1. I am well aware that a large language model is nothing more than a complex mathematical construct, not a real entity or consciousness. I use the term “partner” as a convenient shorthand for how I worked with this particular tool. ︎
2. As we will see in a future post, `ark` has a fallback that allows these sensitive documents to be routed to a local LLM, if one exists. ︎
3. `ark` is version aware and a newer version of a document can “supersede” a previous version. The archive stores all of them, linking them together appropriately and displaying only the current version, unless an earlier version is requested. ︎

I’ve been thinking a lot about my time in college, about the things I did, and didn’t do; thinking about the lifelong friends I made there; thinking about the classes I took and the professors I studied under. I’ve been thinking about the music I listened to while in school, whether it was while studying, or working in the dorm dish room, or walking to-and-from classes listening to cassettes on my Sony Walkman¹. I’ve been thinking about the movies I saw, and the papers I wrote, and the late night, deep philosophical debates I had with friends and dorm-mates.

A university is a place of learning — not just a place to discover the secrets of the universe, or to delve into the mysteries of human psychology, or the life cycles of insects, or whatever interests you — but a place to figure out who you are and what you want to do with your life. It is a place to meet new and interesting people, to try new things, whether it is a food or drink, a sport, a hobby. It is a place designed to encourage exploration and experimentation in all directions. Indeed, it is a unique experience in a life, an island of learning that is in many ways cut off from the real world. It is less haven and more heaven.

In his memoir I. Asimov, Isaac Asimov writes,

I dreamed I had died and gone to heaven. I looked about and knew where I was — green fields, fleecy clouds, perfumed air, and the distant, ravishing sound of the heavenly choir. And there was the recording angel smiling broadly at me in greeting… [I] asked, “Is there a typewriter I can use?”

In W.P. Kinsella’s marvelous novel Shoeless Joe,² after shagging flies and grounders in a cornfield, Shoeless Joe speaks:

“God what an outfield,” he says. “What a left field.” He looks up at me and I look down at him. “This must be heaven,” he says.

“No. It’s Iowa,” I reply automatically. But then I feel the night rubbing softly against my face like cherry blossoms; look at the sleeping girl-child in my arms, her small hand curled around one of my fingers; think of the fierce warmth of the woman waiting for me in the house; inhale the fresh-cut grass smell that seems locked in the air like permanent incense; and listen to the drone of the crowd, as below me, Shoeless Joe Jackson tenses, watching the angle of the distant bat for a clue as to where the ball will be hit.

“I think you’re right, Joe,” I say, but softly enough not to disturb his concentration.

For me, heaven would be a university, populated with countless interesting people: students, professors, faculty, staff. It would not be a place where, after a life well-lived, I would suddenly learn the meaning of the universe in one divine blast of knowledge. Rather, it would be a place where I could mingle with others, drink a pint at a local pub, watch a band, and in between, have limitless time to learn, to read, to study, and to slowly and carefully seek out the secrets of the universe for all eternity.

Of course, this is what I do every day right here on Earth. It is what I have done ever since I picked up a copy of The Nine Planets at the Franklin Township Public Library when I was five or six years old. It is why my hobby is learning. It is why it is a hobby that I will never give up. It is why, despite the turmoil, and trying times we live through, I see life very much like a university, populated with countless interesting people, a place where I can mingle with others, drink a pint at a local pub, watch a band, and in between, learn, read, study, and slowly, carefully, seek out the secrets of the universe. Would that I had the limitless time.

1. This was long before MP3s, iPods, or streaming media. ︎
2. The film Field of Dreams is based on this book. ︎

As they scoured the store for any signs of Squeezy Squishy things, I wandered around, looking at the various tables of books. I couldn’t recall the last time I’d been in a Barnes & Noble, but as I browsed through the various themed tables, I noticed something: at every table I looked at, no matter the theme, there was at least one book that I had read. At some tables, there were three or four. I pointed this out to the girls. They pointed to a random shelf (World History) , asked if I’d read anything there. I immediately pointed to Churchill by Andrew Roberts. It was an bit of an odd, yet oddly satisfying experience.

It was a less satisfying experience for the girls. There were no Squishy Squeezy things to be had in Barnes & Noble.

I’ve always admired prolific people, whatever their profession. For a time, I was even mildly prolific writing here on the blog. But I was never a prolific writer of fiction, for instance. It would take me a month or more to write a story, and that is a non-starter when it comes to being prolific. Yet somehow, slowly, but steadily, I became a prolific reader. This pleases me, as silly as it may seem. I’ve written before that my hobby is learning. What pleases me even more is the range of my reading. No matter what area of the store I wandered through, there was something I’d read.

Once we realized there were no Squeezy Squishy things at Barnes & Noble, we headed out home.

And yet, the story has a coda. Being prolific runs in the genes. The girls are prolific shoppers. They located Squeezy Squishy things at a nearby Target. So I drove them over there and the store indeed had the Squeezy Squishy things, stored in a cabinet at guest services. Mission accomplished. With that success in hand, I took the girls to lunch at a diner next door.

It was a prolific morning.

The most remarkable thing, however, is the change in technology that has taken place over that three decade span. I stood at my desk last night, listening to Elvis Costello on a Bose Bluetooth speaker that sat on a shelf to my right, the music streaming from my iPhone which stood on a wireless charger on the opposite side of my desk. Instead of a single, bulky CRT on my desk, four flat screens sweep across the surface of my desk today: a MacBook Pro, connected to two large external monitors, my MacBook Air, all of which work in concert. The nest of wires that used to accumulate across my desk have been replaced by wireless keyboards and trackpads.

And the software! I remember installing Microsoft Office on something like 26 3-1/2-inch floppy disks. Now, I can download and install a much more advanced Office in far less time than I could install it from those floppy disks. And with tools like Claude and Claude Code, I have a personal assistant to handle routine things for me, leaving me more time to focus on the things that I am most interested in.

Still, despite all of these changes, some things don’t change much. I am writing this essay in Vim, which I used back then, and which I have used off-and-on ever since. The muscle memory of the keyboard commands has become second nature. I still spend the vast majority of my time in Unix command line terminals, or in similar LLM chats. In some sense, with the advent of LLMs, UXs have started moving back toward the command line, an irony, which, as a command line enthusiast, I relish.

And while I can simply ask Siri to define or spell a word for me, I prefer to reach for one of the ten dictionaries that sit on a shelf beside my desk. And while streaming music is convenient, and I do it frequently, right here on a cabinet beside me, and sitting next to my ScanSnap ix1300 scanner, is a Victrola turntable, which my son got me as a present, and for which I have started to amass a small collection of vinyl, and which I much prefer for listening to things like Miles Davis’ Kind of Blue or the Eagles’ Hotel California.

One thing that old office had that my current office lacks is a cabinet full of file folders. Over the three-decade span, my files have slowly made their way into the digital realm, slowly at first, then at a more accelerated pace in the early 2010s when I adopted Evernote. When I gave up on Evernote in the early 2020s, I moved a lot of things into Obsidian, but I found that Obsidian was more of a tool for writing than managing an archive. So after searching around and looking at tools like DevonThink, among others, I finally built my own archive, first testing things out in what I call my Personal Archive System (PAS), and then really studying the problem the way an archivist would, and with the help of Claude Code, building a new and sophisticated command-line system called ark.

Today, ark hosts my entire archive, more than 670,000 items, including every email I’ve sent or received since late 1994, and digital documents going back to 1992, scanned documents going back to my birth, as well as a pipeline of continually updated data for capturing browser history, CLI history, my entire social media history, photos, calendars, diaries, blog posts, comments, reading history, annotations, Apple Health data, you name it, it is in the archive. The archive has a sophisticated search capability that allows not only full-text search, but embedded searches¹ (every item gets an embedding, every photo gets an LLM description and embedding). Because I’ve created an MCP server² for the archive, I can “chat” with the archive using Claude. And I’ve built an egress layer to prevent sensitive information from leaving my local machine (for sensitive data, embeddings and LLM activities use a local LLM³.)

This is a big difference to my office from 30 years ago. Instead of hunting through paper documents, or hacking my way through files in the file system, I can type a simple command on the command line, or even ask Claude, “What posts have I written about my desk?” and it does all of the grunt work for me. That, at least for me, is a change for the better.

(P.S.: if you are curious about Claude’s answer to that question, here it is:

I can see the blog post titles clearly from the social media announcements. Here’s what turned up — looks like you’ve written at least four dedicated desk posts:

1. “My desk, annotated” (September 2010) — the earliest one, with an annotated look at your desk setup.
2. “My Cluttered Desk” (May 2019) — a post about desk clutter.
3. “The Desk and The Desktop: Musings on Productivity, Part 1” (April 2021) — a more reflective productivity piece.
4. “A Newer Healthier Desk for My Office” (July 2021) — likely tied to the L-shaped electric standing desk you bought from Amazon in June 2021.
5. “A Cluttered Desk” (August 2021) — another clutter-themed post, just a month later.

There’s also a tweet from August 2021 quoting the Einstein line about cluttered desks and cluttered minds, which may have been a weekly recap referencing one of those posts.

Want me to pull up the full content of any of these?

I told Claude it didn’t need to do that. I’ll just link to them here instead.

1. An embedded search is a way to find things by meaning as opposed to just pattern matching. ︎
2. MCP (Model Context Protocol) is a way of exposing ark commands to an LLM like Claude. The LLM can then use my tool to answer questions. ︎
3. I use LM Studio for hosting local models, ark uses llama-3.2-3b-instruct for local text-based LLM activity, and it uses qwen3-vl-8b for vision (getting descriptions from photos, documents, etc.) ︎

I pulled out the very first diary, an Avery Record Book which I got in an office supply store in Sherman Oaks, California. It’s a slim volume because I was counting my pennies back then. I turned to the first page and read what I had written.

The very first thing I mentioned in that first entry was that I had finished read I. Asimov, Isaac Asimov’s posthumously published memoir (a retrospective follow up to his massive 1980 double-volume autobiography). This was my 3rd time reading the book. I also mentioned starting to read Robert Heinlein’s Double Star, which it turned became one of my favorite Heinlein books. I mentioned seeing the movie Babe. And I concluded the entry with a renewed desire to write science fiction stories and, someday, see them published.

I have been working slowly over the last year or so to transcribe and digitize these diaries as part of my personal digital archive. The archive now contains digitized schoolwork and artwork going back to kindergarten, and email going back to October 18, 1994—even before my diary. Indeed, as of the this morning, the archive contains more than a quarter of a million messages. By the time my diary is fully digitized, I’d expect there to be about 10,000 entires. I also have more than 7,000 blog posts archived there a well, making the archive a comprehensive collection of my output during my lifetime.

There is another coincidence today: The Artemis II / Orion spacecraft and crew will surpass Apollo 13’s distance record from Earth later today, and head behind the moon, the first time humans have done so in 54 years. I’ve been following the mission assiduously, and I can’t help but think that Isaac Asimov would have been delighted by this mission. Even more incredible than 30 years of diary entries is that fact that once again, we’ve gone to the moon.

The new typewriter arrived yesterday and what a joy it is!

Admittedly, I originally set out to get a manual typewriter, but those seemed much harder to come by than a new Royal. Besides, I wanted a typewriter that was in working condition, not for show. As it turns out, the model that I got has some advanced features, including a screen that you can see what you’re typing before it types it on the page so you can make corrections. I have avoided this feature so far. After spending about an hour learning the basics, I obtained a set of blank postcards, and set about writing half a dozen notes to friends and family on my new machine. I made two surprising discoveries:

1. I am not nearly as good a typist as I think I am when editing capabilities and auto-correct are taken out of the picture. I type fast, but I make a typo every 3 or 4 lines of text. This discovery is a blessing in disguise: by using the typewriter, I think I can make myself a better typist.
2. When I get into a rhythm, with the staccato RAT-TAT-TAT of my typing as background music, I get what I only describe as an endorphin rush that spurs me on, and is like nothing I’ve ever felt typing on a computer keyboard and watching the words appear on a screen.
   

There are a few useful features that I’ve already made use of. The device has a whopping 16K of text memory, which can be used to store text files, or format settings. I’ve used it for the latter, setting up my margins and tab for standard letter paper, as well as 4×6 postcards. This makes it convenient to easily switch formatting settings based on my task. The typewriter can produce underlined text, bold text, and 3 different text pitches. It can auto-center a line, and right-justify a line. It has word correction features that make use of the correction tape, but I haven’t played around with that yet, preferring the old-fashioned X-ing out of my mistakes. This gives me visual cues of my progress over time.

I’m excited to put my new toy to more use. The feeling I got when I used it yesterday was just wonderful. As I sit here typing these words into my laptop, I am imagining typing them into my new typewriter. It is not the same.

— Arlington, Virginia

1. I have a manual Royal typewriter circa 1950, which I inherited from my grandfather, but it is no longer in working condition. ︎
2. Someone will note that I could also simply write longhand and at a significantly lower cost. The fact is I do write longhand, every day, in my diaries. That gives my hand and wrist enough of a workout. ︎