Field notes on the Web

Hyperlinks vs. the web

2025-08-18T11:16:00.002-04:00

In the previous post on the 1968 Mother of all Demos by Doug Engelbart and company, I mentioned stumbling on the fact that Andries van Dam and Ted Nelson had put together HES, which is generally regarded as the first hypertext system, the year before.

All the parts of that were familiar. I've written extensively, though not always favorably, about Nelson's later project, Xanadu. Foley, van Dam et. al.'s Computer Graphics: Principles and Practice was on the shelf at the first place I coded for money, and I'm pretty sure I've run across the idea of hypertext at some point over the years. What I hadn't realized was that Nelson and van Dam had worked together, and that hypertext went back quite that far.

To be fair, I wasn't exactly shocked that hypertext dated to 1967, particularly in the context of Engelbart & co.'s demo, which included what were recognizably hyperlinks. What did catch my attention was that all the building blocks of Web 1.0 were in place in 1968, twenty-three years before the first actual web servers appeared in 1991:

the concept of hyperlinks and hypertext
the realization of that concept in running code
connectivity between computers in different physical locations (ARPANET itself would come along a bit later, but computers were already talking to each other)
interactive graphic displays
the mouse

The bullet point I didn't quite add is file servers, but FTP dates to 1971. I haven't dug up solid evidence that there was such a thing as a file server in 1968, but I certainly wouldn't bet against it. If you prefer to put the "Web 1.0 could just as well have happened" point at 1971 and say it was only twenty years later that it actually happened, fine.

What took so long? According to van Dam's account, the people who saw the Mother of all Demos were generally impressed, but the overall reaction was to say "Wow, that was some demo" and then get back to work. Was this a failure of the imagination, that the computer researchers of the time couldn't wrap their heads around something that wasn't 80-column punched cards, even if they'd seen it with their own eyes?

Possibly, but there were also more mundane concerns. That list of pieces in place comes with a few disclaimers:

Connectivity was generally at 1200 Baud, or approximately one millionth of a gigabit per second. This will deliver text faster than you can read it, but it amounts to a megabyte every two minutes. You can actually fit quite a bit of information into a megabyte, and 1.5Mb/s T1 lines were available, (for a hefty charge, generally to institutions or large corporations), but you're not going to run YouTube or Netflix on the bandwidth available at the time.
Interactive graphic displays were a thing, but they were normally vector-based (think Asteroids, if you've heard of that) and in any case they were expensive specialized equipment. Graphic displays (bitmapped) didn't become commonplace until the 80s. Even then they weren't cheap and they looked absolutely primitive by today's standards

I think it's fair to say that in 1968 you could have put something together that looked quite a bit like Web 1.0, but it would have been a curiosity: slow, expensive and without anywhere near enough content to make for a compelling experience.

The first actual web servers were intended as an easier way to get to the content that had accumulated on various FTP/Gopher/UUCP/... servers over the years, which was getting to the point where it was hard to just know where something you were interested in was located. Not too long after that (AltaVista came along in 1995), there were enough index pages that it was hard to keep track of where a good index for what you were looking for was, much less the data itself, and web search was born.

So it wasn't simply a matter of the world turning its back on the wonderful potential of Engelbart's NES and Nelson and van Dam's HES and then suddenly waking up in 1991 to realize what they should have known all along. Things were happening in the meantime:

Computing power, storage and bandwidth were increasing exponentially (as in, actually exponentially, at a more-or-less constant proportion per unit time, and not just "by a lot")
As both a driving cause and an effect, the number of people with access to computing power, and the amount of data they wanted stored, also increased exponentially
People continued to experiment with ways of organizing information and navigating complex webs of connections (1987's HyperCard comes to mind, but it's not the only example)

I think this is a common thread in technology in general: Things take a while to develop. It's not enough just to have a good concept, or even a working realization of it. You generally need a certain amount of infrastructure and a growing need that your new concept will meet, and some partial successes along the way.

Some other examples that come to mind:

SketchPad, which anticipated several major developments, particularly the Graphical User Interface, was written in 1963, GUIs weren't really widespread until the 1980s
The object-oriented language Simula came out in 1962 and SmallTalk in 1972. Objective-C was introduced in 1984, but OO languages didn't really get major traction until the mid 1990s
The concept of a neural network dates back to 1943, at least. Perceptrons were introduced in 1969. Hopfield networks were a topic of research in the 1980s. Transformers were proposed in 2017, now eight years ago. ChatGPT came out five years later.

It shouldn't be surprising that concepts run ahead of implementation. Concepts are a lot easier to come up with. More interesting is how we get from (some) concepts to widespread implementation, and which concepts get there.

The future still isn't what it used to be: Doug Engelbart

2025-08-17T10:37:00.008-04:00

If you're exploring the origins of the web and pondering how early ideas ended up incorporated -- or not -- into what we see now, sooner or later you're going to run into "The Mother of all Demos", a 1968 presentation by Doug Engelbart and company demonstrating the oN-Line System, or NLS for short, produced by the Augmentation Research Center at the Stanford Research Institute.

I watched it a few months ago. Having somehow missed it for all these years, I went in cold and -- deliberately -- without researching what I was supposed to be looking at. I took copious notes, in some cases replaying to try to make sure I got things right ... and then got distracted by other things. What follows is not fresh and detailed, but I want to at least include something about the demo, so here goes ...

NLS was directly inspired by Vannevar Bush's Memex concept, so if nothing else it's an interesting case study in what happens when you try to turn someone's architectural vision into a real system.

Everything's in glorious black and white, the text on the screens is UPPERCASE and clunky, with actual capital letters indicated by an OVERBAR, and some of the participants look more than a bit like the 1980s stereotype of nerds, because that's where the 1980s stereotype comes from. Actual 1980s nerds were hopelessly uncool in completely different ways (so I hear).

On the other hand, Engelbart is wearing a headset mic, moving a cursor (the team called it a "bug") around on the screen using a mouse, pointing, clicking, narrating the whole way through, linking up with colleagues in a different county in real time, cross-fading and compositing live video and generally giving the appearance of an ordinary livestream, just in black-and-white and what now qualifies as period costume, and without a discernible boss level.

About 20 minutes in, Engelbart says "I'm going to do something called 'jump on a link'" and explains that a link points to a particular location in a particular file with a particular "view type" saying what kind of content you're looking at -- yeah, that's hypertext.

Physically, NLS is nothing like Bush's original description of a desk-shaped piece of furniture holding a library of microfilm and some sort of magnetic recording medium. More than twenty years had passed since Bush's original article and whole new technologies had developed in the interim, particularly in computing hardware.

NLS was running on a 24-bit SDS-940, clocked at somewhere on the order of 0.1MHz, with approximately 200KB of main memory, 5MB of swap space and 100MB of disk(-ish) storage. It had a text and graphic-capable display and a 1200-baud modem for connectivity. It also ran QED (Quick EDitor), the very first text editor I learned to use (a bit later, though), and a direct ancestor of vi, which I still occasionally use.

Tiny as this might seem today, I'd argue that there's a bigger gulf between it and 1945's ENIAC than between it and equivalent commercially-available equipment from 1991 (as many years from NLS as NLS was from ENIAC): say a farm of a couple dozen 80MHz SPARCstation 2s, each with 128MB of ram and a few GB of disk. Yes, the SPARCstations could do a lot more, but the SDS has essentially the same pieces: Enough RAM to be meaningfully programmable (as opposed to requiring switches to be set manually), offline storage, connectivity^*, a graphical display, the ability to connect a variety of peripherals and so forth, which the ENIAC had essentially none of. The hardware available when NLS was developed was essentially a smaller, slower version of the hardware that Web 1.0 was built on. Nothing of the sort was available when Bush was thinking up Memex.

NLS is clearly recognizable, almost 60 years later, for what it is: a computer system. Memex, just as clearly, is a thought experiment, though one with enough detail to make it the next best thing to a demo.

To be sure, even if the difference between NLS hardware and today's is mainly a matter of quantity, the quantities have changed by a lot. Today's hardware is several orders of magnitude beyond the SDS, and there are several orders of magnitude more computers in the world now, and they are connected by networks several orders of magnitude faster than 1200 baud. That all has to make a difference, and it has. I hope to dig into this a little more deeply at some point.

There is another key difference between NLS and Memex. Memex is intended for a single person working alone. NLS is inherently multiuser and connected. While the actual demo takes place in San Francisco, the system itself is running in Menlo Park, about 50km/30mi away and probably about as much of a pain to drive to and from then as now. After a fair bit of introductory material, Engelbart is joined by Bill (English, or possibly Paxton?) in Menlo Park, and the two proceed to edit shared documents together.

In the post on Memex, I argued that this difference is one of the main reasons that as far as I'm concerned Bush did not invent the Web. I wouldn't say that Engelbart's team invented the Web, either, but just as the SDS-940 has much more in common with modern computers than with ENIAC, NLS has much more in common with the modern Web than with Memex.

Along with demonstrating the NLS hardware and software, Engelbart made a point of discussing the project that produced it and the approach behind it. That approach was

Empirical: Try things out and see what works
Evolutionary ("steepest ascent"): You can't see everything in advance, so take a heuristic approach and look for highest-impact changes at every point
Whole system: While different people had different skills and responsibilities, the project itself encompassed hardware, software and anything else needed to produce a working system
Bootstrapping (later called "eating you're own dogfood"): The people developing the system used the system. Past a certain early point, further development of the system was done in the system itself.

"It's a struggle doing it that way," Engelbart said, "but it's beginning to pay off."

I get the feeling both parts of that were knowing understatements.

What we now call dogfooding is a powerful approach that provides a constant reality check, but with the pitfall that it can skew toward the researcher's use cases. A system that's useful for developing software might or might not be useful for other things. Engelbart appears to have been aware of this hazard. One of his prominent examples is his own shopping/errand list. With its repetitions and loose categorization it looks natural. Maybe it wasn't his real list, but it feels like it could have been.

Besides making the system more relatable by giving an example that the audience would be familiar with, constructing a loosely-organized shopping/errand list on the fly demonstrates the system's ability to deal flexibly with free-form content.

This wasn't a given. I could easily imagine someone trying to demo a more rigidly structured system: "OK, now first I create a shopping list template ... Oh, I forgot about vegetables. Let me go back and add a vegetables section to the template. Now where was I?" as the audience's eyes glaze over. Engelbart just puts a list together. It doesn't hurt that he's a fairly engaging speaker and is intimately familiar with the system, but even taking that into account, it doesn't seem like the system is putting arbitrary barriers in his way.

That's not to say that using NLS would be simple for someone encountering it for the first time. After a while, you come to understand that Engelbart and company have memorized quite a few one-letter commands and established quite a few conventions to help find their way around. This is a classic ease-of-use vs. ease-of-learning tradeoff (it's not always a tradeoff, but I'll leave it at that before I start mumbling things about Pareto optimality).

There are some high-level goals that read a lot like what we would now call mission statements:

Improve the effectiveness with which individuals work at intellectual tasks
as a subgoal: better use of human capabilities
Develop a system-oriented discipline for designing the means by which greater effectiveness is achieved

Again, Engelbart makes a point of calling that last point out. Just as important as producing a usable product is producing a way of developing such products.

NLS is meant to be "an instrument/vehicle helping humans to operate within the domain of complex information structures". While it's not explicit in this statement, the demo makes clear that, unlike Memex, NLS is intended for collaboration. It's also subtly larger in that it aims at "humans" rather than "men of science".

Engelbart puts forth several principles that sound surprisingly modern:

"content represents concepts" Note the use of "content". It didn't start life as a marketing term.
"structure represents relationships, or human-thought product"
said structures are "too complex for direct human study", that is, the computer is taking on some of the weight of dealing with a complex structure
there is a "control metalanguage" note use of "meta". Again, not a new term (well, it goes back to Aristotle's Metaphysics at least).

So it's 1968, Doug Engelbart has just demonstrated hypertext, computers are starting to get networked together, a bunch of standards ending in "TP" for "Transport Protocol" are about to be developed (FTP is announced in 1971) ... so we can expect the Web as We Know It to appear sometime around ... 1991? Wait, what? That's like, 20 years later.

Audience member Andries (Andy) van Dam's assessment of the impact of the demo was that computing mostly went on unchanged afterwards. Besides writing one of the most important early textbooks in computer graphics, van Dam, along with Xanadu founder Ted Nelson -- and I only just now learned this while researching this post -- developed HES, the Hypertext Editing System, at Brown University in 1967. As far as I can tell this was independent of the NLS work, though clearly the two teams would have known about each other.

Van Dam's assessment notwithstanding, several NLS alumni ended up at Xerox PARC (Palo Alto Research Center), which produced, among other things, the influential object-oriented language Smalltalk and a bitmapped mouse-driven GUI ... and these went nowhere until, so the story goes, Steve Jobs stopped by and basically grabbed the whole concept for the Apple Lisa.

The Lisa also went nowhere for a while, but was later reworked to the Macintosh, which eventually took off. I had the fortune of stumbling across a demo of a Lisa at school. This being a tech school, the hardware-oriented folks talked the Apple rep into taking the cover off so they could poke around. The Apple rep, though visibly nervous, was generally a good sport. But I digress.

The famous Macintosh ad was in 1984, 16 years after Engelbart's demo, itself 18 years after the project was initiated. We'd like to think we move much faster now, but do we, really, once you measure comparable points of development: someone's initial concept, start of project, first major demo, first real product, with error bars representing variability in how long things take?. I hope to dig a bit deeper into that as well.

*Though NLS used its modems to allow people to connect to it, computers were already connecting to each other. The ARPANET project was already underway and would be live in a year or two.

The future still isn't what it used to be: This whole "web" thing

2025-01-07T12:35:00.014-05:00

It's worth keeping in mind that the building blocks of today's web, particularly HTTP and HTML, were developed by academic researchers. One thing that academic researchers have to do, a lot, is follow references, because most academic work builds on, critiques and otherwise refers to existing work.

Let's take a moment to appreciate what that meant before the web came along. Suppose you're reading a reference work and you run across a reference like this:

4. Ibid, 28-3

That is, you've just run across a passage like this totally made-up example:

It is also known that the shells of tropical snails vary widely in their patterning⁴.

That little raised 4 tells you to skip to the bottom of the page and find footnote 4, which says "Ibid, 28-3", which means "look in the same place as the previous footnote, on pages 28 through 33". So you scan up through the footnotes and find

3. Ibid, 17

OK, fine ... keep going

2. McBiologistface, Notes on Tropical Snails, 12-4

OK, this is the something previously referenced, in particular something written by McBiologistface (likely the eminent Biologist McBiologistface, but perhaps the lesser-known relative General Scientist McBiologistface). Keep going ...

1. McBiologistface, Something Else About Tropical Snails, 254

OK, looks like this person wrote at least two books on tropical snails. The one we're looking for must be referenced in a previous chapter [it should also be listed in the References section at the end --D.H. Mar 2026]. Ah, here it is:

7. McBiologistface, Biologist, Notes on Tropical Snails (Hoople: University of Southern North Dakota at Hoople Press, 1945), 32-5

Great. Now we know which McBiologistFace it was, and which edition of which book published by which publisher. Now all we have to do is track down a copy of that book, and open it to ... let's see, what was that original reference? ... oh yes, page 28. The publisher and publication date are at least potentially important since different editions might have different page numberings, or even different content.

To be fair, "McBiologistface, Notes on Tropical Snails" from reference 2 is probably enough to find the book in the card catalog at the library, and if a reference is "Ibid", you may already have the book and have it open from following a previous reference to it. It's also quite possible that your department or office has copies of many of the books and journals that are likely to be referenced.

Nonetheless, thinking of the tasks I mentioned when describing the Olden Days -- navigating an unfamiliar place, communicating by phone, streaming entertainment and searching up information -- simply following a reference from one book or article to another could be more work than any of them.

Even answering a question like "where was the Touch-Tone™ phone invented" would have been easier, assuming you didn't already have a copy of Notes on Tropical Snails on hand: go to the library, walk right to the easily-located reference section that you've already been to, pull out the 'T' volume of one of the encyclopedias, flip to Telephone and chances are your answer is right there (or you could just ask someone who would know).

To find the reference on snails, you'll have to look up the book in the card catalog, note down its location in the stacks, go there and scan through the books on those shelves until you find the book itself (and then open it and flip to the right page, but you already know that from the reference). This is all assuming there's a copy of the book on the shelves that no one's checked out (who knows, maybe there's been a sudden interest in tropical snails in your town). Otherwise, you could call around to the local bookstores, or your colleagues and friends, to see if anyone has a copy. If not, your favorite bookstore could special-order a copy from the publisher, and with luck it would be there in a few days (but maybe much longer).

Chasing a link in an HTML document is more or less instant. You can probably see the appeal.

My point here is that the interlinked nature of the web, that ability to click on a link, immediately see what's on the other end and easily get back to where you were, was an absolute game-changer for the sort of people who created the early web. Your own milage may vary.

To make this work, you need a few key pieces

A way of referencing data that's available on the network (URLs)
A way of embedding URLs in a body of text, similar to the way footnotes are embedded in ordinary text (HTML)
Ideally, a standard way of accessing something referenced by a URL (HTTP)

I say "ideally", because it was already possible to access data on the web using protocols like FTP and Gopher, and you could reference those with a URL. Nonetheless, having a integrated suite of {URL, HTML, HTTP} working together fairly seamlessly meant that http:// URLs (or later, https://) quickly came to dominate.

You also need one more thing, namely that there should actually be something on the other end of the link (it's OK if links are sometimes dangling or become broken, but that should be the fairly rare exception). By the time the web standards were developed, there was already enough interesting data and text on the internet to make links useful. To some extent, the early web was just an easier way to get at this kind of information. If you had the pieces, you could easily pull together an HTML page with a collection of links to useful stuff on your server, stuff like interesting files you could fetch via FTP, with a little bit of text to explain what was going on, and anyone else could use that.

The truly webby part of the web, the network of links between documents, is still around, of course, but as far as I can tell it's not a particularly important part of most people's web experience. Links are more a way of getting to content -- follow a link from a search result, or follow a reference from an AI-generated summary to see whether the AI knows what it's talking about -- but following links between pieces of content is not a particularly important part of the web experience. Some articles include carefully selected links to other material, but a lot don't. Personally, I've mostly stopped doing it so much, because it's time-consuming, though these recent Field Notes posts have a lot more linkage than usual.

One sort of link that I do follow quite a bit is the "related article" link in a magazine or news source -- articles by the same author or on the same topic, or just to stuff that their server thinks you might find interesting, or that the publisher is trying to promote. But again, this seems more like navigating to something. The articles themselves largely stand alone, and I generally finish one article before moving on to the next. A truly webby link, like a footnote before it, links from some specific piece of text to something that's directly related to it.

And, of course, I do click on ad links, though usually by mistake since you just can't get away from them.

Realizing this, I think, is a big reason that this blog went mostly quiet for a couple of years. If the webby part of the web is really only of interest to a few people, except in a few special cases like sharing social media content and browsing Wikipedia, why write field notes about it, particularly if the blog writer doesn't find social media particularly appealing?

Conversely, this latest spate of posts is largely the result of relaxing a bit about what the "web" is and talking about ... dunno, maybe "the online experience" in general? Or just "internet-related stuff that doesn't really seem to fit on the other blog?"

Whatever you call it, I seem to be enjoying writing about it again.

[Naturally, this is the last post for several months, followed by another (so far) months-long gap. Still enjoying the writing, but ... life --D.H. 2026]

The future still isn't what it used to be: Vannevar Bush

2025-01-06T21:38:00.008-05:00

(According to Blogger, this is the 700th post on this blog, which seems like a completely arbitrary milestone to note, but I noticed it nonetheless, so now you get to. You're welcome.)

Vannevar Bush casts something of a long shadow. He held several high-level technology-related posts in the FDR and Truman administrations, had a long and distinguished academic career at MIT and elsewhere, and won several prestigious awards, including the National Medal of Science. His students included Claude Shannon, whose work in information theory is still directly relevant, and Frederick Terman, who was influential in the development of what we now call Silicon Valley (I used to work fairly near Terman Drive in Palo Alto).

Bush is also often credited with anticipating the World-Wide Web in his Atlantic Monthly article As We May Think. Since I've been comparing early visions of the Web with what actually happened, I thought I'd take a look. I've linked to the ACM version rather than the Atlantic's version, which may or may not even be online, since the ACM version highlights the relevant passages. Though there's a Wikipedia page on the piece, I've deliberately skipped it in favor of Bush's original text (with the ACM's highlights).

Two things jump out immediately, neither directly relevant to the web:

The language is relentlessly gendered. Men do science. Girls [sic] sit in front of keyboards typing in data for men of science to use in their work. A mathematician is a particular kind of man, technology has improved man's life, and so forth. Yes, this is 1945, and we expect a certain amount of this, but from what I can tell Bush's style stands out even for the time. I mention this mainly as a heads-up for anyone who wants to go back and read the original piece -- which I do nonetheless recommend.
There is an awful lot of technical detail about technologies that would be obsolete within a couple of decades, and in several cases nearly fossilized by the dawn of the Internet in the 1970s. Bush speculates in detail about microphotography, facsimile machines, punch cards, analog computers, vacuum tubes, photocells and on and on for pages. Yes, all of these still existed in the 1970s (I spent many an hour browsing old newspapers and magazines on microfilm as a kid), but digital technology would make most if not all of them irrelevant before much longer. As far as predicting the technology underpinning the web, Bush's record is nearly perfect: If he speculated about it, it almost certainly isn't relevant to today's web.

Two thoughts on this. First, it's almost impossible to speculate about the future without mentioning at least something that will be hopelessly out of date by the time that future arrives. In our own time, all we have are the tools and mental models of the world of that time. I don't fault Bush for thinking about the future in terms of photographic storage, and I don't this takes anything away from his thoughts on the "Memex", which is what people are referring to when they talk about Bush anticipating the web.

I just wish he hadn't done nearly so much of it. Alan Turing's Computing Machinery and Intelligence spends two sentences on the idea of using a teleprinter so that it's not obvious whether there's a human or machine on the other end of the conversation, and one of those sentences just says that this is only one possible approach. That seems about right for that paper. In Bush's case, I could see a few paragraphs about how to store large amounts of information (for those days, at least) on film or magnetic media, and so forth. The article would have been much shorter, but no less interesting.

Second it's worth noting how many things were possible with mid 1900s technology. You could convert, both ways, between sound, image and video (in the sense of moving images) on the one hand and electrical signals on the other. You could store electrical signals magnetically. You could communicate them over a distance. You could store digital information in a variety of forms, including the famous punched cards, but also magnetically.

There were ways to produce synthesized speech and read printed text. Selecting machines could do boolean queries on data (Bush gives the example of "all employees who live in Trenton and know Spanish"). Telephone switching networks could connect any of millions of phones to any other in about the time it took to dial (and less time than it sometimes takes my phone to set up a call using my WiFi). Logic gates existed. For that matter, the first general-purpose digital computer, the ENIAC, existed in 1945 and Bush would certainly have known about its development.

In other words, even in 1945, Bush isn't drawing on a blank canvas. He's trying to pull existing pieces of technology together in a new way in order to deal with what was, even at the time, an overwhelming surplus of information. The gist of the argument is "If we make these existing technologies smaller, faster and cheaper, and put them together in this particular way, we can make it easier to deal with all this information."

The particular problem Bush is really interested in isn't so much storing information as retrieving it ("selecting" as Bush says). This is totally understandable for a national science adviser who had until recently been working on one of the largest technological efforts to date (the Manhattan Project). Bush cites Gregor Mendel's work having been essentially unknown until decades after the fact as just one example of a significant advance nearly being lost because no one knew about it, even though it was there to be found. Bush's desire to prevent this sort of thing in the future is palpable.

Bush mentions traditional indexing systems that can find items by successively narrowing down the search space (everything starting with 'F', everything within that with second letter 'i' ... ah, here it is, Field Notes on the Web), but he's much more interested in following a trail of connections from one document to another. That is, he's envisioning a vast collection of documents traversable by following links between them. That's the world-wide web. Ok, we're done.

Except ...

Bush sees the Memex as literally a piece of furniture, looking pretty much like a desk but with a keyboard attached along with various projection screens and a few other attachments. Inside it is a store of microfilmed documents together with some writable film, which takes up a small portion of the space under the desk, and a whole bunch of machinery to be named later, taking up most of the space.

Associated with each document is a writable area containing some number of code spaces, each of which can hold the index code of a document. There's also a top-level code book to get you started, and when you add a new document, you add it to the code book. To be honest, this seems a bit tedious.

To link two documents together, you pull them both up, one on one projection screen and the other on the other, and press a button. This writes the index code for each document in the other's next open code space. The next time you pull up either of the documents, you can select a code space and pull up the document with that code.

Codes are meant to have two parts: a human-readable text code and a "positional" numeric code (probably binary or maybe decimal). Linking this post to Bush's article might add "Bush-as-we-may-think" to a code space for this post, along with (somewhere offscreen) the numeric index for Bush's article, and "Field-notes-future-ramblings-Bush" to a code space on Bush's article (along with the numeric code for this post). At that point you've got one link in a presumably much larger web. Actually, you have two links, or one-bidirectional link if you prefer. Not quite Xanadu's transclusion, but arguably closer than what we actually have.

Pretty webby, except ... coupla things ...

For one thing, this is all happening on my Memex. My copy of this post is linked with my copy of Bush's article. Yours remains untouched. If there's a way of copying either content or links from one Memex to another, I didn't catch it. Bush's description of how document linking works is hand-wavy enough that it wouldn't be particularly more hand-wavy to talk about a syncing mechanism (and/or an update mechanism), but I doubt Bush was thinking in that direction.

Bush seems to be thinking more about a memory aid for an individual person (or possibly a household or small office/laboratory). Functionally, it's a personal library with much larger capacity and the ability to leave trails among documents. It's certainly an interesting idea, but it misses the "world-wide" part. When I link to the ACM's version of Bush's paper, the link is from my blog to the ACM's site. If you write something and link it to Bush's paper, we're pointing at the same thing, not separate copies of it, and we're pointing to a thing that might be stored anywhere in the world (and someplace else next time we access it).

In the same post I mentioned above, I talk about a couple of features that make the web the web, particularly that a link can be dangling -- pointing to nothing -- and it can become broken -- you pointed at a page, but that page is no longer there (early posts on this blog are full of these, though at the time it wasn't clear whether rotting links would be an issue as storage got cheaper; they are). There's also some ambiguity as to what exactly a link is pointing to. If I point to the front page of a news site, for example, the contents on the other end of that link will probably be different tomorrow. In other cases, it's worth going to some effort to ensure the contents don't change significantly.

These may seem like bugs at first glance, but for the most part, they're features, because the flexibility they provide allows the web to be decoupled. I can do what I like with my site without caring or even knowing what links to it. Since a Memex is a closed system, none of this really applies. On the one hand, it's not a problem, but on the other hand, it's not a problem because a Memex is not a distributed system, which the web as we know it very much is.

Finally, the mechanism of linking is noticeably different from what HTML does. You have a pair of links between documents (or maybe pages within documents, given that that's what's on the screen when you press the "link" button?). An HTML link is between a particular piece of the source document to, in the general case, a particular anchor on the destination document. To be fair, this doesn't seem like an essential difference. You could imagine a Memex with a linking mechanism that goes from a piece of one document to a piece of another, which would be much more like an HTML link (and, arguably, more like a Xanadu transclusion).

[Really, though, an HTML link points to whatever the server on the other end serves up in response to that particular URL. The resulting page is often, one way or another, maintaining live connections to any number of other servers and updating its appearance accordingly. In writing the post, I implicitly assumed that everything was static text, since that was the world Bush was dealing in. The dynamic nature of real web resources is a whole separate dimension. The point here is that even without that Memex isn't really the Web -- D.H. April 2025]

So did Vannevar Bush anticipate the web by nearly half a century?

I think the fair answer is "not really", because the distributed, dynamic nature of the web is critical.

Did he anticipate the idea of an interconnected web of documents? I think the fair answer is "sorta". Again, actual web links are one-directional and non-intrusive. You can link from document A to document B without doing anything at all to document B or its associated metadata. You don't need a backlink and you generally won't have one.

This one-way form of link was not a new idea. Documents have been referencing each other forever. Bush's notion of linking is different from an HTML link, and since an HTML link is structurally the same as a reference in a footnote in a book, it's different from that as well.

In other words, the original idea in Bush's work is more an evolutionary dead end than an innovation. A pretty interesting dead end, but a dead end just the same.

Postscript:

There's one more thing that I'd been meaning to mention but, embarrassingly enough, forgot to: search. Bush is quite right in saying that people access information by content, but in the Memex world everything eventually boils down to an index number. You access document 12345, not "any documents mentioning Memex" or whatever.

Search is probably the aspect of the web with the least precedent in mid-1900s technology. There were ways to attach index numbers to things, or even content tags, and retrieve them, with a minimum of human intervention. Bush goes into those at length. But if you wanted to get to something by what was in it, you needed a person for that, if only to add indexing information. Indeed, Memex is aimed directly at making it easier for a human to do that task, by making it easy to leave a trail of breadcrumbs a human could easily follow.

It would be almost a half-century before documents could be easily accessed by way of what was in them.

Oh, and also ... in Bush's vision, linking documents together would be a frequent activity for anyone using a Memex. In today's web, not so much, except, I think, in the particular case of re-whatevering a piece of social media content. I think the reason for that is also search (see this early post for a take on that).

The future still isn't what it used to be: Cyberspace

2025-01-05T19:28:00.008-05:00

In the previous post, I said

Telecommuting and remote work exist, but they don't dominate, they only really make sense for some professions and they don't mean jacking into a Snow Crash or Neuromancer virtual world, even though one of the largest corporations in the world has rebranded itself around exactly that vision.

This very morning, I decided to add David Foster Wallace's Infinite Jest to my reading list [not really my jam, but I wanted to see what all the fuss was about -- D.H. Apr 2025]. In the preface to the 20th anniversary edition (in 2015), Tom Bissell writes

Yes, William Gibson and Neal Stephenson may have gotten there first with Neuromancer and Snow Crash, whose Matrix and Metaverse, respectively, more accurately surmised what the internet would look and feel like.

Um, did they? Bissell goes on to say

(Wallace, among other things, failed to anticipate the break from cartridge- and disc- based entertainment)

Fair, but ...

Yes, there is a major difference between on-demand streaming and broadcast streaming, where a broadcaster puts out content according to its schedule. There is also a difference, though it seems like a smaller one, between obtaining a physical object that allows you to view something when you want to and being able to view something more or less instantly via an always-on connection (using "view" in a fairly general sense here that would include listening to audio).

Having the combination of "what you want" and "when you want it" without the friction of obtaining a physical artifact like a book, record, tape or disk does seem like something new and significant (more musings on that here), so in that sense, to the extent Wallace's world is limited to physical media which somehow include DRM that won't work in the real world, it's farther from our reality than one with data flowing freely over networks.

With one exception, though (which I'll get to) the modern web/internet that I'm familiar with has little to do with Neuromancer's matrix or Snow Crash's metaverse either.

Let's start with how you get there (one small disclaimer: While I finally got around to reading Snow Crash a couple of years ago, the last time I read Neuromancer was, um, closer to when it came out, so I'm relying on fairly old memories plus secondary sources for that one; for reference, Neuromancer was published in 1984, Snow Crash nearly a decade later in 1992, Infinite Jest in 1995).

You get to Gibson's cyberspace by jacking in, that is, connecting your central nervous system to a computer interface that delivers a completely immersive experience. To access Stephenson's metaverse, you need a terminal and googles, either a high-quality private terminal or a free public one which provides only a grainy, black-and-white experience. In either case, the experience in Snow Crash is immersive in that you are generally not aware of the outside world, but it's not the full-sensory experience of Neuromancer.

Back in our world, of course, people generally access the web through their own computing devices, whether a phone, a tablet, a TV set, a laptop or even a desktop computer. There is no scarcity of devices. If you have access to any at all, you probably have easy access to several. You can even visit a public library and use a computer there. You do need an internet connection, but those are nearly everywhere, too. You can get on the internet in a cafe, for example, by connecting to their WiFi (as far as I can tell, actual internet cafes are nearly extinct).

In most cases, you're aware of the world around you, or at least, the internet experience doesn't take over your entire sensorium. The semi-exception is gaming, which in some cases makes an effort to be truly immersive, more or less along the lines of Snow Crash. VR headsets have been around in some form since the 80s (if not before), and they're a natural fit for applications like FPS games, so this is not exactly a surprise.

Long story short, in much of the world the internet is easy to access with readily available equipment. Going online often means using your phone or watching TV, that is, using something that's recognizably derived from a technology that existed before the internet. Immersive experiences are only a bit harder to get to, but in any case they're not the norm.

In Neuromancer, jacking in requires special equipment on both the human and computer end (though Gibson does speak elsewhere of billions of people having access). The bar is lower in Snow Crash, but it's not something that most people spend much time on. It's interesting that the 1992 version is a bit more mundane than the 1984 version, almost as though computing in the real world had become more commonplace. It's also telling, I think, that access to the virtual worlds of the novels is difficult enough to hang a plot point on, particularly in Gibson's earlier version, almost as though stories were written by writers.

OK, once you're in the virtual world, how do you get around? I'll focus more on Snow Crash here, mainly because memories are fresher. The key point about Cyberspace is that it's a space. In particular, it's a three-dimensional construct centered around a 100-meter-wide road 2¹⁶ (65,536) kilometers long following a great circle on a virtual sphere.

If you want to meet with someone else online, you arrange to go to the same space by moving your avatars. You can move your avatar around by walking or running, or use a vehicle, or take the transit system, which has 256 express ports, with 256 local ports in between each, at one kilometer intervals. There are special spaces within the metaverse, many with restricted access.

From an immersive gaming perspective, this makes perfect sense. From the perspective of the web, it makes no sense at all. If you chase a link from here to the Wikipedia article on Snow Crash, you just go. This page goes away and you see the Wikipedia page. Or it opens in a separate tab and you can flip back and forth, or whatever. You don't do anything even metaphorically like moving from this page to that. There's no concept of distance. At worst, one or the other of the pages might load slowly, but you don't have a sense of motion while that's happening (well, I don't, at least).

In other words, the key feature of Cyberspace, that it's a space, is at best completely irrelevant to the modern web, and at worst it's actually in the way. As I recall, Gibson's matrix is similar. For example, if you encounter ICE (Intrusion Countermeasures Electronics) you see an actual wall of ice or some other material that you have to get through.

Gibson's matrix, at least, is also spatial in another way: its contents are tied to physical computers in the real world. In particular, the two AIs Wintermute and Neuromancer are physically located in Bern and Rio de Janeiro, respectively. That is, they are presumably running on hardware located in those cities. Wintermute would like to be able to join with Neuromancer, its other half (Neuromancer is less concerned about this).

Data in today's internet is much more distributed. Not everything is in the cloud in the sense that there's no single well-defined physical location for data or the processors that process it, but a lot is, and even when a service or database is single-homed in a particular place, it usually doesn't matter exactly where that is. Even if two servers are located on different continents, they can still communicate easily because of the internet.

In the end, the technology of Neuromancer and Snow Crash isn't particularly prescient. The parts that are still around, such as a data-carrying network that's accessible across the world, or an immersive VR, were already under development in the 1980s. Gibson and Stephenson were drawing on cool and experimental, but real, technology as a jumping-off point for fiction. Moreover, they also copied some of the limitations of the technology of the time, particularly the need for specialized access terminals and on services being hosted on particular equipment located in particular places.

But in the end, Neuromancer and Snow Crash are not really about the technology. Snow Crash is more an exploration of Anarcho-Capitalism in a world where the official government has collapsed and ceded power to a collection of private entities. Neuromancer is in large part a conventional thriller, even including a physical ROM module as a MacGuffin (not withstanding what Bissell says about breaking away from physical media).

But for my money the computing technology and its relation -- or lack thereof -- to today's web isn't the interesting part of either book. Neuromancer is a ripping yarn set in a magical world whose magic happens to be presented narratively as a computerized virtual world. Snow Crash is a philosophical novel that uses an array of inventions, including but very much not limited to the metaverse, to frame its investigations.

In both cases, the strange but also familiar technology is telling us that the novel's world is a different world from ours. The authors, particularly Stephenson, use those differences to explore our own world. As such, there's no particular need for them to have predicted the actual world of a generation later.

The future still isn't what it used to be: Tog

2025-01-05T16:05:00.017-05:00

In 1994, so about 30 years ago, UX designer Bruce "Tog" Tognazzini's Tog on Software Design was published with this introduction. I wrote a post about it a mere 15 years later with a take on which predictions had and hadn't panned out. Another 15 years having passed, this seems as good a time as ever to take another look.

My first post included several direct quotes, which had the advantage of showing Tognazzini's actual words, but the disadvantage of leaving out some of them. This time around, I'm going to try summarizing the main point of each paragraph, with a few direct quotes for statements that seem particularly notable. Please have a look at the Tog's original page, as well. Unlike many old links on this blog, it still works, and kudos for that.

Tog's main points, as I see them, in the order originally written, were:

Phones, fiber and computers are [in 1994] about to converge. The whole world will be wired and national boundaries will no longer matter. Governments are trying to control this, but it's not going to work.
In particular, the Clipper Chip is a fool's errand because people can do their own encryption on top of it. Individuals will have access to strong encryption while banks and other institutions will be forced to use weak, government-approved encryption.
For example, the government of Singapore banned Wired magazine for an unfavorable article, but an online version was available immediately. "Traffic on the Internet cannot be selectively stopped without stopping the Internet itself"
Intellectual property laws can't keep up with new forms that build on putting together bits of existing content "as graphic designers and artists generate new art from old, snipping bits and pieces into a new creative whole [...] we are also seeing the emergence of a new and powerful form of expression, as works grow, change, and divide, with each new artist adding to these living collages of color, form, and action"
There will be increasing repression as corporate lawyers try to stop this. But this will end as corporations find ways to monetize content by having lots of people pay a little instead of a few people paying a lot [licensing fees at the time could run into the thousands of dollars] "As the revolution continues, our society will enjoy a blossoming of creative expression the likes of which the world has never seen."
While everyone's attention is focused on script kiddies, corporations will sneak around "America's boardrooms and bedrooms", destroying any illusion of privacy.
Security is also an illusion, but "The trend will be reversed as the network is finally made safe, both for business and for individuals, but it will be accomplished by new technology, new social custom, and new approaches to law."
The previous computer revolution, in the 1980s, resulted in a completely unexpected result: self-published paper zines. However [in 1994] it's hard to get distribution. Cyberspace [sic] will fix that, and creators will no longer need publishers in order to be heard. "[R]eaders will be faced with a bewildering array of unrefereed, often inaccurate (to put it mildly), works"
Tablets with high-resolution, paper-white displays will put an end to physical bookstores.
Retail will see increasing pressure from "mail-order, as people shop comfortably and safely in the privacy of their own homes from electronic, interactive catalogs"
"More and more corporations are embracing telecommuting, freeing their workers from the drudgery of the morning commute"
Schools will come to accept "that their job is to help students learn how to research, how to organize, how to cooperate, create, and think" and textbooks "will be swept away by the tide of rough, raw, real knowledge pouring forth from the Cyberspace spigot"
The term "information superhighway" is obsolete, because it doesn't do justice to Cyberspace, which will be "just as sensory, just as real, just as compelling as the physical universe"
A new economy will arise, based on barter and anonymous currencies that no government will be able to touch [this was written over a decade before the Bitcoin paper came out].
Initially, there will be digital haves and have-nots, but this will improve quickly as hardware becomes cheaper. The real problem is that the internet of the 1990s was built by mostly male hackers for their own use. There needs to be an "an easier, softer way" to access it, and only then will it see widespread adoption.
It's crucial to supplant the obsolete operating systems of the 1990s -- UNIX, Windows and Mac -- with object-oriented technology. Even 15 years after bitmapped displays were widely available [i.e., the first Macintosh came out in 1984], computers are barely shedding their old teletype-based look. We can't afford to wait another 15 years for OO to become widespread.
If all this is going to work, we need coordinated long-term strategies instead of each major player doing their own thing and hoping it all works out.

Honestly, I don't think my take on this has changed greatly in the past 15 years, because I think Tog's take is just as true as it was 15 years ago, or when it was written, even. That is, some parts are true and some parts are way off base, and which parts those are hasn't changed much. And, of course, it's likely that my opinions haven't changed greatly in the past 15 years.

Instead of comparing this post to the previous one, I'd like to look at the same themes from (I hope) a somewhat different angle. Last time around, I opined that the predictions that missed were mainly the result of assuming that a new development that's on the upswing will continue that way until it replaces everything that came before. I still think that's true, but what stands out to me more this time around is the apparent motivation behind the predictions.

Tog seems mostly to be grappling with the idea that computing technology of the 90s was poised to fundamentally overhaul our social structures. It should be clear to even the occasional reader of this blog (I'm pretty sure there are at least some) that I'm on the skeptical side of this one, but what really comes through in Tog's writing is a strong desire for this to be true, and in particular ways:

National boundaries will be obsolete. Government attempts to rein in technology will fail. Publishers will be irrelevant as entirely new forms of creativity emerge. Schools will change their entire mission. We will escape our physical bonds by working and living in a Cyberspace that's only distinguishable from the real world by its being more vibrant and vivid. OO will fundamentally change the way software is developed and open up whole new possibilities. Corporations and other major players will have to learn to work together in whole new ways.

No boundaries. No gatekeepers. No government interference. No physical bounds at all. New possibilities. New forms of expression. New ways of working. If you zoom out to that level, I don't think it would be much trouble to find a similar set of predictions from the 1960s, or the 1860s, or as far back as you want to go.

Or the 2020s, for that matter.

But national boundaries are still here. Reserve currencies are still around. Banking regulations still matter, even in the crypto world. Publishers, studios and record labels are still gatekeepers. To the extent schooling has changed, technology hasn't been a primary force (and remote schooling certainly did not replace students physically going to class). Telecommuting and remote work exist, but they don't dominate, they only really make sense for some professions and they don't mean jacking into a Snow Crash or Neuromancer virtual world, even though one of the largest corporations in the world has rebranded itself around exactly that vision.

Within this, a few particulars seem worth particular notice.

Tog wasn't the only one musing about new forms based on quoting existing material [the summary above doesn't mention sampling/quoting explicitly, but I'm pretty sure that's where all this was aimed at]. Ted Nelson's Xanadu project was all about that, and by the time Tog was writing audio sampling had found its way from 1970s hip hop into the mainstream, eventually giving rise to whole new genres.

But this was neither a new idea nor anything revolutionary (see these old posts for more detail). Quotations and allusions have been around forever. It's more a matter of how they're used. Sample-based sound fonts are widely-used, for example, but the whole point of most of them is to imitate live instruments as closely and unobtrusively as possible. In practice, sampling is quite often done in support of existing forms.

On the other hand, answer songs, which have been around forever, are all about the reference to a known song. It's common for an answer song to use the original tune or quote the original lyric, but it doesn't have to. The point is the reference to an existing work, regardless of how that reference is made.

A sample of the Amen break might be a deliberate reference that the audience is meant to recognize -- even if they most likely recognize it from other samples of the break -- or it might be reshaped or reprocessed beyond all recognition, or maybe some of both.

In short, there mere act of sampling or quoting is neither necessary nor sufficient for the creation of a new form. To the extent that there's even such a thing as a truly new form, people create them because that's what creative people do. Some new forms may make use of new technology.

I think "new form" is somewhat of a red herring anyway. I can think of several examples of encountering something wildly new, only to later understand its deep and direct connections to what came before. An album that sounded like it was from another planet suddenly made a new kind of sense after I'd heard a different album from decades before. And then it turns out that the songwriter behind that one had studied poetry in college and cut their teeth in Tin Pan Alley (I'm deliberately being a bit coy about which particular albums these might be, because this is just one example and my claim here is that the particulars don't really matter).

The newness was real -- nothing quite like either album had been produced before -- but so were the connections. And a lot of the newness was newness to me. As exciting as that sensation may be, it tends to dull a bit as you get more and more of the whole picture. But that's fine. The connections are just as interesting.

It's easy to get excited about something new and to want the world to look like the new thing. I think this is particularly easy for technologists, since our whole gig is to try to make new and (ideally) better things.

Tog in particular played a key role in developing Apple's early UIs (the term user experience (UX) was just coming into usage when Tog published Tog on Software Design). Apple products were, by and large, much easier to use than MS-DOS PCs. It's not hard to understand someone who'd helped make that happen wanting to sweep away obsolete rules and systems. Given that Windows was announced in 1985, the year after the famous 1984 Macintosh ad, it's not hard to understand the feeling that this was actually happening in real time. The ad itself does a great job of conveying the desire to change the world.

The world, for its part, has its own opinions.

Before I go, I wanted to touch on the predictions that did pan out.

The Clipper Chip did, in fact, fall into oblivion, not long after Tog was writing about it. Tog was hardly a Cassandra here, though. If anything, the Clipper Chip was a great example of how a group of people really, really wanting something to happen doesn't necessarily make it happen. The idea that you can use end-to-end encryption to get around an insecure transport layer, whether that insecurity is accidental or a deliberate back door, is old. Arguably, it's ancient, but in any case PGP, for all its flaws, had been around for a few years by 1994. Even government agencies seem to have thrown in the towel on this one in recent years.

Overall, there is a pattern of yes ... but.

Corporations did, of course, figure out how to make money by charging a bit at a time, mostly by running ads or by charging for subscriptions ... but neither of these is a new business model (in-app purchases are an interesting case, though [Insert Coin to continue ...]).
New case law and social conventions have developed around digital property ... but these look a lot like adaptations of existing law and conventions rather than something wholly new
Corporations have collected huge amounts of personal data about people, some of it, like genetic data, very personal indeed ... but it's hard to argue that "the internet has finally been made safe" from this as predicted. In fact ...
Security on the internet did indeed become a nightmare ... and it's still a nightmare
Zines morphed into blogs ... but even during the heyday of blogs, most of them went unread, and the same is true for podcasts, social media channels and so on today ("zines morphed into blogs" seems like one of those test sentences linguists use to show that we can understand a certain portion of language even if the words are totally made up)
Tablets did happen ... but they'd been a staple of science fiction for decades, and Apple itself had been working on the idea for a while by 1994 (the Newton came out in 1993), so this was more a matter of Tog asserting that eventually some kind of tablet would take off. Again, an assertion like that doesn't necessarily mean it will happen on a large scale, but it wasn't exactly a shot in the dark ... and, of course, bookstores are still around.
Online retail has had a huge impact ... but as I said the first time around, the term "mail order" is a big hint that this was more a shift in the mix of how goods are delivered (the original post snarkily mentioned WebVan, eToys and Pets.com, all of which were long gone by that time)
Telecommuting is a thing ... but it's also not a thing
"Information superhighway" stopped being a cool thing to say, if it ever was ... but (as I snarked the first time around) "cyberspace" also stopped being a cool thing to say, if it ever was
Cryptocurrencies happened, which seems striking since the Bitcoin paper was over a decade in the future ... but as to a "new economy [...] based on barter and anonymous currencies that no government will be able to touch" ... I've beaten this one pretty much into the ground here, so you be the judge
Object-oriented platforms have become mainstream ... but ... I'm not going to wade into the discussion of why software is the way it is, at least not here, but it's safe to say there are ills that the advent of OO platforms has not cured.

And then there are a few points where Tog's original post contains contradictory ideas because, I think, the underlying reality contains them as well:

The operating systems that Tog complained about (UNIX, Windows and Mac) are still around, but in a Ship of Theseus sort of way (see this followup post from the time -- just to muddy the waters, today's MacOS is a mashup of the original and UNIX by way of BSD and NeXTSTEP). So take your pick: Tog was wrong since they're still around, Tog was right since they've all been completely restructured over time, or some of each
In some sense, the internet knows no boundaries, but the Great Firewall shows no sign of going away and other regimes have found ways to severely restrict access. One way to look at it is that by default the internet knows no boundaries, but it can in practice if the local regime works hard to make that happen. This doesn't seem that much different from the earlier mass media, particularly TV, radio and print
The contrast between "often inaccurate (to put it mildly)" web publishers and "raw real knowledge" was jarring the first time around, and it's still jarring. The actual web/internet has been a mixture of both more or less from the outset.
Similarly, the tension between an internet built for geeks by geeks and an internet built for the whole world has been around from early days, and it's still around. Likewise for the underlying social issues around who gets access to technology and who pays the costs. Underneath this, particularly now that so many people are online, is the question of how much technology reflects society and how much it shapes society.

As I said above, I don't think my take on all this has changed much. I think I've mellowed on how I feel about the missed predictions, from "this is just horribly wrongheaded" to more like "this is a particularly clear example of something we all do", but what I think hasn't changed is the feeling that, however much I may disagree with many of the points, Tog is worth engaging with, by virtue of putting forth a strong and clear vision of the world, backed up by examples.

Tell us about the olden days

2024-12-29T18:59:00.006-05:00

The previous post talked in generalities about how the web and internet may or may not have changed how we communicate and live. To go along with that, I thought it might be interesting to consider some specific examples. Since these are drawn from personal experience, this post will show my age more than most do, but so be it. If you find it amusing to append old man to the questions here, well, I don't suppose I can stop you.

I'm going to answer these on the assumption that you have no memory of anything before, say, the 1990s, so please bear with me if some of this is already obvious. I'm honestly not sure how much of this will be "wow, I didn't know that" to a typical reader and how much will be "well, yeah, no kidding." Also, though I'll generally write in past tense, many of the things I'll mention are still true. I'll call that out here and there, but not necessarily everywhere, so if you find yourself thinking "but ... they still have those", you're probably right.

If nothing else, this post will probably serve as a reminder that as much as I grumble about ~~kids~~technology these days, a lot of this stuff is nice to have.

What did you do before GPS and mapping apps?

I grew up in a midwestern town that was built on a grid. The north/south blocks were long (eight to a mile) and the east-west blocks were short (twelve to a mile), so most addresses were on the north/south streets. First street was at the south end, running east-west, then second street and so on. The north-south streets had names in alphabetical order from east to west.

The first digit or digits of an address on a north-south street were the number of the nearest numbered street to the south. The last two digits of the address were 00 for the northeast corner lot and 01 for the northwest corner and generally increased by 4 per house from there. If I lived at 1234 Elm street and I knew your address was 2133 Maple, I knew to go nine blocks north (just over a mile) and, um, several blocks west, and your house would be on the west side of the street, toward the north end.

The main thoroughfares were a mile apart, since they'd started out as section roads, so you knew you could take 3rd, 11th, 19th or (later, as things got built out) 27th to get across town from east to west, and Cedar, Oak or (again later) Agate to get from north to south. A lot of towns were laid out using some version of this kind of scheme, and for that matter so were a lot of cities. San Francisco is a notable example -- a lot of folks would have built streets to follow the contour of the hills (to be fair, some do).

I say "were" and "could", but of course they didn't rename or renumber anything just because GPS came along, though it does certainly seem to matter less now. I currently live in an area with a large-scale grid of section roads, and many of the towns are on small-scale grids, but I've never bothered to learn the exact numbering schemes, even in my own neighborhood, because GPS is just easier. I do know the section roads reasonably well.

My first answer, in other words, was "you just got to know your way around town" and "the addresses were set up to make that easier".

That worked fine until I moved to an area on the East Coast where nothing was on a grid. At that point MapQuest was around, but I didn't have a smartphone. I ended up doing a fair bit of printing out directions off the web, trying to mostly memorize the way before starting out, peeking at the directions while stopped at stoplights and keeping a weather eye out for street signs and house numbers.

And getting lost fairly often.

Gradually, I learned the main roads and how they connected together, and how the smaller connectors connected to those, and where the main places I wanted to get to related to all that, and things got easier. People would also give general directions like "It's near Chestnut and Amethyst where the main library is. Turn left on Locust Street after the light and Smith Court will be a few streets down". If you already knew where the main library was, or even where Chestnut and Amethyst were, you had a pretty good shot.

There were also some clues like the common pattern of naming a main road after a city it was headed toward. For example, Richmond Road in Twickenham goes toward Richmond and Mortlake Road in Richmond goes toward Mortlake, and it's probably not a coincidence that in both cases you're heading toward London proper (there is no Twickenham Road in Richmond or Richmond Road in Mortlake, but on the other hand, back Stateside, Chapel Hill Road in Durham goes toward Chapel Hill, where it becomes Durham Road in Chapel Hill ...).

Later, I realized that learning the main road/smaller road pattern was something I'd dealt with before before, traveling in Europe, except that instead of main streets it was usually the public transit system -- get on the subway at your stop, follow the subway maps to your destination stop, find your actual destination from there.

My main problem then was that I don't have a great sense of overall direction. If a road takes a bend here and a curve there, I might think I'm headed pretty much the same direction I was before, when in fact I've turned almost 90 degrees.

So I really like having GPS available as a backup, even if I wish there were an easy way to say "yeah, I know this part, start giving me directions when we get to this part and just let me listen to my music until then."

The other part, of course, particularly before MapQuest, was knowing how to read a street or road map, which seems to be something of a lost art, a clear sign that GPS is just plain easier (particularly if your brain doesn't deal well with maps).

For long trips you had the Rand McNally Road Atlas, which showed all the interstates, federal highways and main state roads, along with cities and towns, with mileage shown on each segment. The distance between one town or exit and the next was shown as a number halfway between in one color. Some of those waypoints had a special dot in a different color, and the distance between those with special dots was shown in that color, so you didn't have to add up all the segments in between. There was also a schematic depiction of the interstate system with mileage numbers.

In other words, the road atlas encoded exactly the same kind of edge-weighted graph that mapping software uses, and you could use that to figure out the shortest route from point A to point B along main roads. If you had time, you could look for cutoffs on secondary roads. If you were adventurous, you could try to find local shortcuts and hope that at least you could find your way back to something that was on the map.

You might also carry smaller-scale state or regional maps, which you could get at any gas station (maybe still can). If you were staying in a city for a while, you'd pick up a city map, too. The road atlas also included maps of the main roads in most cities, and you could usually get by with that if you were just passing through [re-reading, I realize I forgot to mention that you could also ... stop and ask directions].

Any of these maps would be overlaid with a square grid with numbers in one direction and letters in the other, and there would be an index, so you could find out that Springfield was in square 5A and quickly find exactly where it was and figure out how to get there.

When I lived in the LA area, the Thomas Brothers map worked basically the same way (as does London's A to Z, along with, I'm sure, many, many others), so you could figure out that to get to the Sherman Oaks Galleria you take Wilshire to the 405, get off at the Ventura exit, hang a left on Sepulveda and a right onto Ventura and there you are.

But what about traffic? To this day, many local radio stations will provide frequent updates on road conditions and traffic, and make money off this information by selling ads. Just sayin'

Summing this all up

Many places were designed to be easy to get around
Pretty much any city has a system of main roads, secondary roads and side streets that you can just learn if you need to
There are maps available at several scales. Larger scale maps include distance information and pretty much all have grids and indexes.
Getting around is easier now, but it wasn't really that hard before smartphones and GPS, because there was already quite a lot of infrastructure to make it easier, particularly if maps are friendly to your brain.

What did you do before cell phones and texting?

Cell phones may have had the most noticeable effect on day-to-day life of all the web/internet/telecommunications advances of the past few decades.

Besides the clothes and hairstyles, one sure-fire sign that a movie is old (or the screenwriter is a bit behind the times) is a plot device that depends on a phone call. Our hero needs to get in touch with someone urgently. Can they make it to a payphone? Will the person they're calling be at home or at their desk? Will the line be busy? Will the wrong person answer the phone? Or maybe the right person is at home but they're afraid to pick up the phone because it might be the villain calling? If the hero had to leave a message, will the other person get home to check their answering machine in time? Will the wrong person overhear them leaving the message on the machine?

None of these really works today because

Phones are now associated with people, while they used to be associated with places
Today's phones can do more. For most of the landline era there was no caller ID and most phones could only handle one call at a time
Messages can now be stored in the cloud rather than locally on analog tape

Since a landline is associated with a place, the vast majority of households had a single phone line, though there might be multiple phones in the house connected to it. If you called the number for that phone, you were calling the house. Someone would answer ("Hello?"), you'd say who you were ("Hi, it's Dave") and, if you wanted to talk to someone else at the house, who you wanted to talk to ("Could I speak to Earl?"). If that person was somewhere else, you could ask the person you were talking to to leave a message.

You could also just hang out and chat with them -- if you know Earl, you probably know Chris, the housemate, or Chris's good friend Sam, who doesn't live there but hangs out enough that everyone's comfortable with them answering the phone. The chance of talking to someone other than the person you were calling for wasn't necessarily a bad thing, though of course it could be.

The other half of a phone being associated with a place was that if you wanted to make a call, you had to get to a phone. That's why there were payphones (still are, here and there, I'm pretty sure). Or you could stop by a friend's house and ask to borrow their phone. In a pinch, you might be able to drop into a nearby business and ask to use their phone, but it had better be an emergency.

You could also call to a payphone, since they each had their own number, but that was pretty rare, to the point that a lot of people weren't aware that you could even do that. You'd mostly see it done in a movie, where the villain tells the hero to wait at the payphone at 12th and Main, and some innocent bystander steps in to make a call at just the wrong moment.

But that also meant that if you were away from a phone, no one could call you and no one expected to be able to. Earl's not home? Cool, I'll try later, or maybe I'll run into him. Likewise, no one expected you to be able to call them. The most likely answer to "Why haven't they called me back??" was "They're not home yet."

Honestly, this was kinda nice. I still miss it from time to time. Sure, you can unplug today, but it's not the default.

Voicemail today is mostly the same as it was fifty years ago. You could record whatever outgoing message you liked. When someone called your phone and the answering machine was turned on, it would play your outgoing message, beep and start recording whatever was on the line until the connection ended or (I'm pretty sure) until you picked up the phone on your end.

Depending on how a switch was set, it would also play what it was recording on its speaker, so you could hear the message that was being left. People who were at home could and did screen calls that way, so leaving a message might look like:

... Please leave your message at the beep

Hey, it's me ...

[picking up phone] Oh hey! I was hoping you'd call

but it might also look like:

... Please leave your message at the beep

Hey, it's me ...

[muttering to self and not picking up phone] Yeah well you can just take that phone and ...

... and I just wanted to say ... again ... I'm sorry I'm sorry I'm sorry

Sure, you can still screen calls and now even block people (which you couldn't do), but there's something special about listening in in real time.

Again, the main difference is that the answering machine is tied to a landline, which is tied to a place, so typically you'd check your answering machine for messages when you got home and either turn it off, or leave it on and screen calls. If you turned it off, you had to remember to turn it back on the next time you left ("Oh no, I'm sorry you couldn't leave a message. I forgot to turn my machine on."). It was also possible to access your voicemail by calling in and using a Touch-Tone™ keypad to put in a PIN, but to do that ... you'd have to get to a phone (and even late in the game, a lot of phones still had rotary dials, so not just any phone).

But of course, and this is the part that surprised me enough I remember discussing it in at least one post here, nobody leaves voicemail any more. I mean, you can still do it, but I'm not sure when I last left a voicemail for a person, as opposed to a business or doctor's office.

It took me a while to understand why. If you call someone and it goes to voicemail, surely it's easier to just say a message than to hang up and type out a text. Fair enough, but it's even easier to just type out the text without calling and waiting for an answer or voicemail. The setup for a voice connection is heavier weight than one might expect.

It's also a lot easier for the receiver to glance at a text than to access voicemail and then listen through. After hearing "Why didn't you just text me?" over and over, voicemail starts to look less and less attractive. With smart keyboards and speech-to-text, texting isn't that hard anyway, at least in my experience. And so came the return of telegram style and enough abbreviations, slang and conventions to (arguably) constitute a new dialect.

So the main differences here are that it's harder to unplug and ... text.

I was going to emphasize how utterly disruptive it is to always be connected but ... maybe not. Yes, I'm reachable by phone most of my waking hours, but I don't actually get that many phone calls. In particular, I don't get a lot of cold calls. Most of the time if someone calls me it's an actual person that's either a friend/family member or someone I'd asked to call me. I don't get a lot of spam calls to begin with, and if I do, I can either decline and let it go to voicemail, to check (and delete) at my leisure, or use a screening feature to ask them to leave a voicemail (which they never do).

I think some of this is regulatory, but spammers/scammers don't generally let regulation get in their way too much, so this must mostly be a matter of there being cheaper and more effective ways to spam and scam.

Most of the interruptions I get, by far, are notifications from apps which I chose to get. Or at least, I didn't diligently ask not to get. Many of these are email notifications. I don't get a ton of email, but I do get a steady stream through the day, just almost enough to want to Do Something About It. I also get notifications for texts, which are generally from people I know, so I tend to look at them right away, and from a couple of news sources, which are pretty selective about only sending out alerts for major news. The main thing that's bugging me right now is the stream of "hey this movie just came out" notifications that I don't recall asking for, but that seems to have tapered off (or I turned them off?).

In other words, being "always on" doesn't seem to require being very "on", and there are a few things I could do to make it less disruptive yet. On the flip side, I can call or text pretty much any time I want, as long as I'm not driving, and even then it's usually not that hard to pull over. If I'm at the grocery store and I want to double-check what a household member wanted, that's easy. If I'm in an accident, I can call 911 (or if I can't, I have bigger problems). And so forth.

It's also easier to meet up, which is nice. I remember arranging to meet friends in Berlin not long after the Wall came down. After a few phone calls (and maybe even letters and postcards?), we arrived at a plan: meet on such-and-such date at the Kaiser-Wilhelm-Gedächtniskirche at high noon. If not everyone was there, come back at 1:00 and so forth, so that whoever was already there wouldn't have to sit around waiting. I don't think we set a time to give up waiting, but it was understood that if it got to be too late, whoever was there would just go on and see the city and whoever couldn't make it couldn't make it.

As it turned out, most of us were there at noon and the others showed up at 1:00 and off we went. During the visit, we'd occasionally arrange a rendezvous point to meet up at if we got separated, which may or may not have actually happened. This was pretty normal and it tended to work pretty well, but again, I'm not sure when was the last time I've made a plan like that, because why bother when you can just text or call? It's still a good trick to keep in mind, though, I think.

So staying connected is disruptive, but not really all that disruptive. Being more connected also has some conveniences, but is it really all that much more convenient?

What did you do before search?

For the purposes of this post, let's assume that search Just Works: you can easily find any particular bit of information you need, assuming it's on the web somewhere (and "the web" basically means "whatever your search engine can find").

Sometimes, you'd end up just not finding something out. But there were options.

If you wanted a phone number or address, you could look someone up in the phone book. Since phone books were physical things, and fairly hefty ones in many cases, you could really only access them locally, though many libraries had phone books for major cities. In other words, there was an element of privacy protection built in, which tended to be enough for most purposes, though people did get unlisted numbers for various reasons.

Adding on to my claim that changes in how phones work have been among the most disruptive, that whole paragraph is from another age. If I want to call a business, their number is on their web page. If I want to call a person, we'll have exchanged phone numbers (likely by text, of course). My phone will remember my contacts, but that's actually not such a big deal. It doesn't take a lot of space to write down names and addresses of people, and you could get a miniature notebook for just such a purpose (I still have one somewhere).

The main convenience is being able to tap on an icon and have the phone place the call without even having to know the number -- there are only a few numbers I have memorized now, but mostly because I use them for supermarket loyalty programs and such, not because I dial them.

But what would one actually search for?

There are two main categories, I think. One is day-to-day information: Where is there a good restaurant that serves X kind of food? When is the DMV open? Does the local hardware store carry left-handed socket wrenches? (No, that's not a thing).

This is largely a matter of advertising (which, for the purposes of this post, at least, is distinct from search). Businesses have an incentive to let as many people as possible know that they're around, so there's probably a local restaurant guide that will tell you who serves what, and there are probably multiple copies of it in various drawers in the house, or under the couch, because they just seem to keep turning up and, yikes, maybe they can reproduce?

You probably got something in the mail telling you when local government offices like the DMV are open, and they're probably listed in the yellow pages as well (back to phones ... there were actually two kinds of phone books: the white pages had residential listings for anyone with a phone who didn't opt out, and the yellow pages had paid listings for various businesses and similar entities. It's been so long since I've used one that I almost forgot that DMV would be in there).

Unless you lived in a major city, there probably weren't that many restaurants in town anyway, and it didn't take long to get to know them. As to hours, it was a good bet that anything that was open for business would at least be open between 10:00 and 4:00 on a weekday, and anything retail was at least open on Saturday (though maybe not Sunday, depending on where you were). Again I say "was" and "would", but as far as I can tell, that's still mostly true.

In other words, if you search for "X restaurant near me", you're not asking something that could only be answered, or only be conveniently answered, once search engines came along. You're asking something that used to be reasonably easy to answer and is now somewhat easier, in principle.

As to what's available for sale where, some outlets would put out catalogs (the Sears Catalog is a famous example -- I hope that when you chase that link it says more about the cultural significance of that catalog) and many stores would put out flyers in the local newspaper saying what they had on sale that week.

Or (once more back to phones) you could call the hardware store and ask whether they had left-handed socket wrenches, and most likely someone would actually pick up the phone on the other end and tell you (and try not to giggle too loudly).

Long story short, most of the "Where can I find this in the physical world right now?" questions could be answered pretty easily, because people had an interest in making them easy to answer, just as they do now. The main difference is that there were more people involved. For example, there were more people working at a typical retail store to help customers and also to answer the phone if someone called. And that was kinda nice. I still miss it from time to time.

The other main thing I use search for is research, for example looking up material to put in a blog post. Having so much material online and searchable changes things considerably, but despite what the cartoon might suggest, it wasn't impossible to find things out.

To be sure, this wasn't something most people could do at home, if it wasn't in the dictionary or encyclopedia that were both much more commonplace then, or in some book or magazine that you happened to have on hand, and it helped, a lot, to have a university library or similar institution in your area. If you could get to one of those, though, there were definitely resources:

There were probably microfilm copies of major newspapers and magazines plus local and regional publications
Reference books like The Reader's Guide to Periodical Literature would tell you what was in those publications
There would be copies of major scientific journals
There would be a large reference section full of reference books on a wide variety of subjects
And, of course, there would be a large collection of fiction, and non-fiction history, science, art, music and many other subjects
Along with a card catalog to tell you where to find all of the above
Your local school library would be a miniature of this, so you could practice finding books in the catalog and maybe even reading a microfilm copy of a news article you found in the Reader's Guide.

The main problem, besides not having access if you didn't live near such a library, is that it's harder to keep a collection of physical texts up to date. Even so, the library would have subscriptions to many major publications, and the Reader's Guide published updates biweekly, so you still could get a pretty good idea of the latest developments.

All this depended on your library carrying the type of information you were interested in and the major reference publishers indexing it. These being human endeavors and resources being limited, there was plenty of room for conscious bias, unconscious bias and plain old budgetary constraints to skew the picture.

Today, of course, you can search anything the major search engines index, major news sources update their pages continuously, preprints are up on ArXiv as soon as the authors want them to be, and information is generally available more widely and more quickly than it used to be. I'm going to steer well clear of how the current web-centric view of the world of information may be biased, and why, but I'll certainly acknowledge that it's a worthy topic of discussion.

But then, how many people are in the business of serious research? For us amateurs, how much does it really matter whether I find out about a new development right away, or in a month or a year when it finds its way into the library system, or a friend mails me a photocopy of someone's lecture notes, or whatever else. For the pros, even a good search engine will only get you so far.

Beyond that, as far as I can tell, you still need a good network of sources, whether primary sources or people who can point you to them or pull together, assess and summarize the information from primary sources. It remains to be seen what role LLMs will end up playing in that, particularly in professional-level research.

Search engines make day-to-day questions more convenient to answer, and they make the amateur researcher's job quite a bit easier, but were those all that hard to begin with?

What did you do before streaming?

Bought CDs, bought/rented DVDs or videotapes, watched TV, went out to movies, not to mention quite a few live shows.

Sometimes even talked to people.

What did you do before LLMs?

Dunno ... what did you do? It hasn't been that long!

What changed?

2024-12-29T13:55:00.002-05:00

This is one of those posts that started as one thing, trying to make some sort of Larger Point, but ended up as ... something. It started out on the long-running theme of not-so-disruptive technology, then devolved into a technical exploration as I tried to back that point up, and then went a somewhat different direction because of what I actually found when I went researching, before sorta circling back to the general vicinity of the of the original theme and pulling together some threads from some of the first posts on this blog from, oh, a minute or two ago. Rather than try to polish all this up into some sort of coherent essay, I've decided to leave it pretty much as written. Perhaps as some sort of compensation, I've included a lot more links than I usually do.

Looking back I see that in 2024, I've already doubled my output from 2023 (by a score of two posts to one), so maybe I should quit while I'm ahead. But I had an idea for a post, and after re-reading back to July of 2020 (that is, seven posts), I'm pretty sure I haven't explored this particular point before, at least not recently. Or rather, I have, given that the not-so-disruptive technology tag is in second place behind annoyances, but if I've stepped back and surveyed it from a broader point of view, it hasn't been in the last four years.

(I also notice that the link to Intermittent Conjecture is for a four-year-old post, probably because that particular feature is no longer particularly supported, because of course it's not. Grandpa, what's a "blogroll"?)

I considered editing that last bit of snark out, especially since annoyances is already well represented, but I think that it's probably in line with the rest of this post, though maybe in a roundabout way.

It's almost an axiom that newly-developed technology will Change the World. I say "almost" because technically an axiom is a statement that you assume to be true because it's essential to the rest of your logical framework, but you don't have any other way to prove it to be true, so you have to just assume it. I'm thinking of mathematical axioms like "a thing is equal to itself" or, more esoterically, "if you have a collection of sets, you can form a new set by choosing one element from each" (it took quite a bit of work to figure out that you can't prove that from other axioms like "two sets are equal if you can match up their elements one-to-one in both directions").

"New technology changes everything" is a statement that people often assume to be true, and it's essential to at least some people's logical frameworks, but I wouldn't call it an axiom because you can actually look at any given new technology and, I claim, come to a reasonable conclusion as to whether it changed everything. And then, maybe, as a followup question, by how much?

To take a couple of easy, well-known examples, it's not hard to argue that, say agriculture changed everything, or antibiotics changed everything. Except ... depending on what you call "agriculture", you could argue that agriculture was around for thousands of years before cities like Shuruppak or Dholavira arose. On a smaller timescale, the first modern antibiotic was extracted from mold growing on a bacterial culture in 1928, but it wasn't available in useful quantities until the early1940s.

It's not the discovery of a technology that makes the difference. There wasn't even any one event that you could call "the discovery of agriculture." There was an event that could be called "the discovery of (modern) antibiotics (that were known to work by killing microbes)", but that in itself didn't change anybody's life greatly.

The point here that simple statements like "agriculture/antibiotics changed everything" turn a bit mushy after even a little prodding. More accurate versions might be "over the millennia, developments in agriculture have had a significant impact on human population and living patterns" or "the development, mass manufacture and widespread deployment of several types of antibiotics in the latter half of the 1900s had a significant impact on human health outcomes."

Clearly there have been significant changes in how people live, and clearly developments in agriculture and medicine, including the development of antibiotics, have played a significant role in that, but it's not a simple matter of "agriculture happened" or "antibiotics happened" followed by "everything changed". The actual stories are full of false starts, backtracks, accidental discoveries, social upheavals, twists of fate and all sorts of other seemingly extraneous factors. Which is the interesting part.

What got me started on all this was thinking about how the web has changed communication, and in particular telecommunication. Except, as soon as I wrote that, I realized that it's more a matter of the internet changing communication, since I've already argued that it's the web of links that makes the web webby, and I'll just claim here that this webbiness hasn't had a large impact on how we communicate with each other.

We could just as well have Skype and Zoom without the web. For that matter, to a large extent each social media platform is its own web, and not "the" web. But that way lies yet another round of fretting over what exactly am I blogging about here ... For now, let's file communication technology under "the web at large" or something and get on with it.

For most of human existence, the only way to communicate detailed information over a long distance was by people moving around. Travelers would bring stories and knowledge and trade items with them and information would diffuse across large areas, but if that traveler wanted to send a specific message to someone they'd met years ago while traveling someplace far from their current location, well, good luck with that. It may not have been impossible, but it couldn't have been commonplace.

Several thousand years ago, digital communication came along and changed this. With writing came the option of moving a written message with the sender's exact words (there wasn't any single "invention of writing", either, but let's just roll with it). Messages could be sealed so that their contents couldn't be easily changed, signed so that you could tell who they came from, and even encrypted so that only the intended reader could read them, or at least that was the idea.

Digital telegraph systems, also dating back thousands of years, could transmit text from point A to point B without even needing a person having to carry it. The Greek phryctoria, a system of towers on mountaintops with torches, are a good example but not the only one.

Two key measures of telecommunication are bandwidth, which is how many bits can be transmitted in a given amount of time, and latency, which is how long it takes to transmit any particular bit from sender to receiver. As usual, the actual definitions are more subtle, particularly for bandwidth, but these will do here. If you're feeling technical, feel free to read bandwidth as bitrate.

For example, if it takes three seconds to switch the torches in a telegraph tower around to show a new letter, and there are 24 possible letters, then the bandwidth is about 4.6/3 bits per second, or about 1.5bps. The latency from one tower to the next, around 30km away, is negligible (about 0.1 milliseconds).

If the message is supposed to be relayed to the next tower in a series of towers, it will take some amount of time for someone to read the arrangement of torches in the sending tower and put the same torches up so the next tower can see them. Let's say there are two people in the tower, one reading and one putting up torches, and it takes an extra second for the reader to read and announce the next letter, on top of three seconds to arrange the torches. Latency is then four seconds per tower. That is, if the first tower is sending a message and the second is relaying it to the third, the third tower is getting the message four seconds after it is sent. A fourth tower would be eight seconds behind, and so forth.

Suppose I want to send a message to someone ten towers away. Latency is still pretty good, relatively speaking. The last tower will be 36 seconds behind the sender (nine relays for ten towers). If that receiver sends a reply, I can get it just over a minute after sending my message (in more technical terms, round-trip latency is on the order of a minute). While this is glacial by today's standards, it's outstanding in comparison to a multi-day journey to get from where I am to where the receiver is, and I don't have to worry about someone waylaying my messenger along the way (or my messenger deciding they have better things to do with their time).

Bandwidth, though, is not so great. If I'm sending a short message like "Prepare for attack from the north," that's not a problem. Transmitting that message will take a couple of minutes and my receiver will have the whole thing half a minute after I finish sending it. But suppose I'm sending a trade agreement proposal that amounts to 12,000 bits -- still tiny by today's standards. That will take a couple of hours, which is still doable, though not a lot of fun for anyone involved.

But the people on the other end will want to respond with their own counterproposals, and so on. Pretty soon we're into days, and spare a thought for the twenty people up in the towers shuffling torches around and looking out for torches at other towers through the night (I'm going to go out on a limb and say this system works better at night).

Probably better to send a trusted emissary with the text of my proposal and maybe some other written instructions. And while they're at it, they could carry messages from other people in my area to people in the receiver's area, or anywhere along the way, and we have ourselves the beginnings of a postal system. The latency of a postal system is measured in days, but the bandwidth is essentially limited only by how fast people can actually write and read and how many people are sending and receiving messages -- you can fit a lot of sheets of paper onto a horsecart. Not to mention that you can also send drawings and diagrams easily on a sheet of paper.

This may seem like a lot of speculative detail about ancient systems of communication, and it probably is, but it covers the bulk of human history (the written-down part, as opposed to prehistory, which is most of human existence). From ancient times until the late 1800s, long-distance communication was mainly a matter of moving physical texts around, with limited use of alternatives that were much faster (in latency) but also much, much slower (in bandwidth), and quite a bit more expensive. This includes the era of the modern optical telegraph (late 1700s) and electrical telegraph (mid 1800s).

What happens next is interesting. I originally wrote "then came along the telephone," with the idea that it was a major leap to have the bandwidth to carry voice instead of the dots and dashes of morse code. Fortunately, I did a little double-checking and discovered that

The bandwidth of a telegraph was not that low. A punched-tape system around the time of the telephone's invention could transmit upwards of 400 words per minute. At roughly 12 bits per word, that comes out to about 80 bits per second. That's nothing by modern standards, but it's about 50 times my guess for the phryctoria. Some of that is because Morse code encodes text more efficiently than torches, but most of it is due to the switch to electromagnetic transmission (um, light from torches is also electromagnetic ...).
The bandwidth of human speech is not that high. In this old post I cited a world record of 10 words per second, or about 120 bits per second, but normal speech is much slower.

In other words, a telephone and a high-speed telegraph are transmitting words at about the same rate, though the telephone has the advantage of carrying tone of voice and not requiring someone to transcribe words onto a paper tape. I suppose this shouldn't be too surprising since both the telephone and telegraph are using the same underlying transmission medium of electromagnetic waves traveling along copper wires or, a little later, over the air.

The same technology could also transmit images. The first facsimile machine (perhaps you've heard of "faxes"?) was developed around the same time as the telephone. Later, in the 1920s, a number of inventors on a number of continents (including Leon Theremin, better known for the musical instrument) developed various systems for transmitting moving images. Early television station WRGB ("RGB" can't be a coincidence, can it?) transmitted 40-line images at 20 frames per second. Let's guess that a 40-line image equates to 1600 8-bit pixels. That comes out to about 260 thousand bits per second (260kbps).

This is already a remarkable increase in bandwidth*, from a hundred or so bits per second in the mid 1800s to hundreds of thousands in the early 1900s. By the dawn of the internet, let's say 1974 -- fifty years ago -- when the proposal for TCP was published, a leased telephone line could carry around 50kbps (56kbps as I recall and Wikipedia seems to confirm). That was the basic unit -- it was entirely possible, and typical, to lease more than one. By the mid 1980s, NFSNET was using 1.5Mbps T1 lines. Later came T3 lines at 45Mbs (so a T3 is worth 30 T1, go figure), and today we're talking gigabits or more.

This is all a matter of how bandwidth is sold. The actual transmission cables are much heftier. Fiber optic cables can carry petabits per second (Pbs). A peta is a million gigas, that is, a petabit per second is a quadrillion bits per second, or about 125 thousand bits per second for every person on the planet. Commercially available cables are somewhat smaller, but not much, measured in hundreds of terabits, that is, hundreds of trillions of bits per second.

There are still some specialized applications that can give that much bandwidth a workout, but in human terms the amount of bandwidth available is absolutely ridiculous ("available to whom?" is a fair question). Which brings me back to one of the earliest themes on this blog: limits on human bandwidth. That is, how much information can any individual person deal with? I discussed several aspects of this in this post about, oh, seventeen years ago.

In terms of bits per second, our highest use of bandwidth is probably the visual system,.which processes somewhere around a gigabit per second considered as raw pixels, but there's a lot of redundancy in there. A good MP4-compressed video stream, which includes audio, is more like 10Mbps. Since a format like MP4 is tuned to provide only the information we actually process, it's probably a better measure of how much data the visual system is actually processing.

There's a lot we don't know about our other sensory input -- touch, smell, proprioception and whatever else, but it's clearly operating at a much lower bandwidth (for example, a walking robot does not need a fiber optic cable to tell the CPU how far its knee is bent or how much pressure its foot is exerting).

In other words, there are many, many ordinary houses with much more than enough bandwidth to saturate the sensory input of all the humans in them, if said sensory inputs could all be magically connected to a stream of bits. In practice, it means that there's enough bandwidth for everyone in the place to spend all their time watching video.

But -- and maybe this really is leading to some sort of point about technology changing everything -- that's been true for quite a while, at least since the advent of 24-hour cable TV, which is to say, also about 50 years ago, which I've just called the dawn of the internet. I don't think this is at all a coincidence. Let's try to boil all the stuff about bandwidth down to a few bullet points:

For most of human existence, long-distance, low-latency bandwidth was zero -- there was no way to get a specific message across a long distance quickly. You could interact with some directly at short distance with high bandwidth and low latency, but that was about it.
For most of human history, long-distance, low-latency bandwidth has been very low. In some times and places it was possible to quickly transmit a short message over a long distance, but even then, latency was measured in minutes and bandwidth in single-digit bits per second.
Starting in the 1800s, electromagnetic transmission led to huge increases in low-latency, long-distance bandwidth, from single-digit bits per second to current rates, which are enough to enable video calls between any two internet-connected points.
In the mid to late 1900s, bandwidth was high enough and cheap enough to enable two innovations:

Cable TV carrying over a hundred channels 24/7
Wide-area digital networking

Of the two, digital networking was by far the slower. Early networks mainly transmitted text, whether in human or computer languages. If you had a terminal at home, you could typically connect to your local network at speeds of 110 to 2400 baud (in general a different unit from bits per second, but in this case the same), and hope that you'd remembered to turn off call waiting on your landline. Then, after a long day of hacking, you could flip on the TV and watch at something like a megabit (resolution was lower in those days).

Even backbone connections were very slow by today's standards. This doesn't seem like a technical limitation, since ordinary coax cable could handle megabits, but more a matter of there not being that much digital information to send. If I wanted to talk to a colleague on the other side of the country, I wouldn't have tried to set up a call over the internet at the time. I would just pick up the phone.

The digital convergence that happened gradually over the next couple of decades consisted largely of building up the internet backbone, which was based on telephone and cable technology (mostly telephone, I believe), to the point where it could carry digital information at a rate comparable to the analog technologies that had been around since the beginning of the whole exercise.

Technically, this was revolutionary. For most intents and purposes, anything that was analog in the mid 1900s, particularly television, telephone and radio, is now carried digitally on the same network infrastructure that you can use to send purely digital information like ... text and emails? Source code?

This is a kind of interesting way to look at it. Hiding inside the massive digital network that delivers sound and video to us is a tiny replica of the original internet, albeit expanded from a few thousand researchers to a significant slice of the world's population. Billions are bigger than thousands, of course, a million times bigger, in fact, but overall digital bandwidth has increased by much more than a factor of a million.

(The early internet wasn't just used for email and source or object code. It was also used to transmit scientific data. Some datasets can be quite large, particularly in astronomy and particle physics, large enough to saturate even the modern backbone. But in such cases data is generally transmitted by putting it on physical media, which is then shipped. The postal service still wins on bandwidth. And yes, I am proudly using both data and media as mass nouns here.)

I think what I'm trying to sort out here is that the digital convergence can be looked at two ways. The original vision was to bring the intelligence of the internet to existing audio and video media. A TV cable brings a fixed set of channels into your house and very little back out. An analog phone circuit delivers voice traffic from point A to point B. A digital network can carry information from any number of senders to any number of receivers and do any kind of processing along the way.

On the other hand, technically, the digital convergence was a shift from sending analog data over analog lines (or over the air) to sending the same data over the same lines, or at least the same types of lines plus the cell network (also fundamentally analog), but encoded digitally, then re-encoded into analog signals and likewise decoded and re-decoded on the other end.

Why do that?

The wilder speculations of the 1990s haven't really panned out. A phone call is still a phone call. True, most of the time it's easier just to text, but texting needs much less bandwidth than calling. It certainly does not require a huge buildout of digital bandwidth. All the texts you send in a year would probably amount to a few seconds of audio.

TV shows are still TV shows and movies are still movies. Exciting new possibilities like interactive choose-your-own-adventure TV are an occasional novelty. Live streams allow viewers to interact with the presenter/performer, but so did call-in TV shows.

The difference is control. Outside the occasional news program or sporting event, I'm not sure I can remember the last time I watched something at the same time it was broadcast, if it was ever broadcast at all. I haven't bought an album in years, even in digital form. I stream what I want to watch or listen to, and I'm hardly a bleeding-edge early adopter. If I want to participate in a livestream, I can choose that. More importantly, if a creator wants to put on a live stream, they can easily do that. If I want to set up a video call with some people at work (or not at work), that's easy, too.

Some of these might be possible with the old technology. I could imagine a high-bandwidth phone service that would allow you to call a special number to connect to a video server and pick out what to watch on your video-enabled phone terminal, but putting everything on a digital network that handles data as bits regardless of its content or where it's going has made all of this much easier.

This is all sliced finely enough that individual people can decide which individual people to communicate with, from friend group to celebrity influencers to major organizations and whatever else. I'm personally not sure how much the behavior that this has enabled is new and how much is stuff that people were doing anyway. I explored that theme fairly early on, here, here and here for example, but I don't really do much with social media, even if you count blogging and the occasional visit to LinkedIn.

I think "Digital communication has changed everything" is true in about the same way as "Agriculture has changed everything". On the one hand, it has to be true. Being able to communicate instantly with any of billions of people has to be different from only being able to communicate instantly with the people around you. Being able to transmit high-resolution video across the world with negligible delay has to be different from being able to send a letter across a continent in days or weeks.

Being able to stream from a wide collection of audio and video is certainly different from having to buy or borrow books, records/CDs and videotapes/DVDs, and since that shift has happened well within living memory, it can certainly seem like things are changing rapidly.

But on the other hand, digital technology, including digital telecommunication, has been around for thousands of years. Analog telecommunication has been around for about a century and a half. What we might call the digital revolution is a change in how we transmit and access information, primarily audio and video, that had previously been analog, sitting on top of a huge increase in overall telecommunication bandwidth that began happening over a hundred years ago.

Just as there is no particular beginning of agriculture, there is no particular beginning of digital communication. Even if you could pinpoint the first time a person deliberately planted a seed with the intention of harvesting food later, or the first time a person deliberately made marks to represent words with the intention of someone else reading them later, it wouldn't tell you much. What matters isn't the particular starting point, but the long history of development and use over the millennia.

So far, advances in communication have been about people communicating with people. Machines do communicate with other machines without direct human involvement, but this is mainly in service of people communicating with people. This may change, but that's for another blog.

As far as people communicating with people, the limiting factor is mainly the people themselves. There are only so many conversations one can have and so many people to have them with. The whole point of a video conversation is to make the call as much like talking face to face as possible, that is, to accommodate our limitations in how we communicate. There are now ways of broadcasting a message from one person to millions of people, or even a billion, but even if one person can broadcast a message to a billion people instantly, those billion people will make sense of it in terms of their own lives, their own views and their own desires.

The how of communicating with other people has changed greatly over the millennia, and particularly greatly in recent decades. This in turn has significantly affected whom we can communicate with. But what we talk about, even if we're talking about how quickly things appear to be changing, doesn't really seem to have changed much at all.

One of the earliest themes of this blog was trying to understand what effect the web and the internet would have on how we talk to each other. My instinct has been generally been to push back against "It's all different now" narratives, and I think my instinct has largely been borne out (but then, I would think that, wouldn't I?).

And yet, I can't believe that nothing has changed. A lot has changed. Some part of me wishes that, after nearly two decades, I could arrive at some sort of grand summing-up of What The Web Is About and what effect it's had, but after all this time, I'm not sure I have much beyond my original take: "It's not nothing, but I'm not sure what it is, except whatever it is doesn't line up that well with the hype."

Do what I say, not what I mean

2024-02-05T11:30:00.003-05:00

While watching a miniseries on ancient history, I got to wondering how quickly people could move around in those days. The scriptwriters mostly glossed over this, except when it was important to the overall picture, which seems fine, but it still seemed odd to see someone back in their capital city discussing a battle they'd taken part in a thousand kilometers away as though it had happened yesterday.

So I did a search for "How far can a horse travel in a day?". The answer was on the order of 40 kilometers for most horses, and closer to 150 for specially-bred endurance horses. That would make it about a week to cover 1000km, assuming conditions were good, except that a horse, even a specially-bred one, needs to rest.

What if you could set up a relay and change horses, say, every hour? At this point we're well off into speculation, and it's probably best to go to historical sources and see how long it actually took, or just keep in mind that it probably took a small number of weeks to cross that kind of distance and leave it at that. But speculation is fun, so I searched for "How far can a horse travel in an hour?"

It may not surprise you that I didn't get the answer I was looking for, at least not without digging, but I did get answers to a different question: What is the top speed of a horse in km/hr? (full disclosure, I actually got an answer in miles per hour, because US, but I try to write for a broader audience here). How fast a person or animal can sprint is not the same as how far can the same person or animal go in an hour.

This seems to be the pattern now that we have LLMs involved in web search. I don't know what the actual algorithms are (and couldn't tell you if I did), but it seems very much like:

Look at the query and create a model of what the user really wants, based on a Large Language Model (LLM)
Do text-based searches based on that model
Aggregate the results according to the model

It's not hard to see how an approach like this would (in some sense) infer that I'm asking "How many kilometers per hour can a horse run?", which is very similar in form to the original question, even though it's not the same question at all. There are probably lots of examples in the training data of asking how fast something can go in some unit per hour and not very many of asking how far something can go in an hour. My guess is that this goes on at both ends: the search is influenced by an LLM-driven estimate of what you're likely to be asking, and the results are prioritized by the same model's estimate of what kind of answers you want.

It's reasonable that questions like "How fast can a horse go?" or even "How fast is a horse?" would be treated the same as "How many km/hr can a horse run?". That's good to the extent that it makes the system more flexible and easier to communicate with in natural language. The problem is that the model doesn't seem good enough to realize that "How far can a horse travel in an hour?" is a distinct question and not just another way to phrase the more common question of a horse's top speed at a sprint.

I wish I could say that this was a one-off occurrence, but it doesn't seem to be. Search-with-LLM's estimate of what you're asking for is driven by the LLM, which doesn't really understand anything, because it's an LLM. It's just going off of what-tends-to-be-associated-with-what. LLMs are great at recognizing overall patterns, but not so good at fine distinctions. On the question side, "How far in an hour?" associates well with "How fast?" and on the answer side, "in an hour" associates strongly with "per hour," and there you go.

That's great if you're looking for a likely answer to a likely question, but it's actively in the way if you're asking a much-less-likely question that happens to closely resemble a likely question, which is something I seem to be doing a lot of lately. This doesn't just apply to one company's particular search engine. I've seen the same failure to catch subtle but important distinctions with AI-enhanced interfaces across the board.

Before all this happened, I had pretty good luck fine-tuning queries to pick up the distinctions I was trying to make. This doesn't seem to work as well in a world where the AI will notice that your new carefully-reworded query looks a lot like your previous not-so-carefully-worded query, or maybe more accurately, it maps to something in the same neighborhood as whatever the original query mapped to, despite your careful changes.

Again, I'm probably wrong on the details of how things actually work, but there's no mystery about what the underlying technology is: a machine learning (ML) model based on networks with backpropagation [well, transformers specifically don't use backpropagation, but it's probably in there somewhere --D.H. 25 Sep 2024]. This variety of ML is good at finding patterns and similarities, in a particular mathematical sense, which is why there are plenty of specialized models finding useful results in areas like chemistry, medicine and astronomy by picking out patterns that humans miss.

But these MLs aren't even trying to form an explicit model of what any of it means, and the results I'm seeing from dealing with LLM-enhanced systems are consistent with that. There's a deeper philosophical question of to what extent "understanding" is purely formal, that is, can be obtained by looking only at how formal objects like segments of text relate to each other, but for my money the empirical answer is "not to any significant extent, at least not with this kind of processing".

Back in the olden days, "Do What I Mean", DWIM for short, was shorthand for any ability for a system to catch minor errors like spelling mistakes and infer what you were actually trying to do. For example, the UNIX/GNU/Linux family of command-line tools includes a command ls (list files) and a command less (show text a page at a time, with a couple of other conveniences). If you type les, you'll get an error, because that's not a command, and nothing will ask you, or try to figure out from context, if you meant ls or less.

A DWIM capability would help you figure that out. In practice, this generally ended up as error messages with a "Did you mean ...?" based on what valid possibilities were close in spelling to what you typed. These are still around, of course, because they're useful enough to keep around, crude though they are.

There are now coding aids that will suggest corrections to compiler errors and offer to add pieces of code based on context. In my experience, these are a mixed bag. They work great in some contexts, but they are also good at suggesting plausible-but-wrong code, sometimes so plausible that you don't realize it's wrong until after you've tried it in a larger context, at which point you get to go back and undo it.

There's always been a tension between the literal way that computers operate and the much less literal way human brains think. For a computer, each instruction means exactly the same thing each time it executes and each bit pattern in memory stays exactly the same until it's explicitly changed (rare random failures due to cosmic rays and such can and do happen, but that doesn't really change the argument here). This carries over into the way things like computer languages are defined. A while loop always executes the code in its body as long as its condition is true, ls always means "list files" and so forth.

Human brains deal in similarities and approximations. The current generation of ML represents a major advance in enabling computers to deal in similarities and approximations as well. We're currently in the early stages of figuring out what that's good for. One early result, I think, is that sometimes it's best just to talk to a computer like a computer.

What's in a headline? Find out here

2024-02-03T15:07:00.000-05:00

Goodness, it looks like 2023 was an all-time low for this blog, with one (1) post. Not sure how that happened. I honestly thought I'd posted at least one more. On the other hand, I suppose it's consistent with the overall handwringing about whether there's even anything to post here. But this post won't be that.

When I was in journalism class in high school, which was more than a few years ago to be sure, I was taught the "inverted pyramid": put the most important information, the who, what, where, when, why and how at the top of the article, then the important detail, then other background information. The headline should concisely sum up the most important facts at the top.

Some typical headlines might be

Pat's Diner closing after 30 years
New ordinance bans parking on Thursdays
Midtown high senior wins Journalism award

If you've noticed that the titles (that is, headlines) of posts here don't exactly follow that rule, that's because I'm writing opinion here, not news. That's my story, and I'm sticking with it even as I go on to complain about other people's headlines.

One of the worst sins in old-school journalism was to "bury the lede", that is, to put the most important facts late in the story (lead as in lead paragraph is spelled lede, probably going back to the days of lead type where the usual spelling might invite confusion). If Pat's diner is closing, you don't start with a headline of Local diner closing and a paragraph about how much people love their local diners and only later mention that it's Pat's diner that's closing.

Except, of course, that's exactly what happens a lot of the time. Here are some examples from the articles currently on my phone:

Windows 11 looks to be getting a key Linux tool added in the future
Nearly 1 in 5 eligible taxpayers don't claim this 'valuable credit', IRS says
46-year old early retiree who had $X in passive income heads back to work -- here's why

I've tried to get out of the habit of clicking on articles like these, not because I think it will change the world (though if everybody did the same ...), but because I almost always find it irritating to click through on something to find out that they could have just put the important part in the headline:

Linux sudo command may be added to Windows 11
Nearly 1 in 5 eligible taxpayers don't claim earned income credit, IRS says
Early retiree with $X in passive income back to work after house purchase and child

One of these rewrites is noticeably shorter than the original and the other two are about the same length, but they all include important information that the original leaves out: which Linux tool?; which tax credit?; why go back to work?

The lack of information in the originals isn't an oversight, of course. The information is missing so you'll click through on the article and read the accompanying ads. The headlines aren't pure clickbait, but they do live in a sort of twilight zone between clickbait and real headline. If you do get to the end of the article, you'll probably see several more links worth of pure clickbait, which is an art form in itself.

Real headlines aren't dead, though. Actual news outlets that use a subscription model tend to have traditional headlines above traditional inverted-pyramid articles. They probably do this for the same reason that newspapers did: Subscribers appreciate being able to skim the headline and maybe the lede and then read the rest of the article if they're interested, and that sells subscriptions.

I'm pretty sure half-clickbait headlines aren't even new. The newspaper "feature story" has been around considerably longer than the web. Its whole purpose is to draw the reader in for longer and tempt them to browse around -- and either subscribe for the features or spend more time on the same page as ads, or both. For that matter, I'm pretty sure a brief survey of tabloid publications in the last couple of centuries would confirm that lede-burying clickbait isn't exactly new.

I started out writing this with the idea that the ad-driven model of most web-based media has driven out old-fashioned informative journalism, and also those kids need to get off my lawn, but I think I'm now back to my not-so-disruptive technology take: Clickbait and semi-clickbait aren't new, and the inverted pyramid with an informative headline isn't dead. In fact, when I checked, most of the articles in my feed did have informative headlines.

In part, that's probably because I've stopped clicking on semi-clickbait so much, which is probably changing the mix in my feed. But it's probably also because the web hasn't changed things as much as we might like to think. All three kinds of headline/article (informative, semi-clickbait, pure clickbait) are older than the web, and so are both the subscription and ad-based business models (though subscription print publications often had ads as well). It's not too surprising that all of these would carry through.

On web standards and social networking

2023-01-09T18:28:00.005-05:00

I started this blog, um, a few minutes ago, back when I was involved with a couple of the committees involved in developing standards for the web, and it seemed like one thing a person in my position was supposed to do was to blog about it, whether to try to make one's expertise generally available, or to promote one's brand or consulting services, or to become visible as part of a group of similar individuals (the term "blogroll" comes to mind), or to help set the future direction of the web by presenting brilliant analyses or ... something of that nature.

Fortunately, I soon separated the blog from my professional life and settled into just putting out whatever thoughts I had as I wandered the web.world, without a lot of concern for what, if anything, it might all lead to. In the event, it hasn't led to all that much ... a few dozen followers, the occasional comment, and readership well below even thinking about trying to monetize anything ... but it has been a satisfying experience nonetheless. I've learned all sorts of things, and writing about it has given me the opportunity to think things through, organize my thoughts and maybe even get better at writing. The whole standards-committee thing seems long ago and far away.

Imagine my surprise, then, when an Ars Technica article on Mastodon popped up in my newsfeed and brought it all back.

When we last left our story, I was musing about how recent turbulence concerning cryptocurrencies and social media, and Twitter in particular, had gotten me thinking about what "web" even meant anymore and concluding that at least the core of the web, namely its interconnections, was alive and well, and also that it didn't have all that much to do with the turbulence. Before that, I had said that there didn't really seem to be all that much to blog about. Things were happening, but not a lot of things that I had much to say about.

But Mastodon is interesting, not just because of its role as a potential "giant killer" for Twitter -- I'm still not inclined to speculate on how that might shake out -- but because of what it is and how it was put together, both in its structure and in the process that led to it. Allow me to take a walk down memory lane:

When I was involved in the standards process, the typical web standard went through a life cycle something like this:

Someone, typically one of the Major Players, would create some new protocol or class of application to try to take advantage of the opportunities to be had on the rapidly expanding web. This was after the major protocols like TCP, FTP, HTTP and the various email protocols were widespread. Those tended to be less blatantly commercial, even though major corporations were generally involved.
If the idea looked useful, other players, large and small, would try to work with it, using whatever documentation was available.
This would often lead to a mess. It's hard to write clear, complete documentation and it's almost guaranteed that once other people start using a system they'll try things that the original authors hadn't thought of. When it's not clear what should happen in a particular situation, people will take their best guess, and different people will guess differently.
People will also want to add new features to cover cases that the original authors hadn't thought of, or did think of but didn't have time to implement, or implemented in what seemed like a less-than-optimal way, or whatever.

In some ways, this is a good problem to have, since it means people are actually using the system, probably because it meets some need that hadn't been addressed before. The flip side, though, is that in a networked world a system only works, or at least only works well, if people agree on how to use it. If my version of AwesomeNet thinks that the command for "make everything awesome" is MKAWSM and yours thinks it's make-awesome, then chances are things won't be so awesome.

There are plenty of other minor disagreements that can happen, either because different people made an arbitrary choice differently, or because there's a genuine disagreement as to the best way to do something, or for any number of other reasons. If this happens enough to be a real problem, there's usually a call for everyone to get together and create an agreement on which way to do things. A standard, in other words.

Writing standards is a tricky thing. You want to capture what people are currently doing, but in an abstract enough way to avoid just having a laundry list of things people currently do. You want to establish boundaries that are tight enough to rein in the chaos, but not so tight to prevent people from finding new and better ways of doing things. There are often several ways to do this. For example, for the difference in command names you might

Just pick one, say make-awesome, and "deprecate" the other, meaning that implementations of AwesomeNet aren't required to support it and if your code does use it, you should update your code.
Pick one as preferred, require implementations to understand the other one, but forbid them from using it. That means that a standard implementation of AwesomeNet will never use MKAWSM, but if someone talking to it does, it will understand (this is an example of Postel's principle).
Allow both, usually by defining one as an "alias" for the other, so that most of the standard text can just talk about make-awesome, except for the part that says "you can use MKAWSM anywhere you can use make-awesome"
Define an "abstract syntax" and talk about <make awesome> as a placeholder for "whatever name the implementation chooses to use for the make everything awesome command". This means that there will need to be some sort of protocol for two implementations to tell each other which names they actually use. That's probably not worth the trouble in most cases, but it's an option nonetheless.
If "make everything awesome" is just a convenient shorthand for a bunch of other commands, define a general way of defining shorthands like that (often called "macros") and leave it out of the "core standard" entirely. There are lots of ways to implement macros, since a macro language is essentially a programming language (even if it's not functionally complete), so this is probably only a good idea if at least some of the existing implementations already support it.
There are probably several other approaches I'm not thinking of.

There are similar issues with features that only some of the existing implementations support. Do you require everyone to support everyone's features? Do you define a "core set" and make the rest optional? Do you say that some existing features aren't allowed in the standard, so implementations that have them have to drop them? Or maybe something else?

Another important question is "what happens in this particular special case?". Sometimes the answer is "it's this particular behavior that you might not have thought of, and here's why". Sometimes the answer is "that's an error, and the implementation should report it like this". Sometimes, though, the answer is "the behavior is unspecified", either to leave room for new features or for the practical reason that different existing implementations do different things, or for whatever other reason. This is more useful than it might seem. It tells implementers that it's OK to do whatever's easiest, and not to expect any particular behavior in certain situations (and maybe try to avoid those situations altogether).

There are lots of other principles and techniques that experienced standards writers employ (learning some of these as a not-so-experienced standards committee member was one of the best parts of the experience). One that sticks in mind is the principle that implementations should be able to safely ignore things they don't understand, so that it's safe to interact with something that implements a newer version of the standard.

In a typical standards process, there are dozens of questions to hammer out, and the answers are often interconnected. It's a tricky technical problem and, of course, there are political considerations as well. In my experience the politics are typically pretty civil, but most participants come in with certain "red lines" they won't cross, usually because it would cost more to implement them than the standard is worth to them since there are already customers out there using the not-yet-standardized products. The usual approach is to stand firm on those and be prepared to be flexible on anything that doesn't cross a red line. If two or more participant's red lines overlap, things can get tricky.

After all the meetings, if all goes well, the end result of this is a carefully crafted standards document full of MUST [NOT] and SHOULD [NOT], which are so important that there's actually a standard defining what they mean.

The new standard captures everything the committee could capture about what it means to implement and use whatever's being standardized. With luck, the standard is at least self-consistent, but there will be parts that are deliberately left unspecified for reasons like those given above. There will also be plain old mistakes, ideally small ones that can be corrected by errata (corrections, ideally short and specific, published in a separate section or document) but sometimes larger ones that will require more meetings and a new version of the standard.

Since standards need to be flexible, a standard doesn't always answer every question you'll need answered when coding to it. Because of this, it's common for a separate "interoperation committee" to form once a standard is out. These committees essentially fill in the blanks to create "profiles" that say how to use a particular standard in particular situations. There might be a lightweight profile that's meant to provide only the core features so it's easier to implement and run. There might be a security-oriented profile that specifies features to enable and parameters to use ("digital signatures are required for each message and private keys must be at least 2048 bits"). There might be a profile aimed particularly at business use, or whatever.

I've gone through all this in detail because, judging by the Ars article, all of these things either have happened with Mastodon, or may happen at some point because the same basic issues have come up. More than that, Mastodon and similar services are, in fact, based on the ActivityPub standard, which specifies two protocols, one for clients to talk to instances to interact with an inbox and outbox, and one for instances to send traffic to each other.

(An instance is a server in the sense of the client-server model, but not in the sense of a particular process or a server machine -- at least in theory, an instance could run on multiple physical servers and a single physical server could host multiple instances)

ActivityPub is meant to be very general-purpose, but in the context of social networking only some of the features are really important, so implementations in the Fediverse (the world of Mastodon and Mastodon-like services that talk to each other) will naturally tend to pay more attention to those features, which is probably fine until someone tries to take advantage of other features. The standard for data encryption and signatures is not officially part of the system, so implementations that support it still have to deal with unencrypted data. ActivityPub relies on a standard called WebFinger to discover what instances are out there, so implementing a real ActivityPub instance also means dealing with WebFinger.

And so on.

All of this is perfectly normal in the world of web standards. A standard is meant to reduce uncertainty and make it clear where the remaining uncertainty is, and establish common vocabulary and practices. It isn't meant to solve all problems with interoperation -- it's great if it can, but that's not always possible. It isn't meant to answer all questions an implementer might have, and in fact it deliberately doesn't answer some questions so that implementers will be free to come up with better answers. In general a standard is more about what than how (though a what in one context can be part of the how in another and vice versa).

In any case, ActivityPub only talks about how instances talk to each other and how clients interact with instances. It doesn't, and shouldn't, talk about how people using it for social networking should interact. That's a matter of community guidelines, that is, what the people running the actual instances will or won't allow on them. Since the instances are deliberately federated, rather than owned by one entity as with Twitter, these decisions will be, too.

One particularly interesting point is whether posts to one instance can be kept private to that instance. This proved contentions, so Hometown was forked off of Mastodon. Hometown allows posts to be private to a particular instance. Mastodon doesn't.

The good news, I think, is that federated services built on web standards, while messy in real life, can and do work. Mastodon's success or failure depends more on how community guidelines shake out and whether people prefer to spend their time there than it does on the strength of the standards. I only spent a lot of time on standards in this post because it was striking to me how much of what I learned back in those days is still relevant.

One other thing struck me while pondering the article. This all seems a whole lot like Usenet, just with different underlying technology.

Usenet used an email-based model where a particular user could subscribe to various newsgroups, receiving emails as updates came in, and publish messages on those groups, either new messages or replies to a thread that was already going. The main technical difference is that early usenet relied on a old protocol called UUCP (Unix-to-Unix Copy) to transfer files of messages point-to-point between sites (often but not always universities or corporations). UUCP isn't used much any more, but, like many old standards, it is still in use here and there.

The main externally visible difference, besides the use of email, was that the newsgroups were owned by administrators (these might not be the server administrators, though server administrators could effectively veto a group by filtering it out). Rather than tag a post with #this or #that, you'd post to comp.sci.this or alt.that.discuss or whatever.

From a social point of view, though, the similarities seem significant. Just as site administrators could establish rules, such as by blocking people from posting or filtering out newsgroups, Mastodon instance owners can establish their own standards. Just as different people would have different ideas as to what sort of discussion was acceptable and different sites and different newsgroups could have different standards, different instances have different rules of conduct.

Newsgroups allowed people with similar interests to meet up and discuss things they were interested in. This included not only technical topics like scientific specialties or programming languages, but social topics like music and art and a fair bit of quite gamy content. They also allowed otherwise marginalized people to gather together, and for the curious to find out a bit about an unfamiliar group of people, but also for trolls and worse to barge in if they could get past the admins.

In other words, from a social point of view, they had pretty much all the features, good and bad, of modern social networking communities. Whether you want to see that as a glass half full -- people keep finding ways to get together -- or half empty -- even after decades we're still up against the same problems -- or a bit of both -- my general inclination -- is up to you. If nothing else it might provide a counterweight to claims that any of what's going on now is unprecedented.

There's one other technical point that jumped out, though I doubt it will interest many people. Two patterns of communication come up over and over again in distributed services. One is the request/response pattern: I ask you a question or to do something, you respond with an answer or "I did it (and this happened)"/"I couldn't do that (for this reason)"/.... The other is the publish/subscribe pattern (pub/sub for short), where any number of parties can publish a message on a topic and any of those (including the publisher) can subscribe to messages on that topic. Those aren't the only possibilities, but they account for a lot of traffic. ActivityPub, as the name might suggest, is a pub/sub protocol.

The whole point of pub/sub is that publishers and subscribers don't know about each other, or whether any of them even exist. I can publish to a topic that no one's listening to, and, depending on the type of service, either that message will be dropped on the floor, or it will be saved for delivery to anyone who does eventually subscribe. In either case, each subscriber will see its own copy of the message. The only difference is whether "each subscriber" means "each subscriber that was there when the message was published (more or less)" or "each subscriber that ever subscribes to this topic".

Since publishers and subscribers don't know about each other, there has to be some sort of intermediary. For a large system, there will actually be many intermediaries communicating with each other in order to make sure published messages are delivered to subscribers. Publishers and subscribers talk point-to-point with those intermediaries. I tend to think of this as "I publish to the cloud, whatever's in the cloud does whatever it does, and the subscriber gets the message from the cloud".

Mastodon and company fit this perfectly: You send a message to the instance you're using, it talks to other instances, and people see your message.

The old Usenet followed the exact same pattern, just that the mechanism for the servers to talk to each other was quite a bit different.

Goblin Mode McGoblin Modeface

2022-12-28T16:19:00.010-05:00

Each year, Oxford Languages, which produces the Oxford English Dictionary among other things, selects a word of the year, "a word or expression reflecting the ethos, mood, or preoccupations of the past twelve months, one that has potential as a term of lasting cultural significance." This year, the choice was opened up to online voting. Over 300,000 people cast their votes over the course of two weeks and the winner was goblin mode

a slang term, often used in the expressions ‘in goblin mode’ or ‘to go goblin mode’ – is ‘a type of behaviour which is unapologetically self-indulgent, lazy, slovenly, or greedy, typically in a way that rejects social norms or expectations.’

The runners up were metaverse and #IStandWith.

The press I've seen about this tends to emphasize the online voting aspect of the selection, with the suggestion that those rowdy internet folks got one over on the stodgy old OED, but I think that misses a couple of important points.

First, the OED as an institution isn't particularly stodgy. While Oxford might suggest the British power structure or Tom Lehrer's indelible image of "ivy-covered professors in ivy-covered halls" (leaving aside that that's more a US reference), the dictionary itself has historically been concerned with documenting how people actually use the English language, rather than trying to dictate what "proper English usage" might be. It is descriptive rather than prescriptive.

The dictionary styles itself "the definitive record of the English language". This is meant to include everything, including dialects from all over the world, terms of art for all kinds of trades and professions, archaic words from a thousand years ago and all manner of other English usage, including today's internet slang.

From the OED's point of view, goblin mode is a perfectly good term to research and define, as is anything else that people actually use. If a bunch of internet trolls had decided to vote for glurglebyte or some other made-up word, and the OED actually went with it, that would have been a different matter, but there are plenty of examples of people using goblin mode prior to the online vote. The word of the year page even gives a couple of examples from The Grauniad and The Times.

One might argue that people weren't using goblin mode all that much, and some other term, whether metaverse, #IStandWith or something else, might have made a better word of the year, but the fact that hundreds of thousands of people voted for it suggests that, even if the votes were meant ironically, there's something there that led people to coalesce around that particular word. You could even argue that the online vote gives an otherwise ordinary bit of internet slang a much better chance of becoming "a term of lasting cultural significance".

The word of the year page goes further and argues that goblin mode is indeed a good word for a year in which people are finding their way out of a world of lockdowns and overflowing hospitals and questioning just which pre-pandemic norms are really worth keeping. Sure, the Oxford folks may just be trying to put a brave face on being pwned, but to me it seems more like they saw the results and weren't particularly bothered.

I think there's another important point to note here. While there have been plenty examples of internet-driven crowds doing bad things, or even horrible things, it's worth remembering that this particular crowd of net.denizens was operating from a completely different mindset: As with Boaty McBoatface, they did it because it was fun, for sheer hack value.

While it would be a mistake to ignore bad behavior, it would also be a mistake to over-focus on it. Like anywhere else, bad things can happen on the web without making the whole place a cesspit. There's lots of questionable content out there and a certain amount of outright lies and hate, but there's also a lot of good information and not a little outright goofiness. However much there are people out there trying to steer us toward conflict and anger, we still have choices about what we browse and what we post.

A few hundred thousand people upvoting a random bit of slang may be a drop in the bucket, but there are a lot more drops like it. That says something about who's out there and what they want, just as surely as the nastiness elsewhere does.

Is it the end of the Web as we know it?

2022-11-25T13:59:00.003-05:00

Or maybe a better question is "What is this Web we speak of, anyway?" My default answer: dunno, I'm figuring it out as I go along.

I think the last time I mulled that second question over, in the context of "Web 2.0" (Remember Web 2.0? I think it was one of the exits on the Information Superhighway), my opinion was that the big division was between everything that came before and "the Web", or "Web 1.0" as I don't recall anyone calling it very much. In other words, that first time someone chased a link from one web page to another using a graphical browser was an epochal event, even if hardly anyone noticed at the time, and what's come after has been a steady stream of technical improvements and services founded on that base.

Two types of service in particular have been prominent over the last decade or so: social media and cryptocurrencies, and both seem to be in questionable shape at the moment. I've cast a somewhat skeptical eye on both over the years, but that hasn't stopped them from intersecting with the lives of billions of people.

Billions in the case of social media, at least. I don't actually know how many people own cryptocurrencies, directly or indirectly, but who among us hasn't seen an ad for one or another, or read about the latest crash/rugpull, not to mention the millions of people living in countries that have made cryptocurrencies a significant part of their monetary system, so I'd say billions there, too, depending on how you count.

But the past year has not been particularly kind to either. This is all over the news at the moment, but just for later reference, let me list a few items of note

Elon Musk's takeover of Twitter is off to a rocky start. My guess is that the new ownership will find some way to keep the servers running and reach some sort of new equilibrium, but with a sizable majority of the workforce either forcibly terminated or choosing "take the severance and get on with my life" over hardcore intensity, it's safe to say there will be a period of adjustment. Major advertisers seem to be sitting on the sidelines in the meantime and, thanks to the billions in debt that came with the leveraged buyout, the burn rate has increased from "we'll be out of cash on hand in a couple of years if nothing changes" to "we'll owe more in interest this year than we have in the bank"
Facebook seems to have wandered off into the Metaverse. This seems to me to be a classic case of optimistic extrapolation run amok. Virtual reality is interesting technology. It clearly at least has good potential for useful applications in areas like design and education. Getting from there to a world where people spend comparable amounts of time in the virtual world to what they currently spend on scrolling through their feeds seems like a stretch. Personally, I've tried out an Oculus, and there were definitely some cool things on offer, from a deeply moving immersive art piece on refugees to super slow-mo of a couple of guys making showers of sparks that you can walk around in. But the age of those links should tell you how long ago that was.
No less than Ian Bogost, of Cow Clicker fame among many other things, has written an article entitled The age of social media is ending. It should never have begun. I'm incorrigibly skeptical about proclamations of the End of an Age, or the beginning of one for that matter, but Bogost makes some good points about the crucial distinction between social networking (good, and computers can be very helpful) and social media (the never-ending pursuit of clicks, shares, followers, content and so forth, not so good in Bogost's estimation).
Crypto exchange FTX has imploded, taking SBF (its colorful founder Sam Bankman-Fried) down with it, the latest of many crypto plays that turned out, shockingly, to have been built atop a house of cards.
Bitcoin, the grandaddy of them all, has fallen from its all-time high of close to $69,000 to, at this writing, around $16,000, down over 75%. Interestingly, the price of BTC had pretty closely tracked the price of the S&P 500, leveraged about 3:1, until the recent FTX fiasco sent it further down. What it didn't do was rise as reserve currencies hit a round of inflation, which as I dimly understand it was what was supposed to happen.
The whole advent of crypto exchanges has only emphasized the disconnect between cryptocurrency in theory -- decentralized, anonymous, free from government interference -- and practice -- centralized by exchanges and mining pools, generally tied to bank accounts in reserve currencies and subject to government regulation from several directions.

Plenty of cold water to be thrown on social media and cryptocurrency enthusiasts, but does this mean the whole thing is coming to an end?

Social media doesn't seem to be going away. There's even been a rush of activity on Twitter, speculating about the demise of Twitter and what to do next, and if you want to use that as a jumping-off point for a rant about modern culture eating itself, be my guest.

Even if cryptocurrency is dead as an alternative to reserve currencies and more conventional payment systems -- I'm not saying it is or isn't, but even if -- I doubt it's going to stop trading anytime soon. My personal benchmark for "crypto is dead" would be something on the order of "I can personally mine and take ownership of 1 BTC using my phone at a nominal cost". We're quite a ways from that, but on the other hand there's still plenty of time left before the mining reward rounds down to zero sometime around the year 2140 at current rates.

In short, there are certainly some major disruptions going on in some of the major features of the Web landscape, but, in answer to the question in the title, they seem more like the kind of shakeup or reining in of excess that seems to happen fairly regularly, rather than some sort of deathblow to the Web itself. Webvan, anyone?

But then, as I asked at the top of the post, what is this Web we speak of, anyway?

Apart from the time constraints of a busy life, I've been a less apt to post here, and in fact started a whole other blog (which I also don't post on very frequently), because I had come to the conclusion that a lot of things I wanted to post about weren't really related to the Web. Even here, one of my more recent posts was me fretting about what even is the Web any more and why am I not writing about it?

That post, though, mainly talked about what the Web means day to day. For better or worse, a lot of that has to do with social media, and I have no interest in devoting a large chunk of my time to what's going on in social media. Plenty of other people do want to do that and do a better job than I would. But what is it that makes the Web webby, and how does that relate to the Web as it impacts our lives?

If you peel back all the layers, all the way back to that first link chased on that first graphical browser, the Web is about links. If you've ever meandered from one Wikipedia article to the next, following links in the page or the "see also", you've been using the Web at its webbiest. Likewise, I think, if you've browsed your favorite magazine and followed the links from one article to the next, within that publication or outside. The web of interconnections is what makes the Web.

That primordial web is still around and likely isn't going anywhere, because this sort of browsing from one topic to the next is probably pretty tightly wired in to the way our brains work. What has happened is that a couple of layers have grown on top of it.

One is search. You can find all sorts of interesting things by browsing, but often you just want to know where to find, say, a replacement battery for your cordless vacuum. Browsing would be a horrible way to go about that, but you don't have to. Just type some likely terms into your search bar and there you are. This is useful enough that companies can make quite a bit of money by running ads on a search platform, and I doubt this business model is going away, whatever the fortunes of the particular companies providing it.

Social media constitutes a different layer on top of the web. As I've mentioned before, I'm not active on social media, but it seems to me that while you can certainly browse the links of your social network to find people that people you know know, and you can follow links from a post/tweet/story/whatever to more things that you might be interested in, the main innovation in social media is the feed, which brings content to you without your having to search for it or stumble onto it.

This isn't limited to social media. I spend quite a bit of time reading my news feed, anti-social though that may be. In any case, I think there is a distinction to be made between information you actively seek out and information that some person you're following, or some algorithm, or some combination of the two, brings to you. I doubt that this is going anywhere either, but it looks like there is some rethinking going on about how to control the feed of incoming information, and, to some extent, how much attention to pay to it at all.

Interestingly there was a lot of interest a while back in social search, where you could ask questions of the crowd and people would dig up answers, and people would get paid, and various companies would take various cuts, one way or another. I think that fell by the wayside because automated search does a better job in many cases, and when it doesn't, asking someone you know without anyone in the middle generally works fine, or at least no worse than trying to ask a pool of random people.

Also interesting: Nothing in those last few paragraphs involves cryptocurrencies, even though I implied earlier that upheaval in that world might have something to do with "the end of the Web as we know it". I think that's because, even if stories about cryptocurrency have been all over the web, cryptocurrency itself doesn't have much to do with the Web, because it just isn't webby in that primordial sense. Following some sort of network of transactions, link to link, is not exactly played up as a major use case.

I've actually found working through this pretty encouraging. A few posts ago (that is, over a year ago), I was ruminating on whether there was anything webby left that I might want to talk about. Going back to first principles about what makes the Web the Web immediately revealed a view in which the very basis for the Web is alive and well, and aspects of it that are prominent now, like search and feeds, can at least be understood in relation to it.

Dear screenwriters: Bits can be copied

2022-07-30T13:08:00.007-04:00

There's a new thriller movie out on one of the major streaming services. I don't think it matters which movie or which service. If you're reading this years from now, that statement will still probably true, at least to the extent there are still streaming services. If you're pretty sure you know which 2022 movie this is referring to, but haven't seen it yet and want to, be warned. There are mild spoilers ahead.

As with many such films, the plot revolves around a MacGuffin, a term apparently coined by Angus MacPhail, which Alfred Hitchcock famously glossed as "the thing that the spies are after, but the audience doesn't care." In other words, it doesn't really matter what the MacGuffin actually is, only that the characters do care who gets it and so spend the whole film trying to make sure it ends up in the right place and doesn't fall into the wrong hands.

The plot device of a MacGuffin is much older than the term itself, of course. The Holy Grail of Arthurian legend is one, and the oldest recorded story known so far, The Epic of Gilgamesh, sends its protagonist to the Underworld in search of one.

Clearly there's something in the human brain that likes stories about finding a magic item and keeping it away from the baddies, and in that sense the MacGuffin in the big streaming service movie is a perfectly good MacGuffin. The protagonists and antagonists vie over it, it changes hands a few times, lots of things explode and eventually the MacGuffin is destroyed, ending its magic powers.

Except ...

The MacGuffin in this case is basically a gussied-up thumb drive containing information certain people do not want to become known. Our protagonist receives the item early in the film (with suitable explosions all around) and promptly sends it off to a trusted colleague for safekeeping and decipherment. Later we learn that the trusted colleague has, in fact, received the drive and cracked its encryption, revealing the damning information.

In real life, this is when you would make a backup copy. Or a few. Maybe hidden in the insignificant bits of JPEGs of cute kittens on fake cloud accounts with several different services. Maybe on some confederate's anonymous server somewhere on the dark web. Or at least on a couple more thumb drives. For bonus points, swap out the contents of the original thumb drive for a clip of the Dancing Baby or some similar slice of cheese.

(As I understand it, there are some encrypted devices that are tamper-resistant and designed not to be readable without some sort of key, so you can't easily copy the encrypted bits and try to crack the encryption offline, but here we're told that the encryption has already been cracked, so they have the plaintext and can copy it at will.)

The problem with that, of course, is that the drive would then cease to be a MacGuffin. Why send teams of mercenaries and a few truckloads of explosives after something that might, at best, be one copy of the damning information? The only real reason is that it makes for an entertaining way to spend an hour or two and screenwriters know all about writing MacGuffin-driven thriller plots.

Which is fine, except ...

If you think about the practicalities, there's still plenty of tension to be had even if the bits are copied. Our protagonist has reason to want the secret information to remain secret except in case of a dire emergency, but they also want to be able to preserve it so that it can be released even if something happens to them. How to do this?

If you've uploaded the bits to one of the major services, then who gets access to them? Do you keep the information in a private file, memorize the account password and hope for the best? What if you're captured and coerced into giving up the password? On the other hand, if you die without revealing the information, it will just sit there until the account is closed, unless someone can figure out enough to subpoena the major service into handing over access to a bunch of cat pictures hiding the real information. Which you encrypted, of course, so who has the key?

Maybe you share the encrypted bits with a journalist (or two, or three ...) with an "in case of my death" cover letter saying where to get the encryption key. But what if they decide to go public with it anyway? The more journalists, the better the chance one of them will publish if something happens to you, but also the better the chance that one of them will publish anyway.

Maybe you put the encrypted bits someplace public but write the encryption key on a piece of paper and lock it away in a safe deposit box in a Swiss bank. Now you've traded one MacGuffin for another. But maybe someone at a different spy agency has a backdoor into your encryption. The baddies at your own agency are going to keep the contents to themselves, but maybe one of them has a change of heart, or gets double-crossed and decides to go public as revenge, and they need your copy since they no longer have access to the original bits and didn't make their own copy.

And so forth. The point is that information doesn't really act like a physical object, even if you have a copy in physical form, but even so there are lots of ways to go, each with its own dramatic possibilities depending on the abilities and motivations of the various characters. Most of these possibilities are pretty well-used themselves. Plots driven by who has access to what information have been around forever, though some have paid more attention to the current technology than others -- "Did you destroy the negatives?" "Yes, but I didn't realize they'd left another copy of the photographs in a locker at the bus station ..."

Opting for a bit more realism here gives up the possibility of a "destroy the magic item, destroy the magic" plot, but it opens up a host of other ones that could have been just as interesting. On the other hand, the movie in question doesn't seem to blink at the possibility of a full-on gun battle and massive explosions in the middle of a European capital in broad daylight. Maybe realism was never the point to begin with, since that seems pretty unlikely.

Oh, wait ...

Check out this new kitchen hack!

2022-06-02T12:23:00.001-04:00

In case that title somehow clickbaited you to this quiet backwater, no this isn't really about cooking, but for your trouble: The easiest and least tearful way I know to slice onions is to cut them in half lengthwise, so each half has a little piece of the roots holding it together. If you think of the roots as the South Pole and the stem end as the North Pole, the first slice is from pole to pole.

Chop off the stalk end and peel off the outer layers, under cold running water if that seems to help (I think this is a little easier than slicing the stem off first, but your mileage may vary). Put the halves down on the flat side and slice vertically with the slices parallel, also running north-south. Julia Child recommends another pass, horizontally, still slicing north-south, and who am I to argue? At this point, the root and the shape of the onion layers are still holding everything together. Finally, slice vertically, but with the slices running east-west. Each cut slices off a little pile of nicely diced pieces.

This isn't new -- I first heard about it on a Chef Tell segment many years ago, Mastering the Art of French Cooking came out in 1961 and I'm sure it's been around much longer -- but it works a charm. Bon Apetit, and remember that a dull kitchen knife is more dangerous than a sharp one.

So it's not new, but is it a hack? And what's with all these "life hack" articles that have nothing to do with writing clever code?

For my money, the onion-dicing method is absolutely a nice hack. A hack, really, is an unexpected way of using something to solve a problem. The usual way to dice something is to slice it, then cut the slices crosswise into strips, then cut the strips crosswise into little dice. If you try that with an onion, the root is in the way of the north-south slices described above, and the easy way to start is to slice it east-west, into rings. You then have to dice up the rings, which are hard to stack since they're already separated, and like to slide around and separate into individual rings, and have a lot of exposed surface area to give off tear-producing onion fumes. In short, you have a mess.

The chef's method takes advantage of the two things that otherwise cause problems: It uses the root end to hold things in place and keep the exposed area to a minimum, and it uses the layering of the onion to save on cutting (if you omit the horizontal slices, as I usually do, you still get decently-diced pieces, good for most purposes, just a bit coarser). This is the essence of a hack: using something in a non-obvious way to get the result you want. It's particularly hackish to take advantage of something that seems to be an obstacle.

Not every hack is nice, of course. The other popular meaning of hacking, that many geeks including myself find annoying, the computing analog of breaking and entering or vandalizing someone's property, stems from a particular type of hacking: finding unexpected vulnerabilities in a system and taking advantage of them to break the system's security. As I've discussed at length elsewhere, this isn't necessarily bad. White hat hackers do just this in order to find and patch vulnerabilities and make systems more secure. The annoying part isn't so much that hack is associated with breaking and entering, but that it's associated with any kind of breaking and entering, regardless of whether there's any skill or actual hacking -- in the sense of making unexpected use of something -- involved.

I should note somewhere that hack often has negative connotations in software engineering for a completely different reason: If you take advantage of some undocumented feature of a system just to get something working, you have a fragile solution that is liable to break if the system you're hacking around changes in a future update. In widely-used systems this leads to Hyrum's law, which basically says that people will write to what your system does, regardless of what you say it does, and with enough people using it, any externally visible change in behavior will break someone's code, even if it's not supposed to.

Hacking lives in gray areas, where behavior isn't clearly specified. "Dice this onion with this knife" doesn't say exactly how to dice the onion. Someone taking advantage of a quirk in an API can usually say "nothing said I couldn't do this". There's nothing wrong with unspecified behavior in and of itself. It's actively helpful if it gives people latitude to implement something in a new and better way. The trick is to be very specific about what can happen, but put as few restrictions as possible on how.

There's an art to this. If you're writing a sorting library, you could say "It's an error to try to sort an empty collection of things". Then you have to make sure to check that, and raise an error if the input is empty, and whoever's using your library has to be careful never to give it an empty collection. But why should it be an error? A collection with only one thing in it is always sorted, since there's nothing else for it to get out of order with. By that reasoning, so is an empty collection. If you define sorted as "everything in order", that raises the question "but what if there isn't anything?".

If you define sorted as "nothing out of order -- no places where a bigger thing comes before a smaller thing", then the question goes away. If there isn't anything in the collection, nothing's out of order and it's already sorted. In math, something is vacuously true if there's no way to make it false. "Nothing out of order" is vacuously true for an empty collection. Often, allowing things to be vacuously true makes life easier by sidestepping special cases.

As a general rule, the fewer special cases you need to specify what happens, the easier a system is to write and maintain, the more secure it is against unwanted forms of hacking like security exploits and Hyrum's law, and the friendlier it is to good kinds of hacking, like people finding clever new ways to improve the implementation or to use the system.

So what about all this "life hacking"? Should people use computing jargon for things that have nothing to do with computing? I have two answers.

First, the term hack isn't really about computing. It's about problem solving. The first definition in the Jargon File (aka Hacker's Dictionary) is "Originally, a quick job that produces what is needed, but not well.", with no mention of computing, and elsewhere it attributes early use of the term to ham radio hobbyists. As it happens, the actual definitions of hack in the Jargon File don't really include "using something in a non-obvious way to get the result you want", but I'd argue that the definition I gave is consistent with the The Meaning of 'Hack' section.

Second, though, even if hack was originally only applied to coding hacks, so what? Language evolves and adapts. Extending hack to other clever tricks reveals something new about what people are trying to get at by using the word, and in my view it's a lot better than restricting it to security exploits, clever or not. Sure, not every "kitchen hack" or "life hack" is really that hackish, and headline writers are notoriously pressed for time (or lazy, if you're feeling less generous, or more apt to make money with clickbait, if you're feeling cynical), but there are plenty of non-computing hacks floating around now that are just as hackish as anything I've ever done with code.

Why so quiet?

2021-10-28T13:26:00.007-04:00

I hadn't meant for things to go so quiet here, and it's not just a matter of being busy. I've also been finding it harder to write about "the web", not because I don't want to, but because I'm just not running across as many webby things to write about.

That got me thinking, just what is the web these days? And that in turn got me thinking that the web is, in a way, receding from view, even as it becomes more and more a part of daily life, or, in fact, because it's more and more a part of daily life.

There is still plenty of ongoing work on the technical side. HTML5 is now a thing, and Adobe Flash is officially "end of life" (though there's a bit of a mixed message in that Adobe's site for it still says "Adobe Flash Player is the standard for delivering high-impact, rich Web content." right below the banner that says "Flash Player’s end of life is December 31st, 2020"). Microsoft has replaced Internet Explorer with Edge, built on the Chromium engine. Google is working to replace cookies. I realize those are all fairly Google-centric examples, and I don't want to imply that no one else is doing important work. Those were just the first examples that came to mind, for some strange reason.

On the one hand, those are all big developments. Adobe Flash was everywhere. It's hard to say how many web pages used it, but at the peak, there would be on the order of billions of downloads when Adobe pushed a release, because it was in every browser. Internet Explorer was the most-used browser for over a decade, and the standard browser on Windows, which would put its user base in the billions as well (even if some of us only used it to download Chrome). Somewhere around 20% of web sites, however many that is, use cookies.

On the other hand, they are all nearly invisible. I can remember a few times, early in the process a couple of years ago, when Chrome wouldn't load some particular website because Flash was disabled, but not enough to cause any real disruption. I'm sure that the shift from Explorer to Edge was disruptive to some, but when I set up a laptop for a relative a little while ago, they were much more concerned with being able to check email, write docs or play particular games than which browser was making that happen. As for cookies, I haven't looked into exactly how they're being replaced, because I don't have to and I haven't made time to look it up.

Because the web is everywhere, the huge number of websites and people browsing means that it's most important to keep everything running smoothly. Unless you're introducing some really amazing new feature, it's usually bad news if anyone knows that you made some change behind the scenes (whatever you think of Facebook as a company, please spare a thought for the people who had to deal with that outage -- even with a highly-skilled, dedicated team keeping the wheels turning, these things can happen, and it can be devastating to those involved when it does).

The upshot here is that I don't really have much interesting to say about much of the technical infrastructure behind everyday web experience. Besides not having been close to the standards process for several years, I figured out very early that I didn't want to write about the standards and protocols themselves -- there are plenty of people who can do that better than I can -- but how they appear in the wild. Thus the field notes conceit.

It was interesting to write about, say, Paul Vixie's concerns about DNS security or what copyrights mean in the digital age, but topics like that seem less interesting today. Regardless of the particular threats, the real benchmark of computer security is whether people are willing to put their money on the web -- buy, sell, send money to friends, check their bank statements or retirement accounts, and so forth. That's been the case for a while now, through a combination of security technology and legal protections. Importantly, the technology doesn't have to be perfect, and a good thing, that.

The question of how creators get paid on the web is still shaking out, but one the one hand, I think this is one of those problems that is always shaking out without ever getting definitively resolved, and on the other hand, I'm not sure I have anything significant to add to the discussion.

As much as I don't want to write a purely technical blog, I also don't want to lose sight of the technical end entirely. I'm a geek by training and by nature. The technical side is interesting to me, and it's also where I'm most likely to know something that isn't known to a general audience.

Obviously, a lot of the important discussion about the web currently is about social media, but I don't want to jump too deeply into that pool. Not only is it inhabited by a variety of strange and not-always-friendly creatures, but if I were commenting on it extensively, I'd be commenting on sociology, psychology and similar fields. I muse about those on the other blog, but intermittently conjecturing about what consciousness is or how language works is an entirely different thing from analyzing social media.

Even so, Twitter is one of the top tags here, ironic since I don't have a Twitter account (or at least not one that I use).

My main point on social media was that some of the more utopian ideas about the wisdom of crowds and the self-correcting nature of the web don't tend to hold up in practice. I made that point in the context of Twitter a while ago, in this post in particular. I wasn't the first and I won't be the last. I think it's pretty widely understood today that the web is not the idyllic place some said it would be a few decades ago (not that that kept me from commenting on that very topic in the most recent post before this one).

On the other hand, it might be interesting to look into why the web can be self-correcting, if still not idyllic, under the right circumstances. Wikipedia comes to mind ...

Finally, I've really been trying to keep the annoyances tag down to a dull roar. That might seem a bit implausible, since it's generally the top tag on the list (48 posts and counting), but in my defense it's fairly easy to tell if something's annoying or not, as opposed to whether its related to, say, copyrights, publishing, both or neither, so it doesn't take a lot of deliberation to decide to apply that label. Also, with the web a part of everyday life, there's always something to be annoyed about.

So if you take out "technical stuff that no one notices unless it breaks", "social media critiques", "annoying stuff, unless maybe it's particularly annoying, funny or interesting", along with recusing myself from "hmm ... what's Google up to these days?", what's left?

Certainly something. I haven't stopped posting entirely and I don't plan to. On the other hand, there doesn't seem to be as much low-hanging fruit as there used to be, at least not in the particular orchard I'm wandering through. Some of this, I think, is because the web has changed, as I said up top. Some of it is because my focus has changed. I've been finding the topics on the other blog more interesting, not that I've been exactly prolific there either. Some of it is probably the old adage that if you write every day, there's always something to say, while if you write infrequently, it's hard to get started.

A little while ago, I went through the whole blog from the beginning and made several notes to myself to follow up, so I may come back to that. In any case new topics will certainly come up (one just did, after all, about why Wikipedia seems to do much better at self-correcting). I think it's a safe bet, though, that it will continue to be a while between posts. Writing this has helped me to understand why, at least.

Please leave us a 5-star review

2021-05-01T19:58:00.002-04:00

It's been long enough that I can't really say I remember for sure, and I can't be bothered to look it up, but as I recall, reviews were supposed to be one of the main ways for the web to correct itself. I might advertise my business as the best ever, even if it's actually not so good, but not to worry. The reviewers will keep me honest. If you're searching for a business, you'll know to trust your friends, or you'll learn which reviewers are worth paying attention to, good information will drive out bad and everyone will be able to make well-informed decisions.

This is actually true, to an extent, but I think it's about the same extent as always. Major publications try to develop a reputation for objective, reliable reviews, as do some personalities, but then, some also develop a reputation for less-than-objective reviews. Some, even, may be so reliably un-objective that there's a bit of useful information in what they say after all. And you can always just ask people you know.

But this is all outside the system of customer reviews that you find on web sites all over the place, whether provided by the business itself, or companies that specialize in reviews. These, I personally don't find particularly useful or, if I were feeling geekly, I'd say the signal/noise ratio is pretty low. It turns out there are a couple of built-in problems with online reviews, that were not only predictable, but were predicted at the time.

First, there's the whole question of identity on the internet. In some contexts, identity is an easy problem: an identity is an email address or a credit or debit account with a bank, or ownership of a particular phone, or something similar that's important to a person in the real world. Email providers and banks take quite a bit of care to prevent those kind of identities from being stolen, though of course it does still happen.

However, for the same reason, we tend to be a bit stingy with this kind of identity. I try hard not give out my credit card details unless I'm making an actual purchase from a reputable merchant, and if my credit card details do get stolen, that card will get closed and a new one opened for the same account. Likewise, I try not to hand out my personal email or phone number to just anyone, for whatever good that does.

When it comes to reviews, though, there's no good way to know who's writing. They might be an actual customer, or an employee of the business in question, or they might be several time zones away writing reviews for money, or they might even be a bot. Platforms are aware of this, and many seem to do a good job of filtering out bogus reviews, but there's always that lingering doubt. As with identities in general, the stakes matter. If you're looking at a local business, the chances are probably good that everyone who's left a review has actually been there, though even then they might still have an axe to grind. In other contexts, though, there's a lot more reason to try to game the system.

But even if everyone is on the up-and-up and leaving the most honest feedback they can, there are still a few pitfalls. One is selection bias. If I've had a reasonably good experience with a business, I'll try to thank the people involved and keep them in mind for future work, or mention them if someone asks, but I generally don't take time to write a glowing review -- and companies that do that kind of work often seem to get plenty of business anyway.

If someone does a really horrible job, or deals dishonestly, though, I might well be in much more of a mood to share my story. Full disclosure: personally I actually don't tend to leave reviews at all, but it's human nature to be more likely to complain in the heat of the moment than to leave a thoughtful note about a decent experience, or even an excellent experience. In other words, you're only seeing the opinions of a small portion of people. That wouldn't be so bad if the portion was chosen randomly, but it's anything but. You're mostly seeing the opinions of people with strong opinions, and particularly, strong negative opinions.

The result is that reviews tend to cluster toward one end or the other. There are one-star "THIS PLACE IS TERRIBLE!!!" reviews, there are five-star "THIS PLACE IS THE MOST AWESOME EVER!!!" reviews, and not a lot in between. A five-point scale with most of the action at the endpoints is really more of a two-point scale. In effect, the overall rating is the weighted average of the two: the number of one-star reviews plus five times the number of five-star reviews, divided by the total number of reviews. If the overall rating is close to five, then most of the reviews were 5-star. If it's 3, it's much more likely that the good and the bad are half-and-half than most of the reviews being 3-star.

The reader is left to try to decide why the reviewers have such strong opinions. Did the car wash do a bad job, or was the reviewer somehow expecting them to change the oil and rotate the tires as well and then get angry when they didn't? Is the person praising a consultant's integrity actually just their cousin? Does the person saying that a carpenter did a great job with their shelves actually know much about carpentry or did they just happen to like the carpenter's personality? If the shelves collapse after a year and a half, are they really going to go back and update their review? Should they, or should they maybe not store their collection of lead ingots from around the world on a set of wooden shelves?

Specifics can help, but people often don't provide much specific detail, particularly for positive reviews, and when they do, it's not always useful. If all I see is three five-star reviews saying "So and so was courteous, professional and did great work", I'm not much better off than when I started. If I see something that starts out with "Their representative was very rude. They parked their truck in a place everyone in the neighborhood knows not to park. The paint on the truck was chipped. Very unprofessional!" I might take what follows with a grain of salt.

There's a difference, I think, between an opinion and a true review. A true review is aimed at laying out the information that someone else might need to make a decision. An opinion is just someone's general feeling about something. If you just ask people to "leave a review", you're going to get a lot more personal impressions than carefully constructed analyses. Carefully constructing an analysis is work, and no one's getting paid here.

Under the "wisdom of crowds" theory, enough general impressions will aggregate into a complete and accurate assessment. A cynic would say that this is like hoping that if you put together enough raw eggs, you'll end up with a soufflé, but there are situations where it can actually work (for a crowd, that is, not for eggs). The problem is that in many cases you don't even have a crowd. You have a handful of people with their various experiences and opinions.

This all reaches its logical conclusion in the gig economy. When ride share services first started, I used to think for a bit about what number to give a driver. "They were pretty good, but I wish they had driven a bit less (or in some cases maybe more) aggressively". "The car was pretty clean, but there was a bit of a funny smell" or whatever.

Then I started noticing that almost all drivers had 5-star ratings, or close. The number before the decimal point doesn't really mean anything. You're either looking at 5.0 or 4.something. A 4.9 is still a pretty good rating, but a 4.0 rating is actually conspicuously low. I don't know the exact mechanics behind this, but the numbers speak for themselves.

It's a separate question to what extent we should all be in the business of rating each other to begin with, but I'll let Black Mirror speak to that.

Following all this through, if I give someone a 4-star review for being perfectly fine but not outstanding, I may actually be putting a noticeable dent in their livelihood, and if I give someone 3 stars for being pretty much in the middle, that's probably equivalent to their getting a D on a test. So anyone who's reasonably good gets five stars, and if they're not that good, well, maybe they were just having a bad day and I'll just skip the rating. If someone actively put my life in danger, sure, they would get an actual bad rating and I'd see if I could talk to the company, but beyond that ... everyone is awesome.

Whatever the reasons, I think this is a fairly widespread phenomenon. Reviews are either raves or pans, and anyone or anything with reviews much short of pure raves is operating at a real disadvantage. Which leads me back to the title.

Podcasts that I listen to, if they mention reviews at all, don't ask "Please leave a review so we can tell what's working and what we might want to improve". They ask "Please leave a 5-star review". The implication is that anything less is going to be harmful to their chances of staying in business. Or at least that's my guess, because I've heard this from science-oriented podcasts and general-interest shows that clearly take care to present their stories as objectively as they can, the kind of folks who might genuinely appreciate a four-star review with a short list of things to work on.

This is a shame. A five-point scale is pretty crude to begin with, but when it devolves to a two-point scale of horrible/awesome, it's not providing much information at all, pretty much the opposite of the model that I'm still pretty sure people were talking about when the whole ratings thing first started.

One thing at a time

2020-09-05T18:24:00.003-04:00

As much as I gripe about UX annoyances (and all manner of other annoyances), I really do try to look out for specific ways to improve. I don't come up with many, most likely because UX is hard and lots of people who are better at it than I am have spent a lot of time on the problem and come up with a lot of good ideas. Much of the low-hanging fruit has been picked, and so has a lot of the not-so-low-hanging fruit.

However, while grumbling at a particular web page today, I think I hit upon a good rule. I doubt it's new, because a lot of sites follow it (and see above regarding fruit), but a lot don't, so I'll put it out here anyway, for my vast audience, just in case.

Changing one setting on a page should only change the corresponding thing(s) on that page

For example, say I'm looking at statistics on farm production in US states. I can rank output by, say, yield per acre, dollar value, amount per capita and dollar value per capita. I can pick a specific list of states or crops. I pick corn and soybeans for crops and North and South Dakota, Nebraska, Kansas and Oklahoma for states. Up comes a nice table, initially sorted alphabetically by state. I change the sorting order to dollars per capita, from high to low. So far so good.

Now I decide to add wheat to the set of crops. On a well-designed page, I will now see the same data, for the new set of crops, sorted the same way as before. On all too many sites, I see the data for corn, beans and wheat, but sorted alphabetically by state, because that's how all tables start life. I changed one thing -- which crops I'm interested in -- but two things changed, namely the data being shown and the sort order. I only wanted one thing to change, namely the set of crops.

This is a small example, but I'd be surprised if you haven't run across something similar. As described, it's a minor annoyance, but as the options get more sophisticated, annoyance turns into unusability. If I've spent five minutes setting up a graph or chart of, say, crop distribution as a function of latitude, I don't want that all to go away if I decide to include Colorado or Iowa in my set of states.

This is not to say you can't have settings with wider-ranging effects. If there's a tab on the page for, say, trends in agricultural veterinary medicine, I wouldn't expect my graph of crop production to stick around (though I would very much like it to still be there if I go back to its tab). That's fine. I changed one setting, but it's a big setting and the "corresponding things" that need changed are correspondingly big.

Again, this is nothing new. For example, it fits nicely into considerate software remembers. Still, it's often useful to find specific examples of more general principles.

Still here, still annoyed with the internet

2020-07-25T13:11:00.035-04:00

Looks like it's been several months since the last post, which has happened before but probably not for quite this long. I've been meaning to put something up, first about working from home (or WFH as we like to call it), then more about machine learning (or ML as we like to call it), which seems to be going interesting places but probably not as far and fast as some might say. I probably will get back to at least one of those topics, but so far, having settled into a new routine, I just haven't worked in much time for the blogs.

I have been reading quite a bit, on various topics, a lot of it on my phone. I've managed to train my news feed to deliver a reasonable mix of nerdy stuff, light entertainment and what's-going-on-in-the-world. I'm often happy to read the light entertainment in particular, since I get to use my analytical brain plenty between work and writing the occasional analytical blog post. The only problem with the light reading is the actual reading.

I've always said that writers, and "content creators" in general, need to get paid, and I don't mind having to look at the occasional ad or buy the occasional subscription to support that. It's just that the actual mechanics of this are getting a bit out of hand.

Generally one of three things happens. For major outlets, or most of the nerdy stuff, or publications for which I do have a subscription, I click through and read. Great.

If there's a paywall, I usually see the first paragraph or so, enough to confirm what the article is about, and then a button asking me to join in order to see more. I pretty much never do, even though I'm fine with the concept and subscriptions are generally pretty cheap, because

Dude, I just wanted to read the article and it sure would have been nice to have seen a paywall notice before I clicked through (sometimes they're there, but usually not).
I'm leery of introductory rates that quietly start charging you more because you forgot to go back and cancel.
And combining the previous two items, I don't really want to dig through the subscription terms and find out how much I'm really paying and what I'm actually paying for.

I'm a bit more amenable to the "You have N free articles left this month" approach, because I get to read the particular article I was interested in and figure out the subscription stuff at my leisure. I seldom get around to that last part, but nonetheless I think all the subscriptions I've actually bought have been on that basis. I'm sure there have been theses written about the psychology behind that.

Having re-read the whole blog a while ago, I recall that Xanadu advocated for a similar pay-as-you-go approach. As far as I could tell from the demo I saw, it would have led to a sort of taxicab-like "meter is running" experience. This seems even slightly less pleasant than paywalls and subscriptions, but Xanadu could probably have supported either model, in theory.

The more common experience, of course, is ads, particularly in the light entertainment department. What happens is interesting: You see the ads so much you don't see them, and depending on your level of patience, you might not bother to see the light entertainment either.

Suppose you run across a suitably light-looking title. Some popular ones are "Learn something new about <your favorite movie, album, artist etc.>" and "N best/worst/most surprising/... Xs". In either case, there are always two or three paragraphs of things you already know. "My Cousin the Vampire Chauffeur [not a real movie that I know of] was one of the great hits of the 1980s, starring Moviestar McMoviestarface as the vampire and That One Actor as their best friend. At first, the friend only thinks it's a little odd that the Chauffeur only drives at night and has removed the rearview mirror from the car, but events take an unexpected turn when ..." Yep, knew that. I clicked through on this because I liked that movie so yes, I've seen it.

About that time the whole screen starts to rearrange itself as various ad-things jostle for position. Often, it all settles back down with the text you were reading still in roughly the same place, but sometimes you have to scroll. About the same time, a video starts playing across the bottom of the screen. There's generally a tiny "x" box at the corner to make it go away, but that's a fool's errand. Another hydra head will regrow to take its place, and there's always the chance you'll accidentally click through instead of dismissing. Instead, stare steadfastly at the text on the screen and nothing else, secure in the knowledge that the whole "subliminal advertising" thing was most likely overblown.

Finish the paragraph you're on and scroll past the display ad between it and the next paragraph. With a fair wind and a favorable moon phase, you'll get to the next paragraph. If not, the game of musical chairs will resume until the new batch of ads have all found places, at which point I generally head for the exit. But you persevere. You quickly realize that this paragraph as well is more filler, so you try to scroll to the bottom for the nugget of information you were really after. You scroll too far, too fast, and land in a column of photos and links for similar articles, some of which you've already read because, well, we're all human, right?

Scroll back up and you find the object of your quest, that last paragraph, derive whatever edification you can from it and hit the back button. Rather than going back to the news feed, you quite likely go back to a previous version of the page you were reading, and maybe another after that, before ending up back in civilization. I could write a whole other rant about "Why doesn't the back button just take me back?" but I'm not sure that would improve either my life or yours.

I mean, in the grand scheme of things this is all pretty trivial, but then, in the grand scheme of things so is this blog, so I guess we're even.

Except for ads in the middle of lists-of-N-things that disguise their click-through buttons as "next item" buttons. Those are pure evil.

So, still here, still annoyed with the internet.

Did the internet kill the radio tower?

2019-10-29T15:18:00.002-04:00

"Half your advertising budget is wasted. You just don't know which half."

The other day I turned on my radio on the drive home from work. There was a breaking news story I was interested in ("breaking" as in, it was actually happening at the time, not "someone told the Chyron writer to make something look important"). I hadn't done that in months. Years ago, listening to the radio was an integral part of making a cross-country trip, just as reading the Sunday funnies and (lightly) browsing the rest of the newspaper used to be a regular habit. Even not so long ago, listening to the news on the way home was the default option.

Then came podcasts. I was a bit late to the game, mostly because I'm somewhat deliberately lazy about adopting new technology, but once I got a suitable app set up to my liking, podcasts rapidly took over. I could pick out whatever information or entertainment I wanted streamed into my brain, rewind or fast forward as needed and never have to worry about reception. The only downside is needing to get everything set up nicely before actually starting the car moving. I'm sure there are apps for that, but as I said, I'm a bit lazy about apps and such.

I know that people still listen to the radio. Somehow my switching over didn't magically cause everyone else to stop tuning in to their favorite on-air personalities and call-in shows. But for a certain kind of listener, there's little reason to fiddle with a radio. Chances are you can livestream your favorite sports events if you like, though then you do have to worry about reception.

Podcasts, livestreams and other online content don't just change things for listeners. There's a crucial difference for the people creating and distributing the content. Even if "podcast" deliberately sounds like "broadcast", it's actually a classic example of narrowcasting -- delivering content directly to the "content consumers" based on their particular preferences.

Broadcasting is anonymous. I send a signal into the ether and whoever picks it up picks it up. I have no direct way of knowing how many people are listening, much less who. Obviously this is much more anonymous than the internet. It also has economic implications.

There are two main ways of paying for content: subscription and advertising. In either case, it's valuable to know exactly who's on the other end. Narrowcasting gives very find-grained information about that, while broadcasting provides only indirect, aggregated information based on surveys or, failing that, the raw data of who's buying advertising and how much they're paying. Between that and satellite radio's subscription model, is there any room left for broadcast radio?

Probably. I mean, it hasn't gone away yet, any more than printed books have.

Sure, broadcasters and the people who advertise on broadcast radio don't have detailed information about who's listening to the ads, but that may not matter. The advertiser just needs to know that spending $X on old-fashioned radio advertising brings in more than $X in business. The tools for figuring that out have been around since the early days of radio.

If people still find radio advertising effective, the broadcaster just has to know that enough people are still buying it to keep the lights on and the staff paid. In a lot of cases that staff is shared across multiple physical radio stations anyway (and the shows, I would expect, are sent to those stations over the internet). In other words, it may be valuable to know in detail who's listening to what, but it's not essential.

On the other hand, if broadcast radio does go away, I probably won't find out about it until I happen to switch my car audio over to it and nothing's there.

Space Reliability Engineering

2019-07-16T02:54:00.002-04:00

In a previous post on the Apollo 11 mission, I emphasized the role of software architecture, and the architect Margaret Hamilton in particular, in ensuring the success of the Apollo 11 lunar landing. I stand by that, including the assessment of the whole thing as "awesome" in the literal sense, but as usual there's more to the story.

Since that non-particularly-webby post was on Field Notes, so is this one. What follows is mostly taken from the BBC's excellent if majestically paced podcast 13 Minutes to the Moon [I hope to go back and recheck the details directly at some point, but searching through a dozen or so hours of podcast is time-consuming and I don't know if there's a transcript available -- D.H.], which in turn draws heavily on NASA's Johnson Space Center Oral History Project.

I've also had a look at Ars Technica's No, a "checklist error" did not almost derail the Apollo 11 mission, which takes issue with Hamilton's characterization of the incident and also credits Hal Laning as a co-author of the Executive portion of the guidance software which ultimately saved the day (to me, the main point Hamilton was making was that the executive saved the day, regardless of the exact cause of the 1202 code).

Before getting too far into this, it's worth reiterating just how new computing was at the time. The term "software engineer" didn't exist (Hamilton coined it during the project -- Paul Niquette claims to have coined the term "software" itself and I see no reason to doubt him). There wasn't any established job title for what we now call software engineers. The purchase order for the navigation computer, which was the very first order in the whole Apollo project, didn't mention software, programming or anything of the sort. The computer was another piece of equipment to be made to work just like an engine, window, gyroscope or whatever. Like them it would have to be installed and have whatever other things done to it to make it functional. Like "programming" (whatever that was).

In a way, this was a feature rather than a bug. The Apollo spacecraft have been referred to, with some justification, as the first fly-by-wire vehicles. The navigational computer was an unknown quantity. At least one astronaut promised to turn the thing off at the first opportunity. Flying was for pilots, not computers.

This didn't happen, of course. Instead, as the podcast describes so well, control shifted back and forth between human and computer depending on the needs of the mission at the time, but it was far from obvious at the beginning that this would be the case.

Because the computer wasn't trusted implicitly, but treated as just another unknown to be dealt with, -- in other words, another risk to be mitigated -- ensuring its successful operation was seen as a matter of engineering, just like making sure that the engines were efficient and reliable, and not a matter of computer science. This goes a long way toward explaining the self-monitoring design of the software.

Mitigating the risk of using the computer included figuring out how to make it as foolproof as possible for the astronauts to operate. The astronauts would be wearing spacesuits with bulky gloves, so they wouldn't exactly be swiping left or right, even if the hardware of the time could have supported it. Basically you had a numeric display and a bunch of buttons. The solution was to break the commands down to a verb and a noun (or perhaps more accurately a predicate and argument), each expressed numerically. It would be a ridiculous interface today. At the time it was a highly effective use of limited resources [I don't recall the name of the designer who came up with this. It's in the podcast --D.H.].

But the only way to really know if an interface will work is to try it out with real users. Both the astronauts and the mission control staff needed to practice the whole operation as realistically as possible, including the operation of the computer. This was for a number of reasons, particularly to learn how the controls and indicators worked, to be prepared for as many contingencies as possible and to try to flush out unforeseen potential problems. The crew and mission control conducted many of these simulations and they were generally regarded as just as demanding and draining as the real thing, perhaps moreso.

It was during one of the simulations that the computer displayed a status code that no one had ever seen before and therefore didn't know how to react to. After the session was over, flight director Gene Kranz instructed guidance software expert Jack Garman to look up and memorize every possible code and determine what course of action to take when it came up. This would take a lot of time searching through the source code, with the launch date imminent, but it had to be done and it was. Garmin produced a handwritten list of every code and what to do about it.

As a result, when the code 1202 came up with the final opportunity to turn back fast approaching, capsule communicator (CAPCOM) Charlie Duke was able to turn to guidance controller Steve Bales, who could turn to Garman and determine that the code was OK if it didn't happen continuously. There's a bit of wiggle room in what constitutes "continuously", but knowing that the code wasn't critical was enough to keep the mission on track. Eventually, Buzz Aldrin noticed that the code only seemed to happen when a particular radar unit was being monitored. Mission Control took over the monitoring and the code stopped happening.

I now work for a company that has to keep large fleets of computers running to support services that billions of people use daily. If a major Google service is down for five minutes, it's headline news, often on multiple continents. It's not the same as making sure a plane or a spaceship lands safely or a hospital doesn't lose power during a hurricane, but it's still high-stakes engineering.

There is a whole profession, Site Reliability Engineer, or SRE for short, dedicated to keeping the wheels turning. These are highly-skilled people who would have little problem doing my job instead of theirs if they preferred to. Many of their tools -- monitoring, redundancy, contingency planning, risk analysis, and so on -- can trace their lineage through the Apollo program. I say "through" because the concepts themselves are considerably older than space travel, but it's remarkable how many of them were not just employed, but significantly advanced, as a consequence of the effort to send people to the moon and bring them back.

One tool in particular, Garman's list of codes, played a key role at a that critical juncture. Today we would call it a playbook. Anyone who's been on call for a service has used one (I know I have).

In the end, due to a bit of extra velocity imparted during the maneuver to extract the lunar module and dock it to the command module, the lunar module ended up overshooting its intended landing place. In order to avoid large boulders and steep slopes in the area they were now approaching, Neil Armstrong ended up flying the module by hand in order to find a good landing spot, aided by a switch to increase or decrease the rate of descent.

The controls were similar to those of a helicopter, except the helicopter was flying sideways through (essentially) a vacuum over the surface of the moon, steered by precisely aimed rocket thrusts while continuing to descend, and was made of material approximately the thickness of a soda can which could have been punctured by a good jab with a ball-point pen. So not really like a helicopter at all.

The Eagle landed with eighteen seconds of fuel to spare. It helps to have a really, really good pilot.

Distributed astronomy

2019-04-16T10:48:00.000-04:00

Recently, news sources all over the place have been reporting on the imaging of a black hole, or more precisely, the immediate vicinity of a black hole. The black hole itself, more or less by definition, can't be imaged (as far as we know so far). Confusing things a bit more, any image of a black hole will look like a black disc surrounded by a distorted image of what's actually in the vicinity, but this is because the black hole distorts space-time due to its gravitational field, not because you're looking at something black. It's the most natural thing in the world to look at the image and think "Oh, that round black area in the middle is the black hole", but it's not.

Full disclosure: I don't completely understand what's going on here. Katie Bouman has done a really good lecture on how the images were captured, and Matt Strassler has an also really good, though somewhat long overview of how to interpret all this. I'm relying heavily on both.

Imaging a black hole in a nearby galaxy has been likened to "spotting a bagel on the moon". A supermassive black hole at the middle of a galaxy is big, but even a "nearby" galaxy is far, far away.

To do such a thing you don't just need a telescope with a high degree of magnification. The laws of optics place a limit on how detailed an image you can get from a telescope or similar instrument, regardless of the magnification. The larger the telescope, the higher the resolution, that is, the sharper the image. This applies equally well to ordinary optical telescopes, X-ray telescopes, radio telescopes and so forth. For purposes of astronomy these are all considered "light", since they're all forms of electromagnetic radiation and so all follow the same laws.

Actual telescopes can only be built so big, so in order to get sharper images astronomers use interferometry to combine images from multiple telescopes. If you have a telescope at the South Pole and one in the Atacama desert in Chile, you can combine their images to get the same resolution you would with a giant telescope that spanned from Atacama to the pole. The drawback is that since you're only sampling a tiny fraction of the light falling on that area, you have to reconstruct the rest of the image using highly sophisticated image processing techniques. It helps to have more than two telescopes. The Event Horizon Telescope project that produced the image used eight, across six sites.

Even putting together images from several telescopes, you don't have enough information to precisely know what the full image really would be and you have to be really careful to make sure that the image you reconstruct shows things that are actually there and not artifacts of the processing itself (again, Bouman's lecture goes into detail). In this case, four teams worked with the raw data independently for seven weeks, using two fundamentally different techniques, to produce the images that were combined into the image sent to the press. In preparation for that, the image processing techniques themselves were thoroughly tested for their ability to recover images accurately from test data. All in all, a whole lot of good, careful work by a large number of people went into that (deliberately) somewhat blurry picture.

All of this requires very precise synchronization among the individual telescopes, because interferometry only works for images taken at the same time, or at least to within very small tolerances (once again, the details are ... more detailed). The limiting factor is the frequency of the light used in the image, which for radio telescopes is on the order of gigahertz. This means that images from the telescopes have to be recorded on the order of a billion times a second. The total image data ran into the petabytes (quadrillions of bytes), with the eight telescopes producing hundreds of terabytes (that is, hundreds of trillions of bytes) each.

That's a lot of data, which brings us back to the web (as in "Field notes on the ..."). I haven't dug up the exact numbers, but accounts in the popular press say that the telescopes used to produce the black hole images produced "as much data as the LHC produces in a year", which in approximate terms is a staggering amount of data. A radio interferometer comprising multiple radio telescopes at distant points on the globe is essentially an extremely data-heavy distributed computing system.

Bear in mind that one of the telescopes in question is at the south pole. Laying cable there isn't a practical option, nor is setting up and maintaining a set of radio relays. Even satellite communication is spotty. According to the Wikipedia article, the total bandwidth available is under 10MB/s (consisting mostly of a 50 megabit/second link), which is nowhere near enough for the telescope images, even if stretched out over days or weeks. Instead, the data was recorded on physical media and flown back to the site where it was actually processed.

I'd initially thought that this only applied to the south pole station, but in fact all six sites flew their data back rather than try to send it over the internet (just to throw numbers out, receiving a petabyte of data over a 10GB/s link would take about a day). The south pole data just took longer because they had to wait for the antarctic summer.

Not sure if any carrier pigeons were involved.

Martian talk

2019-04-04T10:46:00.000-04:00

This morning I was on the phone with a customer service representative about emails I was getting from an insurance company and which were clearly meant for someone else with a similar name (fortunately nothing earth-shaking, but still something this person would probably like to know about). As is usually the case, the reply address was a bit bucket, but there were a couple of options in the body of the email: a phone number and a link. I'd gone with the phone number.

The customer service rep politely suggested that I use the link instead. I chased the link, which took me to a landing page for the insurance company. Crucially, it was just a plain link, with nothing to identify where it had come from*. I wasn't sure how best to try to get that across to the rep, but I tried to explain that usually there are a bunch of magic numbers or "hexadecimal gibberish" on a link like that to tie it back to where it came from.

"Oh yeah ... I call that 'Martian talk'," the rep said.

"Exactly. There's no Martian talk on the link. By the way, I think I'm going to start using that."

We had a good laugh and from that point on we were on the same page. The rep took all the relevant information I could come up with and promised to follow up with IT.

What I love about the term 'Martian talk' is that it implies that there's communication going on, but not in a way that will be meaningful to the average human, and that's exactly what's happening.

And it's fun.

I'd like to follow up at some point and pull together some of the earlier posts on Martian talk -- magic numbers, hexadecimal gibberish and such -- but that will take more attention than I have at the moment.

* From a strict privacy point of view there would be plenty of clues, but there was nothing to tie the link to a particular account for that insurance company, which was what we needed.

Hats off to New Horizons

2019-01-03T19:56:00.000-05:00

A few years ago, around the time of the New Horizons encounter with Pluto (or if you're really serious about the demotion thing, minor planet 134340 Pluto), I gave the team a bit of grief over the probe having to go into "safe mode" with only days left before the flyby, though I also tried to make clear that this was still engineering of a very high order.

Early on New Year's Day (US Eastern time), New Horizons flew by a Kuiper Belt object nicknamed Ultima Thule (two syllables in Thule: THOO-lay). I'm posting to recognize the accomplishment, and this post will be grief-free.

The Ultima Thule encounter was much like the Pluto encounter with a few minor differences:

Ultima Thule is much smaller. Its long axis is about 1-2% of Pluto's diameter
Ultima Thule is darker, reflecting about 10% of light that reaches, compared to around 50% for Pluto. Ultima Thule is about as dark as potting soil. Pluto is more like old snow.
Ultima Thule is considerably further away (about 43 AU from the sun as opposed to about 33 AU for Pluto at the time of encounter -- an AU is the average distance from the Sun to the Earth)
New Horizons passed much closer to Ultima Thule than it did to Pluto (3,500 km vs. 12,500 km). This requires more accurate navigation and to some extent increased the chances of a disastrous collision with either Ultima Thule or, more likely, something near it that there was no way to know about. At 50,000 km/h, even a gravel-sized chunk would cause major if not fatal damage.
Because Ultima Thule is further away, radio signals take proportionally longer to travel between Earth and the probe, about six hours vs. about four hours.
Because Ultima Thule is much smaller, much darker and significantly further away, it's much harder to spot from Earth. Before New Horizons, Pluto itself was basically a tiny dot, with a little bit of surface light/dark variation inferred by taking measurements as it rotated. Ultima Thule was nothing more than a tinier dot, and a hard-to-spot dot at that.
We've had decades to work out exactly where Pluto's orbit goes and where its moons are. Ultima Thule wasn't even discovered until after New Horizons was launched. Until a couple of days ago we didn't even know whether it had moons, rings or an atmosphere (it appears to have none). [Neither Pluto nor Ultima Thule is a stationary object, just to add that little additional degree of difficulty. The Pluto flyby might be considered a bit more difficult in that respect, though. Pluto's orbital speed at the time of the flyby was around 20,000 km/h, while Ultima Thule's is closer to 16,500 km/h. I'd think this would mainly affect the calculations for rotating to keep the cameras pointed, so it probably doesn't make much practical difference.]

In both cases, New Horizons had to shift from pointing its radio antenna at Earth to pointing its cameras at the target. As it passes by the target at around 50,000 km/h, it has to rotate to keep the cameras pointed correctly, while still out of contact with Earth (which is light-hours away in any case). It then needs to rotate its antenna back toward Earth, "phone home" and start downloading data at around 1,000 bits per second. Using a 15-watt transmitter slightly more powerful than a CB radio. Since this is in space, rotating means firing small rockets attached to the probe in a precise sequence (there are also gyroscopes on New Horizons, but they're not useful for attitude changes).

So, a piece of cake, really.

Seriously, though, this is amazing engineering and it just gets more amazing the more you look at it. The Pluto encounter was a major achievement, and this was significantly more difficult in nearly every possible way.

So far there don't seem to be any close-range images of Ultima Thule on the mission's web site (see, this post is actually about the web after all), but the team seems satisfied that the flyby went as planned and more detailed images will be forthcoming over the next 20 months or so. As I write this, New Horizons is out of communication, behind the Sun from Earth's point of view for a few days, but downloads are set to resume after that. [The images started coming in not long after this was posted, of course --D.H. Jul 2019]

Common passwords are bad ... by definition

2018-12-13T16:05:00.001-05:00

It's that time of the year again, time for the annual lists of worst passwords. Top of at least one list: 123456, followed by password. It just goes to show how people never change. Silly people!

Except ...

A good password has a very high chance of being unique, because a good password is selected randomly from a very large space of possible passwords. If you pick your password at random from a trillion possibilities*, then the odds that a particular person who did the same also picked your password are one in a trillion, the odds that one of a million other such people picked your password are about one in a million, as are the odds that any particular two people picked the same password. If a million people used the same scheme as you did, there's a good chance that some pair of them accidentally share a password, but almost certainly almost all of those passwords are unique.

If you count up the most popular passwords in this idealized scenario of everyone picking a random password out of a trillion possibilities, you'll get a fairly tedious list:

1: some string of random gibberish, shared by two people
2 - 999,999: Other strings of random gibberish, 999,998 in all

Now suppose that seven people didn't get the memo. Four of them choose 123456 and three of them choose password. The list now looks like

1: 123456, shared by four people
2: password, shared by three people
3: some string of random gibberish, shared by two people
4-999,994: Other strings of random gibberish, 999,991 in all

Those seven people are pretty likely to have their passwords hacked, but overall password hygiene is still quite good -- 99.9993% of people picked a good password. It's certainly better than if 499,999 people picked 123456 and 499,998 picked password, two happened to pick the same strong password and the other person picked a different strong password, even though the resulting rankings are the same as above.

Likewise, if you see a list of 20 worst passwords taken from 5 million leaked passwords, that could mean anything from a few hundred people having picked bad passwords to everyone having done so. It would be more interesting to report how many people picked popular passwords as opposed to unique ones, but that doesn't seem to make its way into the "wow, everyone's still picking bad passwords" stories.

From what I was able to dig up, that portion is probably around 10%. Not great, but not horrible, and probably less than it was ten years ago. But as long as some people are picking bad passwords, the lists will stay around and the headlines will be the same, regardless of whether most people are doing a better job.

(I would have provided a link for that 10%, but the site I found it on had a bunch of broken links and didn't seem to have a nice tabular summary of bad passwords vs other passwords from year to year, so I didn't bother)

*A password space of a trillion possibilities is actually pretty small. Cracking passwords is roughly the same problem as the hash-based proof-of-work that cyrptocurrencies use. Bitcoin is currently doing around 100 million trillion hashes per second, or a trillion trillion hashes every two or three hours. The Bitcoin network isn't trying to break your password, but it'll do for estimating purposes. If you have around 100 bits of entropy, for example if you choose a random sequence of fifteen capital and lowercase letters, digits and 30 special characters, it would take a password-cracking network comparable to the Bitcoin network around 400 years to guess your password. That's probably good enough. By that time, password cracking will probably have advanced far beyond where we are and, who knows, maybe we'll have stopped using passwords by then.